Target detection method based on multi-modal data fusion and 3D voxel projection

Abstract

The invention discloses a target detection method based on multi-modal data fusion and 3D voxel projection, and the method comprises the steps: firstly, extracting and fusing multi-modal features from an RGB image and a depth image; thirdly, aggregating the multi-scale features by using a feature pyramid network, and accurately mapping the 2D feature map into a 3D voxel space by using real depth information to form a 3D voxel volume; then, local 3D features are extracted through a voxel feature coding network, and global features are further extracted through a 3D convolutional neural network; and finally, inputting the global features into a detection head to obtain a final target detection result. The model is trained on a large public data set and can adapt to data with high feature dimension, large structure difference and unbalanced distribution, the accuracy and robustness of 3D target detection can be effectively improved through fusion of multi-modal features, and the method is suitable for object detection tasks in a logistics sorting scene.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and deep learning, and in particular to a target detection method based on multimodal data fusion and 3D voxel projection to achieve object detection in indoor environments. Background Art

[0002] In modern logistics sorting scenarios, the application of automation and intelligent technologies is becoming increasingly popular. The ability to quickly and accurately detect and identify the three-dimensional size and category of various objects is crucial to improving sorting efficiency. This can not only significantly reduce the cost of manual operations, but also improve the overall efficiency and reliability of the logistics system. Therefore, developing an efficient and robust 3D object detection method has become one of the current research hotspots.

[0003] With the increase in the types of sensors and the improvement in their performance, the data collection capability in logistics sorting scenarios has been significantly improved. Various high-precision and high-sensitivity sensors, such as LiDAR, depth cameras, and RGB cameras, provide rich and diverse three-dimensional spatial data, which not only contain geometric information such as the shape and size of objects, but also contain rich visual features such as color and texture, laying a solid foundation for accurate and efficient 3D target detection. In order to fully explore and utilize the complementary information in these multi-source data and improve the robustness and accuracy of target detection, data fusion technology has become a key link. Based on the role of the fusion results in the model, different fusion methods can be classified according to the fusion level, which can be mainly divided into four categories: data fusion, feature fusion, result fusion, and auxiliary estimation. (1) Data fusion refers to the fusion of data from different sources and the use of the fusion results as the input of the subsequent model. The subsequent model can be a complete independent model or a branch or component in the integrated model. The data to be fused can come from a variety of sources, such as raw data collected by sensors, preprocessed data, and the output results of sub-models in the integrated model, such as feature vectors, feature maps, or other calculation results. (2) Feature fusion usually refers to the fusion of feature maps or feature vectors from different inputs during the model calculation process. The fusion result will be used as the output feature of the current model for subsequent calculations. (3) Result fusion refers to the fusion of multiple output results as the final target output in the output stage of the model, or when the input data itself is the target output, the output result is corrected or optimized through fusion processing. Result fusion mainly includes two forms: one is the fusion of calculation results of parallel model branches; the other is that in the serial model, subsequent sub-models perform fusion optimization of the same target output based on the results of the previous stage. (4) Auxiliary estimation refers to predicting or estimating the data of another modality based on the data of a specific modality in order to optimize the data representation of the model. Unlike the above three fusion methods, the auxiliary estimation process does not reduce the modal types of the data, and the presence or absence of the auxiliary estimation module does not affect the integrity of the fusion model.

[0004] In logistics sorting systems, the application of data fusion technology can significantly improve the accuracy and robustness of the system. For example, the fusion of RGB images and depth maps can provide more comprehensive information for object detection, enabling the system to accurately identify and locate objects in complex environments. With the continuous advancement of sensor technology and data fusion technology, 3D object detection technology has broad application prospects in modern logistics sorting scenarios. Designing efficient and robust multimodal data fusion methods is of great significance to improving the intelligence level and work efficiency of logistics systems.

[0005] In the existing technology, 3D target detection mainly comes from two types of data: LiDAR point cloud data and RGB image data. Single-modality data has limitations in feature expression and target recognition. Although RGB images can provide rich color and texture information, they are insufficient in expressing depth and spatial relationships. Point cloud data provides accurate distance information, but lacks resolution and detail expression in complex scenes, and the acquisition cost is high, which hinders its widespread application in more scenarios.

[0006] In recent years, researchers have tried to combine multimodal data to improve the performance of 3D target detection by fusing information from RGB images and depth maps. However, existing methods have limitations in feature extraction and fusion, and it is difficult to fully utilize information from different modalities. Specific problems include: (1) The feature fusion method is single and cannot fully explore the complementary information between multimodal data. (2) During the feature extraction process, features at different resolutions and scales cannot be effectively processed, resulting in insufficient detection accuracy. (3) 2D features are difficult to directly map to 3D space, and there is a lack of effective 2D to 3D feature conversion methods. The present invention proposes a target detection method based on multimodal data fusion and 3D voxel projection to solve the above problems. Summary of the invention

[0007] The present invention adopts a multimodal data fusion method to fuse RGB information with depth information to improve the accuracy and robustness of indoor 3D target detection. Existing 3D target detection technology mainly faces the following problems: (1) The lack of depth information leads to low accuracy of target positioning and recognition, which limits the effective application of traditional 2D image processing technology in complex three-dimensional scenes. (2) Although point cloud data is rich in depth information, the high acquisition cost limits its popularity and application in a wide range of scenarios. (3) Target detection methods based on a single modality are difficult to fully capture the multi-dimensional information in the scene, affecting the accuracy and robustness of target detection.

[0008] In view of the above problems, this paper proposes a target detection method based on multimodal data fusion and 3D voxel projection, which performs indoor 3D target detection tasks based on large public data sets, fuses RGB images and depth image data, and combines real depth information for accurate 3D voxel projection to improve the accuracy and robustness of indoor 3D target detection. The model structure is as follows Figure 1 shown.

[0009] The specific process is as follows Figure 2 As shown, the following steps are included:

[0010] Step 1: Data preprocessing. Load RGB images, depth maps, and annotation information from the specified path, perform data augmentation operations such as random cropping and flipping on the data, convert RGB images and depth maps into tensors, convert annotation information into bounding boxes and category labels, and finally normalize the data.

[0011] Step 2: Multimodal feature extraction and fusion. This module uses a layered encoder architecture to efficiently extract and deeply fuse the multimodal features of RGB images and depth images.

[0012] The preprocessed data first passes through the Stem layer, which consists of two parallel dual 3×3 convolutional networks, performing convolution operations on RGB images and depth images respectively to achieve preliminary feature extraction and conversion. Then, the preliminary extracted features are input into the hierarchical encoder module, which is based on the Transformer architecture and captures and fuses complementary information from the two modalities at multiple scales through global perception attention mechanism and local enhancement attention mechanism. The specific representation is as follows:

[0013] F Multi-Scale =Encoder(Stem RGB (I RGB ), Stem Depth (I Depth )) (1)

[0014] Among them, I RGB and I Depth Represents RGB image and depth image respectively. RGB and Stem Depth They are the Stem layers used to process RGB images and depth images respectively.

[0015] This module can deeply mine RGB images and depth images, extract color, texture features as well as distance and geometric features, and efficiently fuse them, providing rich and comprehensive feature representation for visual tasks.

[0016] Step 3: Multi-scale feature aggregation. Multi-scale feature maps are aggregated through the Feature Pyramid Network (FPN) to enhance the detection capability of the model at different resolutions and improve the detection accuracy of objects of different sizes. The specific formula is as follows:

[0017] P i =Conv 1×1 (F i )+Upsample(P i+1 ) (2)

[0018] Among them, F i Represents feature maps of different scales, Pi Represents the feature map after FPN aggregation, Conv 1×1 Represents a 1×1 convolution operation, which is used to adjust the number of channels. Upsample represents an upsampling operation, which upsamples the high-level feature map to the same spatial size as the low-level feature map.

[0019] Step 4: 2D to 3D feature voxel projection. The aggregated 2D feature map is projected into the 3D voxel space in combination with the real depth information to construct an accurate 3D voxel representation. Compared with other methods, this invention uses the real depth information obtained from the depth sensor to improve the accuracy of the 3D feature representation. First, the pixel points and their depth information in the 2D image are converted into three-dimensional points in the camera coordinate system using the intrinsic parameters of the camera. Next, the extrinsic parameters of the camera, including the rotation matrix and translation vector, are used to convert these three-dimensional points from the camera coordinate system to the world coordinate system. Finally, based on the predefined 3D space boundary and voxel size, the specific position of each point in the voxel grid is determined to complete the 3D point mapping from the world coordinate system to the voxel grid.

[0020] Step 5: Voxel local feature extraction. After projecting the 2D features into the 3D voxel space, the voxel feature encoding network (VFE) is used to extract local features for each voxel. First, the depth information within the voxel is extracted, and the feature points in each voxel are encoded into a feature vector of fixed length. Next, the local spatial features within the voxel are extracted through the layer-by-layer feature aggregation mechanism in the VFE network to improve the accuracy of the voxel representation. The VFE network contains several fully connected layers and activation functions, which extract features from each point inside the voxel, aggregate these features, and generate a local feature representation for each voxel. The VFE network not only retains the fine-grained information of the three-dimensional points within the voxel, but also can obtain local geometric shape features, providing richer information for subsequent 3D feature extraction and target detection.

[0021] Step 6: 3D feature extraction and object detection. A 3D convolutional neural network is used to further extract higher-level global features from the voxel features. Subsequently, each anchor point is classified and located through the detection head, and the category probability and 3D bounding box parameters of each anchor point are output to achieve 3D object detection.

[0022] First, the voxel feature V is input into a simplified 3D convolutional encoder-decoder network to extract global features. As shown below:

[0023] {F1, F2, F3} = Decoder(Encoder(V))(3) where V is the voxel feature, Encoder(·) represents the encoder, including three layers of downsampling residual blocks, each block consists of three 3D convolutional layers. Decoder(·) represents the decoder, including three layers of upsampling blocks, each block consists of a transposed 3D convolutional layer and another 3D convolutional layer.

[0024] Next, the obtained global features are fed into a multi-scale detection head, which consists of three 3D convolutional layers for classification, localization, and centrality estimation. These convolutional layers share weights across all scales. For each 3D position (x a ,y a , z a ) and three scale features, the detection head outputs the category probability p, the centrality c and the 3D bounding box parameter Δb. It is expressed as follows:

[0025] {p, c, Δb} = FCOSHead(F i ,(x a ,y a , z a ))(4) Among them, F i , i∈{1, 2, 3} represents the multi-scale feature map output from the encoder-decoder network, and FCOSHead(·) represents the FCOS detection head. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments or technical solutions of the present invention, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art.

[0027] Figure 1 Diagram of the target detection framework based on multimodal data fusion and 3D voxel projection.

[0028] Figure 2 Flowchart of multimodal indoor object detection.

[0029] Figure 3 Figure 2. Network structure diagram for voxel feature encoding. DETAILED DESCRIPTION

[0030] The specific implementation steps of the invention mainly include:

[0031] 1. Data preprocessing

[0032] Load RGB images, depth maps, and annotation information from the specified path, perform data augmentation operations such as random cropping and flipping on the loaded data, convert RGB images and depth maps into tensors, convert annotation information into bounding boxes and category labels, and finally normalize the data.

[0033] 2. Multimodal feature extraction and fusion

[0034] The preprocessed data is used as input, and the Transformer architecture is used to capture and fuse multimodal features from RGB images and depth maps. The network uses a hierarchical encoder to obtain low-resolution coarse-grained features and high-resolution fine features. Specifically, given an RGB image of spatial size H×W and the corresponding depth map, the RGB image and depth map are first processed separately using a parallel Stem layer composed of two convolutional networks to obtain RGB and depth features, which are then input into the hierarchical encoder to obtain multi-scale RGB-D features. The specific representation is as follows:

[0035] F Multi-Scale =Encoder(Stem RGB (I RGB ), Stem Depth (I Depth ))(5) Among them, I RGB and I Depth Represents RGB image and depth image respectively. RGB and Stem Depth These are the Stem layers that process RGB images and depth images respectively.

[0036] Here, each encoder consists of multiple attention modules, each of which mainly includes a global awareness attention module (GAA) and a local enhancement attention module (LEA). The purpose of these modules is to capture and fuse RGB and depth features at different scales to improve the model's 3D perception and ability to capture local details.

[0037] (1) Global Perception Attention Module GAA

[0038] GAA fuses depth and RGB features, establishes feature associations across the scene, and enhances 3D perception, further facilitating the capture of semantic objects. Unlike the self-attention mechanism, which introduces pixels or increases the number of tokens, resulting in a quadratic increase in computational complexity, GAA reduces computational complexity by downsampling the query Q to a fixed size. The query Q is obtained by combining RGB and depth features. Adaptive pooling is performed to fix the spatial dimension to size k×k, and then linear transformation is performed to obtain:

[0039]

[0040] The key K and value V are obtained by After feature transformation, we get:

[0041]

[0042] Among them, Pool k×k (·) is an adaptive average pooling operation, which is used to reduce the spatial dimension to k×k size, and linear is a linear transformation.

[0043] After obtaining Q, K, and V, the GAA module calculates the dot product similarity of Q and K, obtains the attention weight through Softmax normalization, and then performs a weighted product with V, and finally restores the result to the original size through a bilinear upsampling operation. The GAA process is expressed as follows:

[0044] in is the scaling factor of the number of depth channels, and UP(·) is a bilinear upsampling operation that converts the spatial size from k×k to h×w. In this way, the GAA module is able to establish a global association between RGB and depth features, capturing important semantic information in the scene while improving 3D perception capabilities.

[0045] (2) Local Enhanced Attention Module LEA

[0046] The role of the LEA module is to supplement the GAA module and further enhance the expression of local features. Different from the global attention of GAA, LEA focuses on the enhancement of local features. It can effectively aggregate adjacent pixel features with similar depth to enhance the expression of 3D geometric information in RGB features.

[0047] First, using the deep convolution kernel DConv k×k For deep features Perform convolution operations to generate local depth features. Through convolution operations, the correlation between adjacent pixels and local geometric information can be extracted:

[0048]

[0049] in, The deep features are linearly transformed, DConv k×k is a depthwise convolution kernel of size k×k.

[0050] Then, the local depth features obtained by deep convolution are used as weights, and the Hadamard product is used with the RGB features. Re-weighted, the enhanced RGB features are finally obtained:

[0051] Where ⊙ is the Hadamard product. It is a linear transformation of RGB features.

[0052] In this way, the LEA module can effectively aggregate local depth features and RGB features, so that the geometric relationship between adjacent pixels can be integrated into the RGB features, thereby improving the model's ability to capture local details.

[0053] 3. Multi-scale feature aggregation

[0054] The multi-scale feature map outputted in step 2 is aggregated through FPN. The series of feature maps outputted are called F1, F2, and F3. A 1×1 convolution operation is performed on the highest-level feature map F3 to obtain a multi-scale feature map P3. Next, the feature maps of each layer are processed in turn from high level to low level as follows:

[0055] P2=Conv 1×1 (F2)+UPsample(P3)(12)

[0056] P1=Conv 1×1 (F1)+UPsample(P2)(13) In each step, a 1×1 convolution operation is performed on the feature map of the current level to adjust the number of channels, and the feature map of the previous level is upsampled and added to the feature map of the current level.

[0057] The upsampling operation is to adjust the spatial size of the high-level feature map to the same as the low-level feature map. The final P1, P2, and P3 are multi-scale feature maps after FPN aggregation, which contain rich contextual information and detailed features, and enhance the model's ability to detect objects of different sizes.

[0058] 4.2D to 3D feature voxel projection

[0059] The 2D feature map obtained in step 3 is projected into the 3D voxel space to construct the 3D voxel volume, laying the foundation for 3D object detection.

[0060] First, use the camera intrinsic parameters to convert the 2D image coordinates into 3D coordinates under the camera coordinates. The formula is as follows:

[0061] Among them, (u, v) is the coordinate in the 2D image, d is the corresponding depth value, and K is the intrinsic parameter matrix of the camera.

[0062] Then, using the camera external parameters, the 3D points in the camera coordinate system are converted to the world coordinate system, achieving accurate mapping from the pixel points in the 2D image to the 3D points in the world coordinate system. The formula is as follows: in, is the rotation matrix and T is the translation vector.

[0063] According to the requirements of the task and dataset, the boundary of the 3D space is set to [xmin ,x max ]、[y min ,y max ]、[z min ,z max ], set the size of each voxel to s. Next, convert the 3D point (X w ,Y w ,Z w ) is mapped to discrete voxel coordinates (i, j, k) in the voxel grid as follows:

[0064]

[0065] Where (i, j, k) is the index in the 3D voxel grid, indicating the position of the point in the 3D voxel space.

[0066] Next, the features of the 2D feature map are aggregated into 3D voxels. The number of feature channels of each voxel is the same as that of the 2D feature map, and multiple features in the same voxel can be processed by averaging aggregation, maximum pooling, etc.

[0067] 5. Voxel local feature extraction

[0068] The VFE network is used to extract meaningful local features from each voxel, and then obtain feature representations that contain spatial structural information. Specifically, the VFE network processes all points of each voxel in parallel and obtains the local feature representation of the voxel through feature aggregation.

[0069] First, the VFE network aggregates the features of all points inside the 3D voxel and obtains a unified voxel feature vector through pooling operations. Next, the VFE network will further encode the features of each voxel to capture local geometric information and semantic information. In addition, the position information of the voxel in 3D space is also crucial. The VFE network encodes the 3D coordinates of the voxel into a vector of the same dimension as the feature and concatenates it with the semantic feature vector of the voxel, so that the network can capture position-related information, which helps the model distinguish similar targets at different positions. In order to improve the nonlinear expression ability of the network, nonlinear factors are introduced after each convolutional layer and fully connected layer in the VFE network, and nonlinear activation functions (such as ReLU) are used to enable the network to learn more complex feature representations. After the above steps, the VFE network will output a local feature vector for each voxel. These local feature vectors not only contain the semantic information within the voxel, but also incorporate the position information of the voxel in 3D space, providing rich and comprehensive effective information for subsequent global feature extraction and target detection.

[0070] 6.3D feature extraction and object detection

[0071] A 3D convolutional neural network is used to further extract global features from voxel features to form a 3D feature volume. Subsequently, the detection head classifies and locates each anchor point, outputs the category probability and 3D bounding box parameters of each anchor point, and realizes 3D object detection.

[0072] First, the voxel feature V is input into a simplified 3D convolutional encoder-decoder network to extract global features. As shown below:

[0073] {F1, F2, F3} = Decoder(Encoder(V))(17) where Decoder(·) represents the decoder, which includes three layers of upsampling blocks, each block consists of a transposed 3D convolutional layer with a stride of 2 and another 3D convolutional layer, V is the input voxel feature, and Encoder(·) represents the encoder part, which includes three layers of downsampling residual blocks, each block consists of three 3D convolutional layers.

[0074] Subsequently, the obtained global features are input into a modified FCOS detection head. The detection head consists of three 3D convolutional layers for classification, localization, and centrality estimation, which share weights across all scales. For each 3D location (x a ,y a , z a ) and three scale features, the detection head outputs the category probability p, the centrality c and the 3D bounding box parameter Δb. It is expressed as follows:

[0075] {p, c, Δb} = FCOSHead(F i ,(x a ,y a , z a ))(18) Among them, F i , i∈{1, 2, 3} represents the multi-scale feature map output from the encoder-decoder network. FCOSHead(·) represents the improved FCOS detection head.

Claims

1. A target detection method based on multimodal data fusion and 3D voxel projection, characterized in that: Indoor 3D object detection is performed based on a large public dataset, RGB images and depth image data are fused, and accurate 3D voxel projection is performed in combination with real depth information to improve the accuracy and robustness of indoor 3D object detection; the specific steps include: Step 1: Data preprocessing: First, load the RGB image, depth image, and annotation information from the specified path, and perform data augmentation operations, including random cropping and flipping. Then, convert the RGB image and depth map into tensors, convert the annotation information into bounding boxes and category labels, and finally normalize the data. Step 2: Multimodal feature extraction and fusion: The multimodal features of RGB images and depth images are efficiently extracted and fused using a hierarchical encoder architecture. The preprocessed data first passes through the Stem layer, which consists of two parallel dual 3×3 convolutional networks that perform convolution operations on RGB images and depth images respectively to achieve preliminary feature extraction and conversion. Then, the preliminary extracted features are input into the Transformer-based hierarchical encoder module, which captures and fuses complementary information from the two modalities at multiple scales through a global perception attention mechanism and a local enhancement attention mechanism, and deeply mines color, texture, distance, and geometric features. Step 3: Multi-scale feature aggregation: Multi-scale feature maps are aggregated through the feature pyramid network FPN to enhance the detection ability of the model at different resolutions and improve the detection accuracy of objects of different sizes. Step 4: 2D to 3D feature voxel projection; Combine the real depth information to project the aggregated 2D feature map into the 3D voxel space to construct an accurate 3D voxel representation; Use the camera's intrinsic parameters to convert the pixels in the 2D image and their depth information into 3D points in the camera coordinate system; Then, use the camera's extrinsic parameters, including the rotation matrix and translation vector, to convert these 3D points from the camera coordinate system to the world coordinate system; Finally, determine the specific position of each point in the voxel grid based on the predefined 3D space boundary and voxel size, and complete the 3D point mapping from the world coordinate system to the voxel grid, providing a basis for subsequent 3D feature extraction; Step 5: Extract local features of voxels. After projecting the 2D features into the 3D voxel space, the voxel feature encoding (VFE) network is used to extract local features of each voxel. The VFE network is used to extract meaningful local features from each voxel and obtain a feature representation containing spatial structure information. Specifically, the VFE network processes each voxel, aggregates the point cloud features within each voxel, and obtains a vector representation of the voxel. Step 6: 3D feature extraction and object detection: Use 3D convolutional neural network to further extract higher-level global features from voxel features; then, classify and locate each anchor point through the detection head, output the category probability and 3D bounding box parameters of each anchor point, and realize 3D object detection.

2. The target detection method based on multimodal data fusion and 3D voxel projection according to claim 1, characterized in that: After data preprocessing, first, the encoder is used to extract and fuse features of the multimodal data to obtain multi-scale features; then, FPN is used to aggregate the multi-scale features to obtain a 2D feature map; then, based on the real depth information, the 2D feature map is projected into the 3D voxel space to achieve feature lifting; further, the VFE network is used to extract voxel local features, and finally, the 3D convolutional network is used to extract global features to achieve the final target detection task.

3. The target detection method based on multimodal data fusion and 3D voxel projection according to claim 1, characterized in that: In the multimodal feature extraction and fusion module, the encoder network consists of multiple attention modules, each of which is equipped with a global perception attention module GAA and a local enhancement attention module LEA. The complementary features of the RGB image and the depth image are fused at different scales through the self-attention mechanism to generate a multi-scale feature map containing depth information; ①1①Global perception attention module GAA; GAA fuses depth and RGB features. GAA reduces the computational complexity by downsampling the query Q to a fixed size. Q, K, and V are obtained as follows: Among them, Pool k×k (·) is an adaptive average pooling operation, which is used to reduce the spatial dimension to k×k size, and linear is a linear transformation; The GAA process is expressed as follows: where UP(·) is a bilinear upsampling operation that converts the spatial size from k×k to h×w; ②2② Local Enhanced Attention Module LEA; As a supplement to the GAA module, the LEA module performs deep convolution on features through large-size convolution kernels and uses the obtained features as attention weights. The weights are then used to reweight the RGB features through simple Hadamard products. This allows 3D geometric information to be effectively integrated into RGB features, thereby enhancing the model's ability to capture local details. Through linear transformation and convolution operations, depth and RGB information are fused to improve the quality and accuracy of global feature representation. The local feature extraction process is expressed as follows: Among them, DConv k×k is a depthwise convolution kernel of size k×k, and ⊙ is the Hadamard product.

4. The target detection method based on multimodal data fusion and 3D voxel projection according to claim 1, characterized in that: In the 2D to 3D feature voxel projection process, the real depth information and the camera internal and external parameters are used to accurately map the pixel features on the 2D feature map to the corresponding 3D voxel grid. First, the camera internal parameters are used to convert the 2D image coordinates into three-dimensional coordinates under the camera coordinates. The formula is as follows: Where (u, v) is the coordinate in the 2D image, d is the corresponding depth value, and K is the intrinsic parameter matrix of the camera; Then, the camera external parameters are used to transform the three-dimensional points in the camera coordinate system to the world coordinate system. The formula is as follows: in, is the rotation matrix, T is the translation vector; According to the specific task and data set, set the boundary and voxel size s of the 3D space, and convert the 3D point (X w ,Y w ,Z w ) is mapped to discrete voxel coordinates (i, j, k) in the voxel grid as follows: Where (j,j,k) is the index in the 3D voxel grid, indicating the position of the point in the 3D voxel space; Aggregate the features of the 2D feature map into 3D voxels; for multiple features in the same voxel, use average aggregation or maximum pooling to process them, and the number of feature channels of each voxel is the same as that of the 2D feature map.

5. The target detection method based on multimodal data fusion and 3D voxel projection according to claim 1, characterized in that: The 3D convolutional neural network includes an encoder-decoder structure, the encoder includes three layers of upsampling blocks, each block consists of a transposed 3D convolutional layer with a stride of 2 and another 3D convolutional layer, which is used to extract hierarchical feature representations from 3D voxel volumes, and the decoder includes three layers of downsampling residual blocks, each block consists of three 3D convolutional layers, which are used to restore spatial resolution and generate fine feature maps for target detection.

6. The target detection method based on multimodal data fusion and 3D voxel projection according to claim 1, characterized in that: The multi-scale detection head consists of three 3D convolutional layers, which output the category probability and 3D bounding box parameters of the target during the detection process. At the same time, it outputs the centrality estimation value for evaluating the confidence of the center of the target in the detection box, thereby improving the accuracy of indoor 3D target detection.

Images