A target detection method based on multi-modal data fusion and 3D voxel projection

Abstract

The application discloses a target detection method based on multi-modal data fusion and 3D voxel projection, first, multi-modal features are extracted and fused from RGB images and depth images; then, a feature pyramid network is used to aggregate multi-scale features, and real depth information is used to accurately map 2D feature maps into 3D voxel space to form a 3D voxel volume; subsequently, local 3D features are extracted through a voxel feature coding network, and global features are further extracted through a 3D convolutional neural network. Finally, the above global features are input into a detection head to obtain the final target detection result. The model is trained on a large public dataset and can adapt to data with high feature dimension, large structural difference and uneven distribution. The fusion of multi-modal features can effectively improve the accuracy and robustness of 3D target detection and is suitable for object detection tasks in logistics sorting scenes.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and deep learning, and in particular to a target detection method based on multimodal data fusion and 3D voxel projection, which realizes object target detection in indoor environments. Background Art

[0002] In modern logistics and sorting scenarios, the application of automated and intelligent technologies is becoming increasingly prevalent. The ability to quickly and accurately detect and identify the three-dimensional size and category of various objects is crucial for improving sorting efficiency. This not only significantly reduces manual operation costs but also improves the overall efficiency and reliability of logistics systems. Therefore, developing efficient and robust 3D object detection methods has become a hot topic in current research.

[0003] With the increase in the types of sensors and the improvement in their performance, the data collection capabilities in logistics sorting scenarios have been significantly improved. Various high-precision and high-sensitivity sensors, such as lidar, depth cameras, and RGB cameras, provide rich and diverse three-dimensional spatial data, which not only contains geometric information such as the shape and size of the object, but also contains rich visual features such as color and texture, laying a solid foundation for accurate and efficient 3D target detection. In order to fully explore and utilize the complementary information in these multi-source data and improve the robustness and accuracy of target detection, data fusion technology has become a key link. Based on the role of the fusion results in the model, different fusion methods can be classified according to the fusion level, which can be mainly divided into four categories: data fusion, feature fusion, result fusion, and auxiliary estimation. (1) Data fusion refers to the fusion of data from different sources and the use of the fusion results as the input of the subsequent model. The subsequent model can be a complete independent model or a branch or component in the integrated model. The data to be fused can come from various sources, such as the original data collected by the sensor, the preprocessed data, and the output results of the sub-model in the integrated model, such as feature vectors, feature maps, or other calculation results. (2) Feature fusion usually refers to the fusion of feature maps or feature vectors from different inputs during the model calculation process, and the fusion result will be used as the output feature of the current model to participate in subsequent calculations. (3) Result fusion refers to the fusion of multiple output results as the final target output in the output stage of the model, or when the input data itself is the target output, the output result is corrected or optimized through fusion processing. Result fusion mainly includes two forms: one is the fusion of calculation results of parallel model branches; the other is that in the serial model, subsequent sub-models perform fusion optimization of the same target output based on the results of the previous stage. (4) Auxiliary estimation refers to predicting or estimating the data of another modality based on the data of a specific modality in order to optimize the data representation of the model. Unlike the above three fusion methods, the auxiliary estimation process does not reduce the modal types of the data, and the presence or absence of the auxiliary estimation module does not affect the integrity of the fusion model.

[0004] In logistics and sorting systems, the application of data fusion technology can significantly improve system accuracy and robustness. For example, fusing RGB images and depth maps can provide more comprehensive information for object detection, enabling the system to accurately identify and locate objects in complex environments. With the continuous advancement of sensor technology and data fusion techniques, 3D object detection technology has broad application prospects in modern logistics and sorting scenarios. Designing efficient and robust multimodal data fusion methods is of great significance for improving the intelligence level and work efficiency of logistics systems.

[0005] Existing technologies primarily rely on two types of data for 3D object detection: LiDAR point cloud data and RGB image data. Single-modality data has limitations in feature representation and object recognition. While RGB images provide rich color and texture information, they fall short in expressing depth and spatial relationships. Point cloud data provides accurate distance information, but lacks resolution and detail in complex scenes, and its high acquisition cost hinders its widespread application in a wider range of scenarios.

[0006] In recent years, researchers have tried to combine multimodal data to improve the performance of 3D target detection by fusing information from RGB images and depth maps. However, existing methods have limitations in feature extraction and fusion, making it difficult to fully utilize information from different modalities. Specific problems include: (1) The feature fusion method is single and cannot fully exploit the complementary information between multimodal data. (2) During the feature extraction process, features at different resolutions and scales cannot be effectively processed, resulting in insufficient detection accuracy. (3) 2D features are difficult to directly map to 3D space, and there is a lack of effective 2D to 3D feature conversion methods. The present invention proposes a target detection method based on multimodal data fusion and 3D voxel projection to solve the above problems. Summary of the Invention

[0007] The present invention adopts a multimodal data fusion method to fuse RGB information with depth information to improve the accuracy and robustness of indoor 3D target detection. Existing 3D target detection technologies mainly face the following problems: (1) The lack of depth information leads to low accuracy in target positioning and recognition, which limits the effective application of traditional 2D image processing technology in complex three-dimensional scenes. (2) Although point cloud data is rich in depth information, the high acquisition cost limits its popularity and application in a wide range of scenarios. (3) Target detection methods based on a single modality are difficult to fully capture the multi-dimensional information in the scene, affecting the accuracy and robustness of target detection.

[0008] In view of the above problems, this paper proposes a target detection method based on multimodal data fusion and 3D voxel projection. It performs indoor 3D target detection tasks based on large public datasets, fuses RGB images and depth image data, and combines real depth information to perform accurate 3D voxel projection to improve the accuracy and robustness of indoor 3D target detection. The model structure is as follows Figure 1 shown.

[0009] The specific process is as follows Figure 2 As shown, the following steps are included:

[0010] Step 1: Data preprocessing. Load the RGB image, depth map, and annotation information from the specified path, perform data augmentation operations such as random cropping and flipping on the data, convert the RGB image and depth map into tensors, convert the annotation information into bounding boxes and category labels, and finally normalize the data.

[0011] Step 2: Multimodal feature extraction and fusion. This module uses a layered encoder architecture to efficiently extract and deeply fuse multimodal features of RGB images and depth images.

[0012] The preprocessed data first passes through the Stem layer, which consists of two parallel dual 3×3 convolutional networks, performing convolution operations on the RGB image and the depth image, respectively, to achieve preliminary feature extraction and transformation. The initially extracted features are then input into the hierarchical encoder module. This module, based on the Transformer architecture, uses a global perception attention mechanism and a local enhancement attention mechanism to capture and fuse the complementary information from the two modalities at multiple scales. Specifically, it is represented as follows:

[0013] F Multi-Scale =Encoder(Stem RGB (I RGB ), Stem Depth (I Depth )) (1)

[0014] Among them, I RGB and I Depth Represents RGB image and depth image respectively. RGB and Stem Depth They are the Stem layers used to process RGB images and depth images respectively.

[0015] This module can deeply mine RGB images and depth images, extract color, texture features as well as distance and geometric features, and efficiently fuse them, providing rich and comprehensive feature representation for visual tasks.

[0016] Step 3: Multi-scale feature aggregation. Multi-scale feature maps are aggregated through the Feature Pyramid Network (FPN) to enhance the model's detection capabilities at different resolutions and improve the detection accuracy of objects of different sizes. The specific formula is as follows:

[0017] P i =Conv 1×1 (F i )+Upsample(P i+1 ) (2)

[0018] Among them, F i Represents feature maps of different scales, Pi Represents the feature map after FPN aggregation, Conv 1×1 Represents a 1×1 convolution operation, which is used to adjust the number of channels. Upsample represents an upsampling operation, which upsamples the high-level feature map to the same spatial size as the low-level feature map.

[0019] Step 4: 2D to 3D feature voxel projection. Combined with the real depth information, the aggregated 2D feature map is projected into the 3D voxel space to construct an accurate 3D voxel representation. Compared with other methods, this invention uses the real depth information obtained from the depth sensor to improve the accuracy of the 3D feature representation. First, the intrinsic parameters of the camera are used to convert the pixels in the 2D image and their depth information into three-dimensional points in the camera coordinate system. Then, the extrinsic parameters of the camera, including the rotation matrix and translation vector, are used to convert these three-dimensional points from the camera coordinate system to the world coordinate system. Finally, according to the predefined 3D space boundary and voxel size, the specific position of each point in the voxel grid is determined to complete the 3D point mapping from the world coordinate system to the voxel grid.

[0020] Step 5: Voxel local feature extraction. After projecting the 2D features into the 3D voxel space, the Voxel Feature Encoding Network (VFE) network is used to extract local features for each voxel. First, the depth information within the voxel is extracted, and the feature points in each voxel are encoded into a feature vector of fixed length. Next, the local spatial features within the voxel are extracted through the layer-by-layer feature aggregation mechanism in the VFE network to improve the accuracy of the voxel representation. The VFE network contains several fully connected layers and activation functions, which extract features from each point inside the voxel and aggregate these features to generate a local feature representation for each voxel. The VFE network not only retains the fine-grained information of the three-dimensional points within the voxel, but also can obtain local geometric shape features, providing richer information for subsequent 3D feature extraction and target detection.

[0021] Step 6: 3D feature extraction and object detection. A 3D convolutional neural network is used to further extract higher-level global features from the voxel features. The detection head then classifies and locates each anchor point, outputting the class probability and 3D bounding box parameters for each anchor point to achieve 3D object detection.

[0022] First, the voxel features V are input into a simplified 3D convolutional encoder-decoder network to extract global features. As shown below:

[0023] {F1, F2, F3} = Decoder(Encoder(V))(3) where V is the voxel feature, Encoder(·) represents the encoder, which includes three layers of downsampling residual blocks, each consisting of three 3D convolutional layers. Decoder(·) represents the decoder, which includes three layers of upsampling blocks, each consisting of a transposed 3D convolutional layer and another 3D convolutional layer.

[0024] Next, the obtained global features are fed into a multi-scale detection head, which consists of three 3D convolutional layers for classification, localization, and centrality estimation, respectively. These convolutional layers share weights across all scales. For each 3D position (x a ,y a , z a ) and three scale features, the detection head outputs the class probability p, centrality c and 3D bounding box parameters Δb. It is expressed as follows:

[0025] {p, c, Δb} = FCOSHead(F i ,(x a ,y a , z a ))(4) Among them, F i , i∈{1, 2, 3} represents the multi-scale feature map output from the encoder-decoder network, and FCOSHead(·) represents the FCOS detection head. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments or technical solutions of the present invention, the following briefly introduces the drawings required for use in the embodiments or description of the prior art.

[0027] Figure 1 Diagram of the target detection framework based on multimodal data fusion and 3D voxel projection.

[0028] Figure 2 Flowchart of multimodal indoor target detection.

[0029] Figure 3 Figure 2. Network structure diagram for voxel feature encoding. DETAILED DESCRIPTION

[0030] The specific implementation steps of the invention mainly include:

[0031] 1. Data Preprocessing

[0032] Loads RGB images, depth maps, and annotations from the specified path, performs data augmentation operations such as random cropping and flipping on the loaded data, converts the RGB images and depth maps into tensors, and converts the annotations into bounding boxes and category labels. Finally, normalizes the data.

[0033] 2. Multimodal feature extraction and fusion

[0034] The preprocessed data is used as input, and the Transformer architecture is used to capture and fuse multimodal features from the RGB image and depth map. The network uses a layered encoder to obtain low-resolution coarse-grained features and high-resolution fine features. Specifically, given an RGB image of spatial size H×W and the corresponding depth map, the RGB image and depth map are first processed using a parallel Stem layer consisting of two convolutional networks to obtain RGB and depth features, respectively. These RGB and depth features are then input into the layered encoder to obtain multi-scale RGB-D features. The specific representation is as follows:

[0035] F Multi-Scale =Encoder(Stem RGB (I RGB ), Stem Depth (I Depth ))(5) Among them, I RGB and I Depth Represents RGB image and depth image respectively. RGB and Stem Depth These are the Stem layers that process RGB images and depth images respectively.

[0036] Here, each encoder consists of multiple attention modules, each of which mainly includes a global awareness attention module (GAA) and a local enhancement attention module (LEA). The purpose of these modules is to capture and fuse RGB and depth features at different scales, improving the model's 3D perception and ability to capture local details.

[0037] (1) Global Perception Attention Module GAA

[0038] GAA fuses depth and RGB features, establishes feature associations across the entire scene, and enhances 3D perception, further facilitating the capture of semantic objects. Unlike the self-attention mechanism, which introduces pixels or increases the number of tokens, resulting in a quadratic increase in computational complexity, GAA reduces computational complexity by downsampling the query Q to a fixed size. The query Q is obtained by combining RGB and depth features. Adaptive pooling is performed to fix the spatial dimension to size k×k, and then linear transformation is performed to obtain:

[0039]

[0040] The key K and value V are both obtained by Perform feature transformation to obtain:

[0041]

[0042] Among them, Pool k×k (·) is an adaptive average pooling operation, which is used to reduce the spatial dimension to k×k size, and linear is a linear transformation.

[0043] After obtaining Q, K, and V, the GAA module calculates the dot product similarity of Q and K, normalizes it through Softmax, obtains the attention weight, and then performs a weighted product with V. Finally, the result is restored to its original size through a bilinear upsampling operation. The GAA process is expressed as follows:

[0044] in is the scaling factor for the number of depth channels, and UP(·) is a bilinear upsampling operation that converts the spatial size from k×k to h×w. In this way, the GAA module is able to establish a global association between RGB and depth features, capturing important semantic information in the scene while improving 3D perception.

[0045] (2) Local Enhanced Attention Module LEA

[0046] The role of the LEA module is to complement the GAA module and further enhance the expression of local features. Unlike the global attention of GAA, LEA focuses on enhancing local features. It can effectively aggregate adjacent pixel features with similar depth to enhance the expression of 3D geometric information in RGB features.

[0047] First, using the deep convolution kernel DConv k×k Deep features Perform convolution operations to generate local depth features. Through convolution operations, the correlation between adjacent pixels and local geometric information can be extracted:

[0048]

[0049] in, The deep features are linearly transformed, DConv k×k is a depthwise convolution kernel of size k×k.

[0050] Then, the local depth features obtained by deep convolution are used as weights, and the Hadamard product is used with the RGB features. Re-weighted, the enhanced RGB features are finally obtained:

[0051] Where ⊙ is the Hadamard product. It is a linear transformation of RGB features.

[0052] In this way, the LEA module can effectively aggregate local depth features and RGB features, so that the geometric relationship between adjacent pixels can be integrated into the RGB features, thereby improving the model's ability to capture local details.

[0053] 3. Multi-scale feature aggregation

[0054] Aggregate the multi-scale feature maps output from step 2 through FPN. The output feature maps are called F1, F2, and F3. Perform a 1×1 convolution operation on the highest-level feature map F3 to obtain the multi-scale feature map P3. Next, perform the following processing on each layer of feature maps from high to low levels:

[0055] P2=Conv 1×1 (F2)+UPsample(P3)(12)

[0056] P1=Conv 1×1 (F1)+UPsample(P2)(13) In each step, a 1×1 convolution operation is performed on the feature map of the current layer to adjust the number of channels, and the feature map of the previous layer is upsampled and added to the feature map of the current layer.

[0057] Upsampling resizes the spatial size of high-level feature maps to the same size as low-level feature maps. The resulting P1, P2, and P3 are multi-scale feature maps after FPN aggregation. They contain rich contextual information and detailed features, enhancing the model's ability to detect objects of different sizes.

[0058] 4.2D to 3D Feature Voxel Projection

[0059] The 2D feature map obtained in step 3 is projected into the 3D voxel space to construct the 3D voxel volume, laying the foundation for 3D object detection.

[0060] First, use the camera intrinsic parameters to convert the 2D image coordinates into 3D coordinates under the camera coordinates. The formula is as follows:

[0061] Among them, (u, v) is the coordinate in the 2D image, d is the corresponding depth value, and K is the intrinsic parameter matrix of the camera.

[0062] Then, using the camera extrinsic parameters, the 3D points in the camera coordinate system are converted to the world coordinate system, achieving accurate mapping from the pixel points in the 2D image to the 3D points in the world coordinate system. The formula is as follows: in, is the rotation matrix and T is the translation vector.

[0063] According to the requirements of the task and dataset, the boundary of the 3D space is set to [xmin ,x max ]、[y min ,y max ]、[z min ,z max ], set the size of each voxel to s. Next, the 3D point in the world coordinate (X w ,Y w ,Z w ) is mapped to discrete voxel coordinates (i, j, k) in the voxel grid as follows:

[0064]

[0065] Where (i, j, k) is the index in the 3D voxel grid, indicating the position of the point in the 3D voxel space.

[0066] Next, the features of the 2D feature map are aggregated into 3D voxels. The number of feature channels per voxel is the same as that of the 2D feature map, and multiple features within the same voxel can be processed through methods such as average aggregation and maximum pooling.

[0067] 5. Voxel local feature extraction

[0068] The VFE network is used to extract meaningful local features from each voxel, thereby obtaining a feature representation that contains spatial structure information. Specifically, the VFE network processes all points of each voxel in parallel and obtains the local feature representation of the voxel through feature aggregation.

[0069] First, the VFE network aggregates features of all points within a 3D voxel and obtains a unified voxel feature vector through pooling. Next, the VFE network further encodes the features of each voxel to capture local geometric and semantic information. Furthermore, the voxel's position in 3D space is crucial. The VFE network encodes the voxel's 3D coordinates into a vector of the same dimension as the feature vector and concatenates it with the voxel's semantic feature vector. This allows the network to capture position-related information, helping the model distinguish similar objects at different locations. To enhance the network's nonlinear representation capabilities, nonlinear factors are introduced after each convolutional and fully connected layer in the VFE network. Nonlinear activation functions (such as ReLU) are used to enable the network to learn more complex feature representations. After these steps, the VFE network outputs a local feature vector for each voxel. These local feature vectors not only contain the semantic information within the voxel but also incorporate the voxel's position in 3D space, providing rich and comprehensive information for subsequent global feature extraction and object detection.

[0070] 6.3D Feature Extraction and Object Detection

[0071] A 3D convolutional neural network is used to further extract global features from voxel features to form a 3D feature volume. Subsequently, the detection head classifies and locates each anchor point, outputting the class probability and 3D bounding box parameters of each anchor point to achieve 3D object detection.

[0072] First, the voxel features V are input into a simplified 3D convolutional encoder-decoder network to extract global features. As shown below:

[0073] {F1, F2, F3} = Decoder(Encoder(V))(17) where Decoder(·) represents the decoder, consisting of three layers of upsampling blocks, each consisting of a transposed 3D convolutional layer with a stride of 2 and another 3D convolutional layer, V is the input voxel feature, and Encoder(·) represents the encoder part, consisting of three layers of downsampling residual blocks, each consisting of three 3D convolutional layers.

[0074] Subsequently, the obtained global features are input into an improved FCOS detection head. The detection head consists of three 3D convolutional layers for classification, localization and centrality estimation, which share weights at all scales. For each 3D location (x a ,y a , z a ) and three scale features, the detection head outputs the class probability p, centrality c and 3D bounding box parameters Δb. It is expressed as follows:

[0075] {p, c, Δb} = FCOSHead(F i ,(x a ,y a , z a ))(18) Among them, F i , i∈{1, 2, 3} denotes the multi-scale feature maps output from the encoder-decoder network. FCOSHead(·) denotes the improved FCOS detection head.

Claims

1. A target detection method based on multimodal data fusion and 3D voxel projection, characterized by: This paper conducts indoor 3D object detection based on a large public dataset, fuses RGB images with depth image data, and combines real depth information for precise 3D voxel projection to improve the accuracy and robustness of indoor 3D object detection. The specific steps include: Step 1: Data preprocessing: First, load the RGB image, depth image, and annotation information from the specified path and perform data augmentation operations, including random cropping and flipping. Then, convert the RGB image and depth map into tensors, convert the annotation information into bounding boxes and category labels, and finally normalize the data. Step 2: Multimodal feature extraction and fusion: A hierarchical encoder architecture is used to efficiently extract and fuse multimodal features from RGB and depth images. The preprocessed data first passes through the Stem layer, which consists of two parallel dual 3×3 convolutional networks that perform convolution operations on the RGB and depth images, respectively, to achieve preliminary feature extraction and transformation. The initially extracted features are then input into a Transformer-based hierarchical encoder module. Through a global perception attention mechanism and a local enhancement attention mechanism, the module captures and fuses complementary information from the two modalities at multiple scales, deeply exploring color, texture, distance, and geometric features. Step 3: Multi-scale feature aggregation: Multi-scale feature maps are aggregated through the feature pyramid network (FPN) to enhance the model's detection capabilities at different resolutions and improve the detection accuracy of objects of different sizes. Step 4: 2D to 3D feature voxel projection: Combine the aggregated 2D feature map with the real depth information and project it into the 3D voxel space to construct an accurate 3D voxel representation. Use the camera's intrinsic parameters to convert the pixels and their depth information in the 2D image into 3D points in the camera coordinate system. Then, use the camera's extrinsic parameters, including the rotation matrix and translation vector, to transform these 3D points from the camera coordinate system to the world coordinate system. Finally, determine the specific position of each point in the voxel grid based on the predefined 3D space boundary and voxel size, and complete the 3D point mapping from the world coordinate system to the voxel grid, providing a basis for subsequent 3D feature extraction. Step 5: Voxel local feature extraction: After projecting the 2D features into the 3D voxel space, the voxel feature encoding (VFE) network is used to extract local features from each voxel. The VFE network is used to extract meaningful local features from each voxel and obtain a feature representation that contains spatial structure information. Specifically, the VFE network processes each voxel and aggregates the point cloud features within each voxel to obtain a vector representation of the voxel. Step 6: 3D feature extraction and object detection: Use a 3D convolutional neural network to further extract higher-level global features from the voxel features. Subsequently, the detection head classifies and locates each anchor point, outputting the category probability and 3D bounding box parameters of each anchor point to achieve 3D object detection.

2. The target detection method based on multimodal data fusion and 3D voxel projection according to claim 1, characterized in that: After data preprocessing, the encoder is first used to extract and fuse features from the multimodal data to obtain multi-scale features. Subsequently, FPN is applied to aggregate the multi-scale features to obtain a 2D feature map. Next, the 2D feature map is projected into the 3D voxel space based on the real depth information to achieve feature lifting. Furthermore, the VFE network is used to extract local voxel features. Finally, a 3D convolutional network is used to extract global features to achieve the final target detection task.

3. The target detection method based on multimodal data fusion and 3D voxel projection according to claim 1, characterized in that: In the multimodal feature extraction and fusion module, the encoder network consists of multiple attention modules, each of which is equipped with a global perception attention module GAA and a local enhancement attention module LEA. The self-attention mechanism is used to fuse the complementary features of the RGB image and the depth image at different scales to generate a multi-scale feature map containing depth information. ①1①Global perception attention module GAA; GAA fuses depth and RGB features. GAA reduces computational complexity by downsampling the query Q to a fixed size. Q, K, and V are obtained as follows: Among them, Pool k×k (·) is an adaptive average pooling operation, which is used to reduce the spatial dimension to k×k size, and linear is a linear transformation; The GAA process is expressed as follows: where UP(·) is a bilinear upsampling operation that converts the spatial size from k×k to h×w; ②2② Local Enhanced Attention Module LEA; As a complement to the GAA module, the LEA module performs deep convolution on features using large-scale convolution kernels and uses the resulting features as attention weights. The weights are then used to reweight the RGB features through a simple Hadamard product. This effectively integrates 3D geometric information into the RGB features, thereby enhancing the model's ability to capture local details. Through linear transformations and convolution operations, it fuses depth and RGB information, improving the quality and accuracy of global feature representation. The local feature extraction process is expressed as follows: Among them, DConv k×k is a depthwise convolution kernel of size k×k, and ⊙ is the Hadamard product.

4. The target detection method based on multimodal data fusion and 3D voxel projection according to claim 1, wherein: In the 2D-to-3D feature voxel projection process, the real depth information and camera intrinsic and extrinsic parameters are used to accurately map the pixel features on the 2D feature map to the corresponding 3D voxel grid. First, the camera intrinsic parameters are used to convert the 2D image coordinates into 3D coordinates under the camera coordinates. The formula is as follows: Where (u, v) is the coordinate in the 2D image, d is the corresponding depth value, and K is the intrinsic parameter matrix of the camera; Then, using the camera external parameters, the three-dimensional point in the camera coordinate system is converted to the world coordinate system. The formula is as follows: in, is the rotation matrix, T is the translation vector; According to the specific task and data set, set the boundary and voxel size s of the 3D space, and convert the 3D point (X w ,Y w ,Z w ) is mapped to discrete voxel coordinates (i, j, k) in the voxel grid as follows: Where (j,j,k) is the index in the 3D voxel grid, indicating the position of the point in the 3D voxel space; Aggregate the features of the 2D feature map into 3D voxels; for multiple features in the same voxel, use average aggregation or maximum pooling to process them, and the number of feature channels of each voxel is the same as that of the 2D feature map.

5. The target detection method based on multimodal data fusion and 3D voxel projection according to claim 1, wherein: The 3D convolutional neural network includes an encoder-decoder structure, where the encoder includes three layers of upsampling blocks, each block consists of a transposed 3D convolutional layer with a stride of 2 and another 3D convolutional layer, which is used to extract hierarchical feature representations from 3D voxel volumes, and the decoder includes three layers of downsampling residual blocks, each block consists of three 3D convolutional layers, which is used to restore spatial resolution and generate fine feature maps for target detection.

6. The target detection method based on multimodal data fusion and 3D voxel projection according to claim 1, wherein: The multi-scale detection head consists of three 3D convolutional layers. During the detection process, it outputs the target's category probability and 3D bounding box parameters. At the same time, it outputs a centrality estimate to assess the confidence of the target center within the detection box, thereby improving the accuracy of indoor 3D target detection.

Images