3D semantic segmentation method based on coupling of voxel model and color information

By combining voxel model and color information, using the fusion method of three-dimensional CNN network and two-dimensional convolutional neural network, the problem of lack of depth information and three-dimensional point cloud noise in the existing technology is solved, and the end-to-end three-dimensional semantics are realized Segmentation improves the semantic segmentation effect of the three-dimensional reconstruction model and provides more comprehensive interactive support for VR and AR applications.

CN110827295BActive Publication Date: 2024-11-05QINGDAO RES INST OF BEIHANG UNIV +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201911051686.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-10-31
Publication Date
2024-11-05
Estimated Expiration
2039-10-31

AI Technical Summary

Technical Problem

In the prior art, the two-dimensional RGB image segmentation method lacks depth information and is susceptible to light. The three-dimensional point cloud semantic segmentation has noise and rotation uncertainty, which cannot meet multi-level virtual reality (VR) and augmented reality (AR) applications. Required, the lack of semantic information of three-dimensional reconstruction technology leads to limited user interaction.

Method used

Using a three-dimensional semantic segmentation method based on the coupling of voxel model and color information, TSDF voxel information is extracted through a three-dimensional CNN network, combined with RGB images, and using the attention mechanism to fuse two-dimensional and three-dimensional channel information to achieve end-to-end three-dimensional semantic segmentation.

Benefits of technology

It improves the semantic segmentation effect of the three-dimensional reconstruction model, provides more comprehensive scene interaction support, provides voxel-level labels for VR and AR applications, and enhances the interaction ability between users and the environment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN110827295B_ABST
    Figure CN110827295B_ABST
Patent Text Reader

Abstract

The three-dimensional semantic segmentation method based on the coupling of voxel model and color information described in the present invention extracts the three-dimensional TSDF voxel information by the three-dimensional CNN network and combines it with the RGB image collected during the three-dimensional reconstruction process, and directly realizes end-to-end three-dimensional semantic segmentation through the channel information fusion strategy, thereby providing strong support for related AR applications after three-dimensional reconstruction technology. The three-dimensional semantic segmentation method includes the following implementation stages: 1) two-dimensional RGB image processing stage; 2) three-dimensional TSDF voxel model processing stage; 3) dual-channel information fusion and three-dimensional information processing stage. The two-dimensional and three-dimensional information are fused; the final feature extraction is performed using a three-dimensional convolutional neural network to obtain the final voxel-level semantic segmentation result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a novel three-dimensional semantic segmentation method applied in the field of computer vision, belonging to the technical field of virtual reality. Background Art

[0002] Currently, the semantic segmentation methods implemented using deep learning are mainly divided into two-dimensional segmentation based on RGB images and semantic segmentation based on three-dimensional models for point clouds.

[0003] The accuracy of two-dimensional RGB image processing is relatively high, but due to the uncertainty of the scale of the two-dimensional monocular camera, the two-dimensional image inherently lacks the depth information of the relevant objects, and the two-dimensional RGB image is easily affected by factors such as lighting conditions, which makes the overall segmentation method have obvious limitations and it is difficult to fully utilize all aspects of the image feature information.

[0004] The currently popular three-dimensional semantic segmentation methods have the disadvantages of large noise during point cloud processing, disorder of point clouds, and rotation uncertainty. Although the related point cloud semantic segmentation methods can utilize spatial depth information to a certain extent, the semantic segmentation effect is still unsatisfactory, and cannot meet the needs of multi-level VR and AR applications, and cannot provide sufficient scene interaction support for VR and AR devices.

[0005] The currently more common 3D reconstruction technology based on depth cameras only generates a mesh model of the scene after 3D reconstruction, but lacks the corresponding scene semantic information, which limits the interaction between users and the scene. At the same time, the scene model generated in the 3D reconstruction technology is stored in the TSDF voxel model. The lack of such voxel-level semantic segmentation methods directly causes semantic segmentation obstacles in existing 3D reconstruction models.

[0006] In view of this, this patent application is proposed. Summary of the invention

[0007] The three-dimensional semantic segmentation method based on the coupling of voxel model and color information described in the present invention aims to solve the problems existing in the above-mentioned prior art and extract three-dimensional TSDF voxel information based on a three-dimensional CNN network, and combine it with the RGB image collected during the three-dimensional reconstruction process. End-to-end three-dimensional semantic segmentation is directly realized through a channel information fusion strategy, thereby providing strong support for related AR applications after three-dimensional reconstruction technology.

[0008] To achieve the above-mentioned invention purpose, the three-dimensional semantic segmentation method described in this application mainly adopts the method of coupling a three-dimensional convolutional neural network with a two-dimensional convolutional neural network, wherein the three-dimensional TSDF voxel information is processed by the three-dimensional convolutional neural network, and the two-dimensional RGB image information is processed by the two-dimensional convolutional neural network, and the information of the two-dimensional and three-dimensional channels is fused and effective features are extracted by using the attention mechanism. Moreover, in the two-dimensional channel, the RGB features under multiple perspectives are weightedly fused to improve the utilization rate of multi-level information.

[0009] The 3D semantic segmentation method based on the coupling of voxel model and color information mainly includes the following implementation stages:

[0010] 1) Two-dimensional RGB image processing stage

[0011] For RGB images under multiple perspectives, the RGB images under multiple perspectives are sent to the backbone network to extract the two-dimensional feature information of the high-dimensional image;

[0012] Get the camera posture information mapped from the current frame to the world coordinate system;

[0013] The RGB channel selection module is used to assign different weights to the RGB two-dimensional features under different viewing angles;

[0014] 2) 3D TSDF voxel model processing stage

[0015] First, the front-end 3D model is reconstructed;

[0016] Secondly, the signed distance function is used to calculate the entire TSDF voxel model;

[0017] Then, the residual module is used to preliminarily extract 3D features;

[0018] The processed TSDF voxel model data and the features extracted by the RGB channel selection module are processed by the residual module to preliminarily extract the relevant three-dimensional features;

[0019] 3) Dual-channel information fusion and three-dimensional information processing stage

[0020] Fusion of two-dimensional and three-dimensional information;

[0021] A three-dimensional convolutional neural network is used for the final feature extraction to obtain the final voxel-level semantic segmentation result.

[0022] Furthermore, in the two-dimensional RGB image processing stage, in obtaining the camera posture information of the current frame mapped to the world coordinate system, three-dimensional reconstruction or SFM technology is used to perform calculations through a multi-view geometry method.

[0023] In the RGB channel selection module, first, the input features are globally pooled; then, the results of global pooling are further extracted using a 1*1 convolutional layer; then, batch normalization is used to normalize the relevant distribution of the features; finally, the weights of the RGB features of different viewing angles extracted by the module are obtained through the sigmoid activation function, and a dot multiplication operation is performed with the input features to obtain the weighted result of the input features.

[0024] Furthermore, in the three-dimensional TSDF voxel model processing stage, during the front-end three-dimensional reconstruction process, the depth information in the input RGB image is integrated into the TSDF model as the input of the three-dimensional network adopted in this application.

[0025] In the residual module, first, the input features are reduced in dimension through a 1*1 convolution layer to reduce the number of parameters of the entire model and accelerate model training. After being activated by the ReLU activation function, they are sent to a 3*3 convolution layer for feature extraction. Then, the 1*1 convolution layer is used to increase the dimension and restore it to the original dimension. Finally, the input features are added to the features after dimensionality increase to obtain the features processed by the residual module.

[0026] Furthermore, in the dual-channel information fusion and three-dimensional information processing stage, in the dual-channel information fusion module, first, the RGB features and voxel features are superimposed together; then, the commonly used 3*3 convolution layer + bacthnormalization + ReLU activation function is used to perform preliminary information extraction on the superimposed information; thereafter, global pooling is performed to extract global features, and further processing is performed through a 1*1 convolution layer + ReLU activation function + 1*1 convolution layer; finally, the sigmoid activation function is used to obtain the corresponding weights of each feature channel, and the weighted features are obtained by multiplying the feature points processed by the 3*3 convolution layer, and then added to the unweighted features to obtain the feature information extracted after the dual-channel information fusion.

[0027] In summary, the three-dimensional semantic segmentation method described in this application has the advantages of: subsequent targeted processing is performed on the three-dimensional reconstruction results, and end-to-end three-dimensional semantic segmentation is achieved by utilizing the color map, depth map, camera posture and final model results in the three-dimensional reconstruction process. Through channel information fusion, a channel selection module designed using two-dimensional and three-dimensional multi-dimensional information and attention mechanism is used to add different weights as effective, direct and comprehensive guidance for the final semantic segmentation results. Finally, voxel-level labels are provided for the three-dimensional reconstructed model after processing, which can classify each position in the space, so that not only the indoor scene model can be obtained, but also what object each part of the model is, which provides convenience for subsequent related AR interactive applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flowchart of the three-dimensional semantic segmentation method;

[0029] Figure 2 This is a schematic diagram of the RGB channel selection module;

[0030] Figure 3 It is a schematic diagram of the front-end 3D reconstruction process;

[0031] Figure 4 It is a schematic diagram of the two-dimensional TSDF model;

[0032] Figure 5 is a schematic diagram of the residual module;

[0033] Figure 6 It is a schematic diagram of a dual-channel information fusion module;

[0034] Figure 7 It is a schematic diagram of a three-dimensional model of the interior scene of a bedroom;

[0035] Figure 8 This is a schematic diagram of the semantic segmentation results of the bedroom interior scene;

[0036] Fig. 9 It is a schematic diagram of a three-dimensional model of the interior scene of a living room;

[0037] Fig.10 This is a schematic diagram of the semantic segmentation results of the interior scene of the living room. DETAILED DESCRIPTION

[0038] The present invention is further described in detail below with reference to the accompanying drawings and implementation examples.

[0039] The three-dimensional semantic segmentation method based on the coupling of voxel model and color information described in this application uses a three-dimensional convolutional neural network to extract TSDF voxel information, which can better identify and segment objects in the scene. However, since the data in the TSDF model obtained after three-dimensional reconstruction is a sparse structure, it is not conducive to the learning of the three-dimensional convolutional neural network. Therefore, the signed distance function is extended to obtain the value of each pixel to improve the extraction efficiency of the three-dimensional convolutional neural network for the voxel model, and finally a model result in which each voxel has a semantic label is obtained, thereby realizing a scene interaction with a greater degree of freedom for the user and the environment.

[0040] like Figure 1 The 3D semantic segmentation method shown in the figure mainly includes the following implementation stages:

[0041] 1. Two-dimensional RGB image processing stage

[0042] For RGB images under multiple perspectives, a variety of network structure designs can be adopted to efficiently extract image features, such as VGG network, ResNet network, MobileNet series network, ShuffleNet series network and Xception network.

[0043] First, the RGB images from multiple perspectives are fed into these mature backbone networks to quickly extract their relevant high-dimensional image features and provide effective two-dimensional feature information for the overall semantic segmentation network.

[0044] Secondly, using 3D reconstruction or SFM technology, the camera posture information of the current frame in the world coordinate system is calculated through multi-view geometry methods, and a projection of the current frame mapped to the world coordinate system is established.

[0045] Finally, the RGB channel selection module is used to assign different weights to the RGB two-dimensional features under different viewing angles to better support the optimization direction of the convolutional neural network, obtain more accurate results and accelerate the subsequent iteration process.

[0046] like Figure 2 Schematic diagram of the RGB channel selection module shown.

[0047] In the RGB channel selection module, first, the input features are globally pooled; then, the results of global pooling are further extracted using a 1*1 convolutional layer; after that, batch normalization is used to normalize the relevant distribution of the features; finally, the weights of the RGB features of different perspectives extracted by this module are obtained through the sigmoid activation function, and the dot multiplication operation is performed with the input features to obtain the weighted result of the input features.

[0048] 2. 3D TSDF voxel model processing stage

[0049] First, the front end is 3D reconstructed.

[0050] like Figure 3 The front-end 3D reconstruction process diagram shown in the figure integrates the depth information in the input RGB image into the TSDF model as the input of the 3D network used in this application. That is, the RGB image is first used as the input of the 2D network to calculate the camera pose of each frame, and then the mapping transformation is performed from the 2D network to the 3D network.

[0051] Secondly, the signed distance function is used to calculate the entire TSDF voxel model.

[0052] Since the TSDF model obtained after 3D reconstruction is a truncated signed distance function and has the characteristics of a sparse structure in space, the signed distance function is used to calculate the entire voxel model to obtain dense data in the entire space, thereby facilitating the application of 3D convolutional neural networks.

[0053] like Figure 4 What is shown is a two-dimensional TSDF model, and the three-dimensional network input is the three-dimensional TSDF model obtained after the three-dimensional reconstruction is completed. The values ​​in the TSDF model grid represent the distance from the reconstructed scene surface, and the crossing points from positive to negative in the grid represent the reconstructed surface.

[0054] Then, a residual module is used to preliminarily extract 3D features.

[0055] The processed TSDF voxel model data and the features extracted by the RGB channel selection module are both processed by the residual module to preliminarily extract their related three-dimensional features.

[0056] like Figure 5 Schematic diagram of the residual module shown.

[0057] First, the residual module reduces the dimension of the input features through a 1*1 convolution layer to reduce the number of parameters of the entire model and accelerate model training. After being activated by the ReLU activation function, it is sent to a 3*3 convolution layer for feature extraction.

[0058] Then, a 1*1 convolutional layer is used to restore the original dimension.

[0059] Finally, the input features are added to the features after dimensionality increase to obtain the features processed by the residual module.

[0060] Since the residual module introduces a high-speed path, the problem of gradient vanishing when the network is deepened is solved, and the final result of the end-to-end network structure training involved in the method described in the present application can be guaranteed to effectively converge to the vicinity of a reliable point.

[0061] 3. Dual-channel information fusion and three-dimensional information processing stage

[0062] Use Figure 6 The dual-channel information fusion module shown in FIG. 1 effectively fuses the above two levels of information; after the dual-channel information is fused, a three-dimensional convolutional neural network is used to perform the final feature extraction to obtain the final voxel-level semantic segmentation result.

[0063] In the dual-channel information fusion module, first, the RGB features and voxel features are superimposed together; then, the commonly used 3*3 convolution layer + batch normalization + ReLU activation function is used to perform preliminary information extraction on the superimposed information; after that, global pooling is performed to extract global features, and further processed by 1*1 convolution layer + ReLU activation function + 1*1 convolution layer; finally, the sigmoid activation function is used to obtain the corresponding weights of each feature channel, and the weighted features are obtained by multiplying the feature points processed by the 3*3 convolution layer, and then added to the unweighted features to obtain the feature information extracted after dual-channel information fusion.

[0064] The three-dimensional convolutional neural network after dual-channel information fusion has too many parameters and has previously extracted more useful spatial feature information. Therefore, the three-dimensional convolutional neural network used in this application has the structural characteristics of 1*1 convolution dimensionality reduction, 3*3 convolution feature extraction, and 1*1 convolution dimensionality increase. The above two stacked modules can still perform voxel-level semantic segmentation operations well.

[0065] 4. About loss function

[0066] The segmentation method described in this application adopts the following cross entropy commonly used in semantic segmentation networks:

[0067]

[0068] Among them, y i is the expected output, y i_ is the actual output. Since our output is a one-hot label, only the correct expected output is 1, and the others are 0. The loss can be simplified to:

[0069] loss i = -log(y i_ )

[0070] The probability distribution is calculated by the cross entropy mentioned above to calculate the distance between the probability distribution of the predicted label and the probability distribution of the true label, thereby obtaining the loss function of semantic segmentation. Using the Adam algorithm for gradient descent can reduce the overall network loss, thereby obtaining the overall parameters of the network model.

[0071] The following specific segmentation examples are used for a more vivid comparison and explanation.

[0072] like Figure 7 The schematic diagram of the three-dimensional model of the bedroom interior scene is shown in FIG. Fig. 9 Schematic diagram of the three-dimensional model of the living room interior scene.

[0073] Firstly, the TOF camera is used to complete the model reconstruction and texture mapping for indoor scenes using the TSDF-based 3D reconstruction method.

[0074] Then, the reconstructed TSDF model, the camera posture during reconstruction, the depth map and the color map are sent to the deep learning network described in this application, and the semantic information of each voxel level in the TSDF model can be immediately obtained.

[0075] like Figure 8 The schematic diagram of the semantic segmentation result of the bedroom interior scene is shown in Fig.10 The schematic diagram of the semantic segmentation results of the interior scene of the living room shown can clearly identify which part of the model belongs to the sofa in the bedroom, which part belongs to the TV in the bedroom, and so on.

[0076] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A 3D semantic segmentation method based on coupling of voxel model and color information, characterized in that: The implementation phases include: 1) Two-dimensional RGB image processing stage For RGB images under multiple perspectives, the RGB images under multiple perspectives are sent to the backbone network to extract the two-dimensional feature information of the high-dimensional image; Get the camera posture information mapped from the current frame to the world coordinate system; The RGB channel selection module is used to assign different weights to the RGB two-dimensional features under different viewing angles; 2) 3D TSDF voxel model processing stage First, the front-end 3D model is reconstructed to integrate the depth information in the input RGB image into the TSDF model; Secondly, the signed distance function is used to calculate the entire TSDF voxel model; Then, a 3D convolutional neural network is used to preliminarily extract 3D features; The processed TSDF voxel model data and the features extracted by the RGB channel selection module are processed by the residual module to preliminarily extract their related features; 3) Dual-channel information fusion and three-dimensional information processing stage Fusion of two-dimensional and three-dimensional information; Use a 3D convolutional neural network to perform the final feature extraction, thereby obtaining the final voxel-level semantic segmentation result; In the two-dimensional RGB image processing stage, the camera posture information of the current frame mapped to the world coordinate system is obtained, and the three-dimensional reconstruction or SFM technology is used to perform calculations through a multi-view geometry method; In the two-dimensional RGB image processing stage, in the RGB channel selection module, first, the input features are globally pooled; Then, the result after global pooling is further extracted using a 1*1 convolutional layer; after that, batch normalization is used to normalize the relevant distribution of the features; finally, the weights of the RGB features of different perspectives extracted by this module are obtained through the sigmoid activation function, and the dot multiplication operation is performed with the input features to obtain the weighted result of the input features.

2. The three-dimensional semantic segmentation method based on the coupling of voxel model and color information according to claim 1, characterized in that: In the three-dimensional TSDF voxel model processing stage, first, the residual module reduces the dimension of the input features through a 1*1 convolution layer to reduce the number of parameters of the entire model and accelerate model training, and then sends them to a 3*3 convolution layer for feature extraction after being activated by a ReLU activation function; then, the 1*1 convolution layer is used to increase the dimension and restore it to the original dimension; finally, the input features are added to the features after dimensionality increase to obtain the features processed by the residual module.

3. The three-dimensional semantic segmentation method based on the coupling of voxel model and color information according to claim 1, characterized in that: In the dual-channel information fusion and three-dimensional information processing stage, in the dual-channel information fusion module, first, the RGB features and the voxel features are superimposed together; then, the commonly used 3*3 convolution layer + batch normalization + ReLU activation function is used to perform preliminary information extraction on the superimposed information; then, global pooling is performed to extract global features, and further processing is performed through a 1*1 convolution layer + ReLU activation function + 1*1 convolution layer; finally, the sigmoid activation function is used to obtain the corresponding weights of each feature channel, and the weighted features are obtained by multiplying the feature points processed by the 3*3 convolution layer, and then the weighted features are added to the unweighted features to obtain the feature information extracted after the dual-channel information fusion.