Scene object classification method and system based on multi-view depth image

Abstract

The invention discloses a scene object classification method and system based on a multi-view depth image. Comprising the following steps: deploying three depth cameras, recording coordinates of the cameras, and shooting depth images; generating point cloud data corresponding to the depth image by using the depth image and the camera coordinates, aligning and fusing the point cloud data of the multiple visual angles to generate a synthesized multi-visual-angle point cloud, and obtaining a two-dimensional image by using the depth image; extracting three-dimensional features from the multi-view point cloud, extracting two-dimensional features from the two-dimensional image, mapping the two-dimensional features to the three-dimensional features, and aligning and splicing the three-dimensional features and the mapped two-dimensional features to obtain features of the multi-view point cloud; and inputting the multi-view point cloud and the features thereof into a decoder for semantic segmentation to obtain a classification result of the objects in the scene. According to the method, the advantages of two point cloud generation methods of a multi-view image and a depth camera can be combined, the point cloud data quality is improved, information of two-dimensional features and three-dimensional features is fully utilized, and the semantic segmentation effect is improved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and semantic segmentation, and in particular to a method and system for classifying scene objects based on multi-view depth images. Background Art

[0002] With the widespread application of computer vision in the field of technology, its importance to contemporary economic development and scientific and technological progress has become increasingly prominent. In the field of computer vision, three-dimensional vision makes it possible for computers to perceive, understand and analyze three-dimensional space, and point cloud is one of the commonly used three-dimensional visual expressions.

[0003] Compared with other 3D visual representations, point cloud only stores key points on the surface of objects, which reduces the storage requirements and transmission volume of point cloud data, helps save storage space and computing resources, and its corresponding processing and analysis algorithms are usually more efficient. These advantages make point cloud data occupy an increasingly important position in 3D visual processing. However, in the actual application of processing point cloud data, there are two key issues, namely the acquisition of point cloud data and the feature extraction and downstream tasks of point cloud data.

[0004] When acquiring point cloud data, 3D scanners are often used. For outdoor scenes or autonomous driving scenes, lidar is generally used for acquisition; while for indoor scenes, the most commonly used methods are multi-view image generation or depth camera generation. The point cloud generation methods of multi-view image and depth camera each have their own advantages.

[0005] In terms of feature extraction and downstream tasks of point clouds, the main extraction methods currently used are point-based deep learning methods represented by PointNet, which have achieved relatively good results in tasks such as point cloud segmentation and point cloud classification. In recent years, in order to further improve the effect of feature extraction, multimodal methods have been introduced, such as multimodal learning of images and point clouds and multimodal learning of text and point clouds. Multimodal learning methods for images and point clouds include projection-based methods, attention-based methods, and cross-modal Transformer-based methods.

[0006] One of the current existing technologies is a classic method for point cloud multimodal target detection applied to radar in the paper "Pointpainting: Sequential fusion for 3dobject detection". In this method, the image is sent to the segmentation module to generate a two-dimensional segmentation score, and the intensity measurement of each point is merged with its corresponding semantics, and then the two-dimensional segmentation score is mapped to the semantic label in the three-dimensional space. This method is a common method for image-point cloud multimodal learning based on projection. The disadvantage of this method is that the PointPainting method is used in combination with radar for outdoor autonomous driving scenarios. Such a usage scenario determines that it only focuses on the information of objects facing the direction of the camera; at the same time, the PointPainting method does not realize the feature combination of the three-dimensional semantic segmentation method and the two-dimensional semantic segmentation method, but directly maps the two-dimensional segmentation label to the three-dimensional segmentation, and fails to give play to the advantages of the three-dimensional semantic segmentation model.

[0007] The second existing technology is a cross-modal unsupervised learning method for three-dimensional point clouds in the paper "PointDC: Unsupervised Semantic Segmentation of Three-Dimensional Point Clouds via Cross-modal Distillation and Super-Voxel Clustering". This method uses point cloud data for projection to obtain multi-view images, and then uses multi-view visual features to back-project into three-dimensional space and aggregate into unified point features to refine the training of point representation. The point features are then aggregated into supervoxels, and an iterative clustering process is performed to mine semantic classes. The disadvantage of this method is that it iteratively trains a three-dimensional unsupervised pre-trained model using a clustering method for two-dimensional features, and trains the two models separately without combining the extraction of two-dimensional features with three-dimensional features for synchronous training. It fails to fully iterate the two-dimensional model, affecting the effect of feature extraction. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings of the existing methods and propose a scene object classification method and system based on multi-view depth images. The main problem solved by the present invention is how to combine the advantages of the two point cloud generation methods of multi-view images and depth cameras to improve the quality of point cloud data, make full use of the information of two-dimensional features and three-dimensional features, and enhance the effect of semantic segmentation.

[0009] In order to solve the above problems, the present invention proposes a scene object classification method based on multi-view depth images, the method comprising:

[0010] Prepare three depth cameras, one of which is set at the center of the scene, vertically downward, and the other depth cameras are evenly arranged around the center, with the lens set diagonally downward. After deployment, record the camera coordinates [x0, y0, z0], use the deployed depth camera to shoot, obtain multiple depth images, and use Gaussian filtering to denoise and obtain I depth ;

[0011] Using the depth image I depth And the camera coordinates [x0, y0, z0] generate the point cloud data P corresponding to the depth image view , P view Contains the coordinates of the point cloud in the unified coordinate system [x w ,y w , z w ], the color information of the point cloud and the mapping relationship between the pixels of the depth image and the point cloud, and the point cloud data P from multiple perspectives view Align and fuse to generate synthetic multi-view point cloud data P s , the depth image I depth The RGB information is extracted to obtain multiple two-dimensional images I rgb ;

[0012] The synthesized multi-view point cloud data P s Perform feature extraction to obtain the three-dimensional feature F 3d , for the two-dimensional image I rgb Perform feature extraction to obtain the two-dimensional feature F 2d , the two-dimensional feature F 2d Mapping to three-dimensional feature F 3d So that each point in the point cloud has a unique corresponding mapping two-dimensional feature F 2d * , the three-dimensional feature F 3d and mapping two-dimensional features F 2d * Align and stitch to obtain the synthesized multi-view point cloud data P s Features F;

[0013] The synthesized multi-view point cloud data P s The feature F is input into the decoder for semantic segmentation to obtain a semantic segmentation result, that is, a classification result of objects in the scene.

[0014] Preferably, using the depth image I depth And the camera coordinates [x0, y0, z0] generate the point cloud data P corresponding to the depth image view , P view Contains the coordinates of the point cloud in the unified coordinate system [x w ,y w , zw ], the color information of the point cloud and the mapping relationship between the pixels of the depth image and the point cloud, and the point cloud data P from multiple perspectives view Align and fuse to generate synthetic multi-view point cloud data P s , the depth image I depth The RGB information is extracted to obtain multiple two-dimensional images I rgb , specifically:

[0015] The coordinates of the point cloud in the unified coordinate system [x w ,y w , z w The calculation method of ] is divided into two steps:

[0016] First, the depth image I depth First, calculate the pixel point cloud coordinates [x c ,y c , z c ], the calculation formula is as follows:

[0017]

[0018] Where d is the pixel depth information of the point, f is the focal length of the camera, [u, v] is the position of the point on the depth image, and [u0, v0] is the center of the depth image;

[0019] Then, the point cloud coordinates [x c ,y c , z c ] Calculate the point cloud coordinates of the point in the unified coordinate system [x w ,y w , z w ], the calculation formula is as follows:

[0020] [x w ,y w , z w ] T =R·[x c ,y c , z c ] T +[x0, y0, z0],

[0021] Where R is the rotation matrix;

[0022] For the color information of the point cloud, illumination estimation is used to eliminate the difference in illumination conditions between different cameras and determine the color of the point cloud, that is, the point cloud is projected into the image space using camera parameters and geometric information of different viewing angles, and the distribution of ambient illumination in the point cloud is estimated by analyzing the brightness and color information in the projected image, that is, the estimated illumination condition;

[0023] The point cloud data P corresponding to the depth image of different viewing angles view The aligned point cloud data are aligned and converted into a common coordinate system, and the aligned point cloud data are fused, that is, the point cloud data are clustered and the coordinates are averaged, and then the color of the point cloud is adjusted according to the estimated lighting conditions to obtain the synthesized multi-view point cloud data P s ;

[0024] Extract the RGB information in the depth image to obtain multiple two-dimensional images I rgb .

[0025] Preferably, the synthesized multi-view point cloud data P s Perform feature extraction to obtain the three-dimensional feature F 3d , for the two-dimensional image I rgb Perform feature extraction to obtain the two-dimensional feature F 2d , the two-dimensional feature F 2d Mapping to three-dimensional feature F 3d So that each point in the point cloud has a unique corresponding mapping two-dimensional feature F 2d * , the three-dimensional feature F 3d and mapping two-dimensional features F 2d * Align and stitch to obtain the synthesized multi-view point cloud data P s The characteristics F are as follows:

[0026] The geometric masked autoencoder (GeoMAE) from the paper “GeoMAE: Masked Geometric Target Prediction for Self-supervised Point Cloud Pre-Training” is introduced to train the synthesized multi-view point cloud data P using the geometric masked autoencoder GeoMAE. s Perform feature extraction to obtain the three-dimensional feature F 3d , where the three-dimensional feature F needs to be maintained during feature extraction 3d The correspondence between the point cloud data coordinates and the two-dimensional features for alignment and splicing;

[0027] The residual network ResNet is used to process the two-dimensional image I rgb Perform feature extraction to obtain the two-dimensional feature F 2d ;

[0028] The two-dimensional feature F 2dAccording to the mapping relationship between the pixel points of the depth image and the point cloud, the synthesized multi-view point cloud data P is mapped to s , that is, for the two-dimensional feature fs of the kth point k , if there exists n k Two-dimensional features Mapped to the same point in the point cloud, these two-dimensional features For averaging, if n k =0, then fs k It is recorded as a characteristic matrix of all 0s, that is,

[0029]

[0030] The synthesized multi-view point cloud data P s There are n points, the mapping two-dimensional feature F 2d * It can be expressed as F 2d * = [fs0, fs1, ..., fs n ];

[0031] The three-dimensional feature F 3d and the mapped two-dimensional feature F 2d * Perform point correspondence splicing to obtain feature F, that is,

[0032] Preferably, the synthesized multi-view point cloud data P s The feature F is input into the decoder for semantic segmentation to obtain the semantic segmentation result, that is, the classification result of the objects in the scene, which is specifically:

[0033] The feature F is used as the feature extracted by the feature extraction module in the KPConv network and input into the prediction head of the KPConv network. Then, the decoder is trained using the actual scene data constructed by the user. The loss function used is the cross entropy of the predicted label category and the true label category, and a trained decoder is obtained.

[0034] The synthesized multi-view point cloud data P s The trained decoder is input to obtain a semantic segmentation result, that is, a classification result of objects in the scene.

[0035] Accordingly, the present invention also provides a scene object classification system based on multi-view depth images, comprising:

[0036] The depth image acquisition unit is used to prepare depth cameras in three directions, one of which is set at the center of the scene, pointing vertically downward, and the other depth cameras are evenly surrounded by the center point, with the lens set diagonally downward. After deployment, the camera coordinates [x0, y0, z0] are recorded, and the deployed depth cameras are used to shoot and obtain multiple depth images I depth ;

[0037] A point cloud data and two-dimensional image acquisition unit is used to obtain the depth image I depth And the camera coordinates [x0, y0, z0] generate the point cloud data P corresponding to the depth image view , P view Contains the coordinates of the point cloud in the unified coordinate system [x w ,y w , z w ], the color information of the point cloud and the mapping relationship between the pixels of the depth image and the point cloud, and the point cloud data P from multiple perspectives view Align and fuse to generate synthetic multi-view point cloud data P s , the depth image I depth The RGB information is extracted to obtain multiple two-dimensional images I rgb ;

[0038] A feature extraction and processing unit is used to extract and process the synthesized multi-view point cloud data P s Perform feature extraction to obtain the three-dimensional feature F 3d , for the two-dimensional image I rgb Perform feature extraction to obtain the two-dimensional feature F 2d , the two-dimensional feature F 2d Mapping to three-dimensional feature F 3d So that each point in the point cloud has a unique corresponding mapping two-dimensional feature F 2d * , the three-dimensional feature F 3d and mapping two-dimensional features F 2d * Align and stitch to obtain the synthesized multi-view point cloud data P s Features F;

[0039] A decoding and result output unit is used to convert the synthesized multi-view point cloud data P s The feature F is input into the decoder for semantic segmentation to obtain a semantic segmentation result, that is, a classification result of objects in the scene.

[0040] The implementation of the present invention has the following beneficial effects:

[0041] The present invention can combine the advantages of two point cloud generation methods, multi-view images and depth cameras, to improve the quality of point cloud data. The multimodal method can also make full use of the information of two-dimensional features and three-dimensional features, take into account illumination and texture information, and better extract spatial and structural information. In addition, the synchronous training of the two models can better balance the weight of multimodal information, adapt to data deployed in different scenes and different devices, and mine the information of multimodal data as much as possible, improve the effect of scene semantic segmentation, and complete the task of scene object classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of a scene object classification method based on multi-view depth images according to an embodiment of the present invention;

[0043] Figure 2 is a data encoding and decoding flow chart of an embodiment of the present invention;

[0044] Figure 3 It is a structural diagram of a scene object classification system based on multi-view depth images according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] Figure 1 is a flow chart of a method for classifying scene objects based on multi-view depth images according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0047] S1, prepare three depth cameras, one of which is set at the center of the scene, vertically downward, and the other depth cameras are evenly around the center, with the lens set diagonally downward. After deployment, record the camera coordinates [x0, y0, z0], use the deployed depth camera to shoot, obtain multiple depth images and use Gaussian filtering to denoise to obtain I depth ;

[0048] S2, using the depth image I depth And the camera coordinates [x0, y0, z0] generate the point cloud data P corresponding to the depth image view , P view Contains the coordinates of the point cloud in the unified coordinate system [x w ,y w , z w], the color information of the point cloud and the mapping relationship between the pixels of the depth image and the point cloud, and the point cloud data P from multiple perspectives view Align and fuse to generate synthetic multi-view point cloud data P s , the depth image I depth The RGB information is extracted to obtain multiple two-dimensional images I rgb ;

[0049] S3, the synthesized multi-view point cloud data P s Perform feature extraction to obtain the three-dimensional feature F 3d , for the two-dimensional image I rgb Perform feature extraction to obtain the two-dimensional feature F 2d , the two-dimensional feature F 2d Mapping to three-dimensional feature F 3d So that each point in the point cloud has a unique corresponding mapping two-dimensional feature F 2d * , the three-dimensional feature F 3d and mapping two-dimensional features F 2d * Align and stitch to obtain the synthesized multi-view point cloud data P s Features F;

[0050] S4, the synthesized multi-view point cloud data P s The feature F is input into the decoder for semantic segmentation to obtain a semantic segmentation result, that is, a classification result of objects in the scene.

[0051] Step S2 is as follows:

[0052] S2-1, the coordinates of the point cloud in the unified coordinate system [x w ,y w , z w The calculation method of ] is divided into two steps:

[0053] First, the depth image I depth First, calculate the pixel point cloud coordinates [x c ,y c , z c ], the calculation formula is as follows:

[0054]

[0055] Where d is the pixel depth information of the point, f is the focal length of the camera, [u, v] is the position of the point on the depth image, and [u0, v0] is the center of the depth image;

[0056] Then, the point cloud coordinates [x c ,yc , z c ] Calculate the point cloud coordinates of the point in the unified coordinate system [x w ,y w , z w ], the calculation formula is as follows:

[0057] [x w ,y w , z w ] T =R·[x c ,y c , z c ] T +[x0, y0, z0],

[0058] Where R is the rotation matrix;

[0059] S2-2, for the color information of the point cloud, using illumination estimation to eliminate the difference in illumination conditions between different cameras and determine the color of the point cloud, that is, using camera parameters and geometric information of different viewing angles to project the point cloud into the image space, and by analyzing the brightness and color information in the projected image, estimating the distribution of ambient illumination in the point cloud, that is, estimating the illumination conditions;

[0060] S2-3, the point cloud data P corresponding to the depth image of different viewing angles view The aligned point cloud data are aligned and converted into a common coordinate system, and the aligned point cloud data are fused, that is, the point cloud data are clustered and the coordinates are averaged, and then the color of the point cloud is adjusted according to the estimated lighting conditions to obtain the synthesized multi-view point cloud data P s ;

[0061] S2-4, extracting RGB information from the depth image to obtain multiple two-dimensional images I rgb .

[0062] Step S3, such as Figure 2 As shown, the details are as follows:

[0063] S3-1, introduce the geometric masked autoencoder (GeoMAE) of the paper "GeoMAE: Masked Geometric Target Prediction for Self-supervised Point Cloud Pre-Training", and use the geometric masked autoencoder GeoMAE to train the synthesized multi-view point cloud data P s Perform feature extraction to obtain the three-dimensional feature F 3d , where the three-dimensional feature F needs to be maintained during feature extraction 3dThe correspondence between the point cloud data coordinates and the two-dimensional features for alignment and splicing;

[0064] S3-2, using the residual network ResNet to process the two-dimensional image I rgb Perform feature extraction to obtain the two-dimensional feature F 2d ;

[0065] S3-3, the two-dimensional feature F 2d According to the mapping relationship between the pixel points of the depth image and the point cloud, the synthesized multi-view point cloud data P is mapped to s , that is, for the two-dimensional feature fs of the kth point k , if there exists n k Two-dimensional features Mapped to the same point in the point cloud, these two-dimensional features For averaging, if n k =0, then fs k It is recorded as a characteristic matrix of all 0s, that is,

[0066]

[0067] The synthesized multi-view point cloud data P s There are n points, the mapping two-dimensional feature F 2d * It can be expressed as F 2d * = [fs0, fs1, ..., fs n ];

[0068] S3-4, the three-dimensional feature F 3d and the mapped two-dimensional feature F 2d * Perform point correspondence splicing to obtain feature F, that is,

[0069] Step S4 is as follows:

[0070] S4-1, input the feature F as the feature extracted by the feature extraction module in the KPConv network into the prediction head of the KPConv network, and then use the actual scene data constructed by the user to train the decoder, and use the loss function as the cross entropy of the predicted label category and the true label category to obtain a trained decoder;

[0071] S4-2, the synthesized multi-view point cloud data P s The trained decoder is input to obtain a semantic segmentation result, that is, a classification result of objects in the scene.

[0072] Accordingly, the present invention also provides a scene object classification system based on multi-view depth images, such as Figure 3 As shown, including:

[0073] The depth image acquisition unit 1 is used to prepare depth cameras in three directions, one of which is set at the center of the scene, vertically downward, and the other depth cameras are evenly surrounded by the center point, and the lenses are set diagonally downward. After the deployment is completed, the camera coordinates [x0, y0, z0] are recorded, and the deployed depth cameras are used to shoot and obtain multiple depth images I depth ;

[0074] Point cloud data and two-dimensional image acquisition unit 2, used to use the depth image I depth And the camera coordinates [x0, y0, z0] generate the point cloud data P corresponding to the depth image view , P view Contains the coordinates of the point cloud in the unified coordinate system [x w ,y w , z w ], the color information of the point cloud and the mapping relationship between the pixels of the depth image and the point cloud, and the point cloud data P from multiple perspectives view Align and fuse to generate synthetic multi-view point cloud data P s , the depth image I depth The RGB information is extracted to obtain multiple two-dimensional images I rgb ;

[0075] Specifically, the coordinates of the point cloud in the unified coordinate system [x w ,y w , z w The calculation method of ] is divided into two steps:

[0076] First, the depth image I depth First, calculate the pixel point cloud coordinates [x c ,y c , z c ], the calculation formula is as follows:

[0077]

[0078] Where d is the pixel depth information of the point, f is the focal length of the camera, [u, v] is the position of the point on the depth image, and [u0, v0] is the center of the depth image;

[0079] Then, the point cloud coordinates [x c ,y c , z c ] Calculate the point cloud coordinates of the point in the unified coordinate system [x w ,y w , z w], the calculation formula is as follows:

[0080] [x w ,y w , z w ] T =R·[x c ,y c , z c ] T +[x0, y0, z0],

[0081] Where R is the rotation matrix;

[0082] For the color information of the point cloud, illumination estimation is used to eliminate the difference in illumination conditions between different cameras and determine the color of the point cloud, that is, the point cloud is projected into the image space using camera parameters and geometric information of different viewing angles, and the distribution of ambient illumination in the point cloud is estimated by analyzing the brightness and color information in the projected image, that is, the estimated illumination condition;

[0083] The point cloud data P corresponding to the depth image of different viewing angles view The aligned point cloud data are aligned and converted into a common coordinate system, and the aligned point cloud data are fused, that is, the point cloud data are clustered and the coordinates are averaged, and then the color of the point cloud is adjusted according to the estimated lighting conditions to obtain the synthesized multi-view point cloud data P s ;

[0084] Extract the RGB information in the depth image to obtain multiple two-dimensional images I rgb .

[0085] The feature extraction and processing unit 3 is used to extract and process the synthesized multi-view point cloud data P s Perform feature extraction to obtain the three-dimensional feature F 3d , for the two-dimensional image I rgb Perform feature extraction to obtain the two-dimensional feature F 2d , the two-dimensional feature F 2d Mapping to three-dimensional feature F 3d So that each point in the point cloud has a unique corresponding mapping two-dimensional feature F 2d * , the three-dimensional feature F 3d and mapping two-dimensional features F 2d * Align and stitch to obtain the synthesized multi-view point cloud data P s Features F;

[0086] Specifically, the geometric masked autoencoder (GeoMAE) of the paper “GeoMAE: Masked Geometric Target Prediction for Self-supervised Point Cloud Pre-Training” is introduced, and the geometric masked autoencoder GeoMAE is used to train the synthesized multi-view point cloud data P s Perform feature extraction to obtain the three-dimensional feature F 3d , where the three-dimensional feature F needs to be maintained during feature extraction 3d The correspondence between the point cloud data coordinates and the two-dimensional features for alignment and splicing;

[0087] The residual network ResNet is used to process the two-dimensional image I rgb Perform feature extraction to obtain the two-dimensional feature F 2d ;

[0088] The two-dimensional feature F 2d According to the mapping relationship between the pixel points of the depth image and the point cloud, the synthesized multi-view point cloud data P is mapped to s , that is, for the two-dimensional feature fs of the kth point k , if there exists n k Two-dimensional features Mapped to the same point in the point cloud, these two-dimensional features For averaging, if n k =0, then fs k It is recorded as a characteristic matrix of all 0s, that is,

[0089]

[0090] The synthesized multi-view point cloud data P s There are n points, the mapping two-dimensional feature F 2d * It can be expressed as F 2d * =[fs0, ffs1, ..., fs n ];

[0091] The three-dimensional feature F 3d and the mapped two-dimensional feature F 2d * Perform point correspondence splicing to obtain feature F, that is,

[0092] The decoding and result output unit 4 is used to convert the synthesized multi-view point cloud data P sThe feature F is input into the decoder for semantic segmentation to obtain a semantic segmentation result, that is, a classification result of objects in the scene;

[0093] Specifically, the feature F is used as the feature extracted by the feature extraction module in the KPConv network and input into the prediction head of the KPConv network, and then the decoder is trained using the actual scene data constructed by the user, and the loss function used is the cross entropy of the predicted label category and the true label category, to obtain a trained decoder;

[0094] The synthesized multi-view point cloud data P s The trained decoder is input to obtain a semantic segmentation result, that is, a classification result of objects in the scene.

[0095] Therefore, the present invention can combine the advantages of the two point cloud generation methods of multi-view images and depth cameras, improve the quality of point cloud data, make full use of the information of two-dimensional features and three-dimensional features, and enhance the effect of semantic segmentation.

[0096] The above is a detailed introduction to a scene object classification method and system based on multi-view depth images provided by an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A scene object classification method based on multi-view depth images, characterized in that: The method comprises: Prepare three depth cameras, one of which is set at the center of the scene, vertically downward, and the other depth cameras are evenly arranged around the center, with the lens set diagonally downward. After deployment, record the camera coordinates [x0, y0, z0], use the deployed depth camera to shoot, obtain multiple depth images, and use Gaussian filtering to denoise to obtain I depth ; Using the depth image I depth And the camera coordinates [x0, y0, z0] generate the point cloud data P corresponding to the depth image view , P view Contains the coordinates of the point cloud in the unified coordinate system [x w ,y w ,z w ], the color information of the point cloud and the mapping relationship between the pixels of the depth image and the point cloud, and the point cloud data P from multiple perspectives view Align and fuse to generate synthetic multi-view point cloud data P s , the depth image I depth The RGB information is extracted to obtain multiple two-dimensional images I rgb ; The synthesized multi-view point cloud data P s Perform feature extraction to obtain the three-dimensional feature F 3d , for the two-dimensional image I rgb Perform feature extraction to obtain the two-dimensional feature F 2d , the two-dimensional feature F 2d Mapping to three-dimensional feature F 3d So that each point in the point cloud has a unique corresponding mapping two-dimensional feature F 2d * , the three-dimensional feature F 3d and mapping two-dimensional features F 2d * Align and stitch to obtain the synthesized multi-view point cloud data P s Features F; The synthesized multi-view point cloud data P s The feature F is input into the decoder for semantic segmentation to obtain a semantic segmentation result, that is, a classification result of objects in the scene.

2. The method for classifying scene objects based on multi-view depth images according to claim 1, characterized in that: The use of the depth image I depth And the camera coordinates [x0, y0, z0] generate the point cloud data P corresponding to the depth image view , P view Contains the coordinates of the point cloud in the unified coordinate system [x w ,y w ,z w ], the color information of the point cloud and the mapping relationship between the pixels of the depth image and the point cloud, and the point cloud data P from multiple perspectives view Align and fuse to generate synthetic multi-view point cloud data P s , the depth image I depth The RGB information is extracted to obtain multiple two-dimensional images I rgb , specifically: The coordinates of the point cloud in the unified coordinate system [x w ,y w ,z w The calculation method of ] is divided into two steps: First, the depth image I depth First, calculate the pixel point cloud coordinates [x c ,y c ,z c ], the calculation formula is as follows: Where d is the pixel depth information of the point, f is the focal length of the camera, [u,v] is the position of the point on the depth image, and [u0,v0] is the center of the depth image; Then, the point cloud coordinates [x c ,y c ,z c ] Calculate the point cloud coordinates of the point in the unified coordinate system [x w ,y w ,z w ], the calculation formula is as follows: [x w ,y w ,z w ] T =R·[x c ,y c ,z c ] T +[x0,y0,z0], Where R is the rotation matrix; For the color information of the point cloud, illumination estimation is used to eliminate the difference in illumination conditions between different cameras and determine the color of the point cloud, that is, the point cloud is projected into the image space using camera parameters and geometric information of different viewing angles, and the distribution of ambient illumination in the point cloud is estimated by analyzing the brightness and color information in the projected image, that is, the estimated illumination condition; The point cloud data P corresponding to the depth image of different viewing angles view The aligned point cloud data are aligned and converted into a common coordinate system, and the aligned point cloud data are fused, that is, the point cloud data are clustered and the coordinates are averaged, and then the color of the point cloud is adjusted according to the estimated lighting conditions to obtain the synthesized multi-view point cloud data P s ; Extract the RGB information in the depth image to obtain multiple two-dimensional images I rgb .

3. The method for classifying scene objects based on multi-view depth images according to claim 1, characterized in that: The synthesized multi-view point cloud data P s Perform feature extraction to obtain the three-dimensional feature F 3d , for the two-dimensional image I rgb Perform feature extraction to obtain the two-dimensional feature F 2d , the two-dimensional feature F 2d Mapping to three-dimensional feature F 3d So that each point in the point cloud has a unique corresponding mapping two-dimensional feature F 2d * , the three-dimensional feature F 3d and mapping two-dimensional features F 2d * Align and stitch to obtain the synthesized multi-view point cloud data P s The characteristics F are as follows: The geometric mask autoencoder GeoMAE is used to encode the synthesized multi-view point cloud data P s Perform feature extraction to obtain the three-dimensional feature F 3d , where the three-dimensional feature F needs to be maintained during feature extraction 3d The correspondence between the point cloud data coordinates and the two-dimensional features for alignment and splicing; The residual network ResNet is used to process the two-dimensional image I rgb Perform feature extraction to obtain the two-dimensional feature F 2d ; The two-dimensional feature F 2d According to the mapping relationship between the pixel points of the depth image and the point cloud, the synthesized multi-view point cloud data P is mapped to s , that is, for the two-dimensional feature fs of the kth point k , if there exists n k Two-dimensional features Mapped to the same point in the point cloud, these two-dimensional features For averaging, if n k =0, then fs k It is recorded as a characteristic matrix of all 0s, that is, The synthesized multi-view point cloud data P s There are n points, the mapping two-dimensional feature F 2d * It can be expressed as F 2d * =[fs0,fs1,...,fs n ]; The three-dimensional feature F 3d and the mapped two-dimensional feature F 2d * Perform point correspondence splicing to obtain feature F, that is, 4. The method for classifying scene objects based on multi-view depth images according to claim 1, characterized in that: The synthesized multi-view point cloud data P s The feature F is input into the decoder for semantic segmentation to obtain the semantic segmentation result, that is, the classification result of the objects in the scene, which is specifically: The feature F is used as the feature extracted by the feature extraction module in the KPConv network and input into the prediction head of the KPConv network. Then, the decoder is trained using the actual scene data constructed by the user. The loss function used is the cross entropy of the predicted label category and the true label category, and a trained decoder is obtained. The synthesized multi-view point cloud data P s The trained decoder is input to obtain a semantic segmentation result, that is, a classification result of objects in the scene.

5. A scene object classification system based on multi-view depth images, characterized in that: The system comprises: The depth image acquisition unit is used to prepare depth cameras in three directions, one of which is set at the center of the scene, pointing vertically downward, and the other depth cameras are evenly surrounded by the center point, with the lens set diagonally downward. After deployment, the camera coordinates [x0, y0, z0] are recorded, and the deployed depth cameras are used to shoot and obtain multiple depth images I depth ; A point cloud data and two-dimensional image acquisition unit is used to obtain the depth image I depth And the camera coordinates [x0, y0, z0] generate the point cloud data P corresponding to the depth image view , P view Contains the coordinates of the point cloud in the unified coordinate system [x w ,y w ,z w ], the color information of the point cloud and the mapping relationship between the pixels of the depth image and the point cloud, and the point cloud data P from multiple perspectives view Align and fuse to generate synthetic multi-view point cloud data P s , the depth image I depth The RGB information is extracted to obtain multiple two-dimensional images I rgb ; A feature extraction and processing unit is used to extract and process the synthesized multi-view point cloud data P s Perform feature extraction to obtain the three-dimensional feature F 3d , for the two-dimensional image I rgb Perform feature extraction to obtain the two-dimensional feature F 2d , the two-dimensional feature F 2d Mapping to three-dimensional feature F 3d So that each point in the point cloud has a unique corresponding mapping two-dimensional feature F 2d * , the three-dimensional feature F 3d and mapping two-dimensional features F 2d * Align and stitch to obtain the synthesized multi-view point cloud data P s Features F; A decoding and result output unit is used to convert the synthesized multi-view point cloud data P s The feature F is input into the decoder for semantic segmentation to obtain a semantic segmentation result, that is, a classification result of objects in the scene.

6. The scene object classification system based on multi-view depth images according to claim 5, characterized in that: The point cloud data and two-dimensional image acquisition unit is used to use the depth image I depth And the camera coordinates [x0, y0, z0] generate the point cloud data P corresponding to the depth image view , P view Contains the coordinates of the point cloud in the unified coordinate system [x w ,y w ,z w ], the color information of the point cloud and the mapping relationship between the pixels of the depth image and the point cloud, and the point cloud data P from multiple perspectives view Align and fuse to generate synthetic multi-view point cloud data P s , the depth image I depth The RGB information is extracted to obtain multiple two-dimensional images I rgb , specifically: The coordinates of the point cloud in the unified coordinate system [x w ,y w ,z w The calculation method of ] is divided into two steps: First, the depth image I depth First, calculate the pixel point cloud coordinates [x c ,y c ,z c ], the calculation formula is as follows: Where d is the pixel depth information of the point, f is the focal length of the camera, [u,v] is the position of the point on the depth image, and [u0,v0] is the center of the depth image; Then, the point cloud coordinates [x c ,y c ,z c ] Calculate the point cloud coordinates of the point in the unified coordinate system [x w ,y w ,z w ], the calculation formula is as follows: [x w ,y w ,z w ] T =R·[x c ,y c ,z c ] T +[x0,y0,z0], Where R is the rotation matrix; For the color information of the point cloud, illumination estimation is used to eliminate the difference in illumination conditions between different cameras and determine the color of the point cloud, that is, the point cloud is projected into the image space using camera parameters and geometric information of different viewing angles, and the distribution of ambient illumination in the point cloud is estimated by analyzing the brightness and color information in the projected image, that is, the estimated illumination condition; The point cloud data P corresponding to the depth image of different viewing angles view The aligned point cloud data are aligned and converted into a common coordinate system, and the aligned point cloud data are fused, that is, the point cloud data are clustered and the coordinates are averaged, and then the color of the point cloud is adjusted according to the estimated lighting conditions to obtain the synthesized multi-view point cloud data P s ; Extract the RGB information in the depth image to obtain multiple two-dimensional images I rgb .

7. The scene object classification system based on multi-view depth images according to claim 5, characterized in that: The feature extraction and processing unit is used to extract and process the synthesized multi-view point cloud data P s Perform feature extraction to obtain the three-dimensional feature F 3d , for the two-dimensional image I rgb Perform feature extraction to obtain the two-dimensional feature F 2d , the two-dimensional feature F 2d Mapping to three-dimensional feature F 3d So that each point in the point cloud has a unique corresponding mapping two-dimensional feature F 2d * , the three-dimensional feature F 3d and mapping two-dimensional features F 2d * Align and stitch to obtain the synthesized multi-view point cloud data P s The characteristics F are as follows: The geometric mask autoencoder GeoMAE is used to encode the synthesized multi-view point cloud data P s Perform feature extraction to obtain the three-dimensional feature F 3d , where the three-dimensional feature F needs to be maintained during feature extraction 3d The correspondence between the point cloud data coordinates and the two-dimensional features for alignment and splicing; The residual network ResNet is used to process the two-dimensional image I rgb Perform feature extraction to obtain the two-dimensional feature F 2d ; The two-dimensional feature F 2d According to the mapping relationship between the pixel points of the depth image and the point cloud, the synthesized multi-view point cloud data P is mapped to s , that is, for the two-dimensional feature fs of the kth point k , if there exists n k Two-dimensional features Mapped to the same point in the point cloud, these two-dimensional features For averaging, if n k =0, then fs k It is recorded as a characteristic matrix of all 0s, that is, The synthesized multi-view point cloud data P s There are n points, the mapping two-dimensional feature F 2d * It can be expressed as F 2d * =[fs0,fs1,...,fs n ]; The three-dimensional feature F 3d and the mapped two-dimensional feature F 2d * Perform point correspondence splicing to obtain feature F, that is, 8. The scene object classification system based on multi-view depth images according to claim 5, characterized in that: The decoding and result output unit is used to convert the synthesized multi-view point cloud data P s The feature F is input into the decoder for semantic segmentation to obtain the semantic segmentation result, that is, the classification result of the objects in the scene, which is specifically: The feature F is used as the feature extracted by the feature extraction module in the KPConv network and input into the prediction head of the KPConv network. Then, the decoder is trained using the actual scene data constructed by the user. The loss function used is the cross entropy of the predicted label category and the true label category, and a trained decoder is obtained. The synthesized multi-view point cloud data P s The trained decoder is input to obtain a semantic segmentation result, that is, a classification result of objects in the scene.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images