An automatic driving vehicle pose estimation method
By combining networks such as Mask-RCNN, PSPNet, PointNet++, and Transformer, instance and category features are extracted and fused to generate an attention matrix for deformation. This solves the accuracy problem of pose estimation in complex scenes in existing methods and improves the accuracy of pose estimation for autonomous vehicles.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHONGQING UNIVERSITY OF SCIENCE AND TECHNOLOGY
- Filing Date
- 2024-10-21
- Publication Date
- 2026-06-02
AI Technical Summary
Existing deep learning-based pose estimation methods ignore the distinction between instance features and category features in complex and varied scenarios, leading to reduced object fitting accuracy. Performance is particularly affected in severely occluded scenarios, and accuracy needs to be improved.
The Mask-RCNN network is used to detect car objects and extract instance images and point cloud features. PSPNet and PointNet++ networks are combined for feature extraction. The Transformer fusion module and FSAM and FCAM modules are used for feature encoding and decoding. GSENet and PointNet++ networks are used to extract global shape and category features, generate attention matrices for deformation, and finally predict rotation angle, size and translation.
It effectively extracts local and global features from RGB images and point clouds, achieving complementary fusion of different features and improving the accuracy and precision of pose estimation.
Smart Images
Figure CN119540898B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the research field of autonomous driving and computer vision, and specifically relates to a method for estimating the pose of autonomous vehicles. Background Technology
[0002] With the rapid development of artificial intelligence and deep learning technologies in recent years, pose estimation methods based on deep learning have become increasingly important. Pose estimation refers to accurately inferring the position, orientation, and pose of a target object based on input image or video data. This technology has wide applications in computer vision, robotics, augmented reality, and other fields.
[0003] The development of deep learning technology has brought about significant improvements in pose estimation. Traditional pose estimation methods typically rely on hand-designed features and models, which perform poorly in complex scenes and under varying conditions. In contrast, deep learning-based methods can automatically learn features and patterns from large amounts of data, thus better adapting to complex scenes and changes.
[0004] To address the challenge of fitting prior point clouds to a target object and performing shape deformation, ensuring sufficient information is available for the deformation process is crucial. Traditional object pose estimation methods include directly regressing the final pose from RGB-D images and regressing the object pose by directly concatenating RGB images and point cloud features. However, these traditional methods neglect the differences between instance features and class features, as well as the differences between RGB image and point cloud features, leading to reduced accuracy in object fitting during pose estimation. Furthermore, they negatively impact performance in severely occluded scenes.
[0005] Therefore, the pose estimation methods described above ignore the differences between features, leading to insufficient extraction of object information in complex backgrounds. Furthermore, the accuracy decreases during target fitting. The accuracy in practical applications needs improvement, thus requiring urgent refinement. Summary of the Invention
[0006] To address the shortcomings of the existing technologies, this invention proposes a pose estimation method for autonomous vehicles, which can better estimate the pose of objects in images even in unfamiliar environments.
[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:
[0008] A method for estimating the pose of an autonomous vehicle, the key technology of which lies in the following steps:
[0009] A method for estimating the pose of an autonomous vehicle, characterized in that the method includes:
[0010] S1: Color images of the road during vehicle movement, obtained from the perspective of the car's camera; use the color images and 3D models of similar cars as input data for the model.
[0011] S2: Using a Mask-RCNN-based network, obtain instance images of cars in the image and 3D point clouds of cars in the car depth image;
[0012] S3: Feature extraction is performed on the obtained car instance images and car point clouds using PSPNet and PointNet++ networks, respectively;
[0013] S4: The instance features extracted above are fused by the Transformer-based fusion module, the features are encoded using FSAM, and the different features are decoded and fused by combining the FSAM and FCAM modules to obtain the global features of the instance;
[0014] S5: The global shape features of the input 3D car model are extracted using GSENet based on ResNet-1 network, and then the PointNet++ network is used to extract features to obtain category-level features;
[0015] S6: By inputting the obtained global shape features and instance features into the prior adaptive module, new class-level global features are obtained. Then, the attention matrix is obtained by fusing the class-level global features and instance global features using the Transformer-based fusion module.
[0016] S7: Introduce the attention matrix obtained in S6 into the Transformer-based attention module to deform the features obtained in S5 in the geometric space and feature space to obtain the rotation angle, size, and translation.
[0017] Furthermore: the specific content of step S1:
[0018] S11: A color image of the road during vehicle movement, captured from the perspective of a car camera. Set the image resolution (width and height) to W and H.
[0019] S12: Take a known 3D model of the car class as input.
[0020] Furthermore: the specific content of step S2:
[0021] S21: Use the Mask-RCNN network to detect car objects in the image and obtain the instance mask.
[0022] S22: Based on the mask, crop out the car object from the image and resize the cropped image to I. o ∈R H×W×3 .
[0023] S23: Based on the depth image obtained from the Mask-RCNN network, crop out the corresponding positions of the car, and generate the car's point cloud based on the camera intrinsic parameter matrix. Randomly sample N from the point cloud. o Each point serves as a scene point cloud.
[0024] Furthermore: the specific content of step S3:
[0025] S31: The image I obtained in step S2 o And point cloud P o The images are then fed into PSPNet and PointNet++ networks respectively to extract features, thus obtaining image features. and point cloud features
[0026] Furthermore: the specific content of step S4:
[0027] S41: The Transformer-based fusion module consists of an encoder and a decoder. The encoder process uses the FSAM module to encode the feature maps, while the decoder process uses both the FSAM and FCAM modules to fuse features from different sources.
[0028] S42: Image features obtained from S31 and point cloud features F o ∈R N×d It uses a series of MLPs to map features into queries, keys, and values required by the FSAM module.
[0029]
[0030] Where q, k, and v represent query, key, and value, respectively, and E represents the parameters used in the encoder process.
[0031] S43: Based on the parameters obtained in S42, use the FSAM module to generate encoded features, and obtain...
[0032]
[0033] Where FSAM represents the FSAM module, satisfying...
[0034]
[0035] Where d k Let k be the dimension.
[0036] S44: Obtain Based on this, Introducing a decoder and using FSAM for feature extraction yields...
[0037]
[0038] Where q, k, and v represent query, key, and value, respectively, and D represents the parameters used in the encoder process.
[0039] S45: Obtained from S44 Use the FCAM module to fuse RGB images and point cloud features.
[0040]
[0041] Where q, k, and v represent query, key, and value respectively, and FCAM represents the FCAM module, satisfying...
[0042]
[0043] In the formula, i,j∈{rgb,o},i=j.
[0044] S46: A decoder composed of S44 and S45, which obtains features. Therefore, RGB images and point cloud features can be complementaryly fused to obtain F. rgb ,F o The feature maps are then concatenated along the channel dimension to obtain instance features. Finally, the MLP should generate global features for the instance.
[0045]
[0046] in Representing local features of an instance, This indicates concatenation along the channel dimension. This represents the global characteristics of the generated instance.
[0047] Furthermore: the specific content of step S5:
[0048] S51: Input the 3D car model input in step S12 into GSENet to extract the global shape G of the car category. s ∈R n×3 .
[0049] S52: Transfer the 3D model features G obtained in step S51 to... s The input is fed into the PointNet++ network to extract features, resulting in point cloud features.
[0050] Furthermore: the specific content of step S6:
[0051] S61: The instance local features F obtained in step S46 are... rgb ,Fo and the global feature F from step S52 r Introduced into the prior adaptive module, F r F o F rgb Using these as queries, keys, and values respectively, prior adaptive semantic features are obtained through multi-head attention.
[0052]
[0053] S = Concat(Y) (1) ,Y (2) ,...,Y (m) (17)
[0054] in m represents the index of multi-head attention. This represents the prior adaptive semantic features.
[0055] S62: Combine the obtained prior adaptive semantic features S and point cloud features F r The system concatenates the data to generate category-level local features, and finally uses an MLP layer to generate category-level global features based on the obtained category-level local features.
[0056]
[0057] in Represents category-level local features. This represents a category-level global feature.
[0058] S63: The category-level global features obtained in S62 and the instance-level global features obtained in S46 are fused using a Transformer-based fusion module, and finally concatenated along the channel dimension to generate the global features.
[0059]
[0060] S64: In order to make full use of local features and make pose estimation more accurate, the global features obtained in S62 are concatenated with the instance local features, and finally an attention matrix is generated using the fused features.
[0061]
[0062] in This represents the attention matrix.
[0063] Furthermore: the specific content of step S7:
[0064] S71: To enable the network to effectively handle the size and shape of the target object, the features F obtained in step S5 are processed. rMLP layers are applied in the feature space and geometric space to obtain features in both spaces.
[0065] S r =MLP(F r ) (twenty two)
[0066] S f =MLP(F r ) (twenty three)
[0067] in Let represent the prior point cloud in geometric space and the prior point cloud in feature space, respectively.
[0068] S72: In the obtained S r S f The feature map is obtained by applying an MLP layer. It also generates queries, keys, and values for attention modules in geometric space and feature space.
[0069]
[0070] in Representing queries, keys, and values in the feature space.
[0071] This represents the query, key, and value in the geometric space. The obtained query, key, and value are then divided into several headers to obtain...
[0072] S73: Based on the attention matrix obtained in step S6, combine it with the attention module in geometric space to obtain S... r The deformation yields the deformed S' r .
[0073]
[0074] Where D K ∈R (C / h) Indicates channel dimension, S represents r The deformation offset. This represents the deformed prior point cloud.
[0075] S74: Similar to step S73, in the feature space, combine it with the attention module to obtain S. f The deformation yields the deformed S' f .
[0076]
[0077] Where D K ∈R (C / h) Indicates channel dimension, S represents f The deformation offset. This represents the deformed prior point cloud.
[0078] S75: The deformed prior point cloud is connected with local features, and the rotation, translation and size of the object are generated using MLP layers.
[0079]
[0080] Where R,t,s represents the rotation angle, size, and translation of the predicted object.
[0081] The beneficial effects of this invention are:
[0082] This patent proposes a pose estimation method for autonomous vehicles that can more effectively extract local and global features from RGB images and point clouds. It utilizes a feature fusion module to achieve complementarity and fusion between different features, and generates an attention matrix from the obtained features to deform the prior point cloud, better fit the target object, and ultimately improve the accuracy of pose estimation. Attached Figure Description
[0083] Figure 1 Flowchart of the method of the present invention
[0084] Figure 2 Feature fusion module diagram
[0085] Figure 3 Attention matrix generation module diagram
[0086] Figure 4 FSAM module schematic diagram
[0087] Figure 5 FCAM module schematic diagram
[0088] Figure 6 Flowchart of the prior point cloud deformation process
[0089] Figure 7 GSENet network flow diagram
[0090] Figure 8 Prior Adaptive Module Flowchart Detailed Implementation
[0091] The present invention will now be described in further detail with reference to the accompanying drawings.
[0092] In practical implementation: such as Figures 1 to 8 As shown, in this embodiment, color images acquired during vehicle movement and the vehicle's prior point cloud are used as the dataset for pose estimation. Specifically:
[0093] An autonomous vehicle pose estimation method, combining Figure 1 As can be seen, its characteristics include the following steps:
[0094] 1. A method for estimating the pose of an autonomous vehicle, characterized in that the method comprises:
[0095] S1: Obtain a color image of the road from the perspective of the car's camera while the vehicle is in motion. Use the color image and a 3D model of a similar car as input data for the model. The specific content of step S1 is as follows:
[0096] S11: A color image of the road during vehicle movement, captured from the perspective of a car camera. Set the image resolution (width and height) to W and H.
[0097] S12: Take a known 3D model of the car class as input.
[0098] S2: Using a Mask-RCNN-based network, obtain instance images of the car in the image and 3D point clouds of the car in the car depth image. The specific details of step S2 are as follows:
[0099] S21: Use the Mask-RCNN network to detect car objects in the image and obtain the instance mask.
[0100] S22: Based on the mask, crop out the car object from the image and resize the cropped image to I. o ∈R H×W×3 .
[0101] S23: Based on the depth image obtained from the Mask-RCNN network, crop out the corresponding positions of the car, and generate the car's point cloud based on the camera intrinsic parameter matrix. Randomly sample N from the point cloud. o Each point serves as a scene point cloud.
[0102] S3: Feature extraction is performed on the obtained car instance images and car point clouds using PSPNet and PointNet++ networks, respectively. The specific details of step S3 are as follows:
[0103] S31: The image I obtained in step S2 o And point cloud P o The images are then fed into PSPNet and PointNet++ networks respectively to extract features, thus obtaining image features. and point cloud features F o ∈R N×d .
[0104] S4: The extracted instance features are fused using a Transformer-based fusion module. The features are encoded using FSAM, and then decoded and fused using both FSAM and FCAM modules to obtain the global instance features. The specific details of step S4 are as follows:
[0105] S41: The Transformer-based fusion module consists of an encoder and a decoder. The encoder process uses the FSAM module to encode the feature maps, while the decoder process uses both the FSAM and FCAM modules to fuse features from different sources.
[0106] S42: Image features obtained from S31 and point cloud features F o ∈R N×d It uses a series of MLPs to map features into queries, keys, and values required by the FSAM module.
[0107]
[0108] Where q, k, and v represent query, key, and value, respectively, and E represents the parameters used in the encoder process.
[0109] S43: Based on the parameters obtained in S42, use the FSAM module to generate encoded features, and obtain...
[0110]
[0111] Where FSAM represents the FSAM module, satisfying...
[0112]
[0113] Where d k Let k be the dimension.
[0114] S44: Obtain Based on this, Introducing a decoder and using FSAM for feature extraction yields...
[0115]
[0116] Where q, k, and v represent query, key, and value, respectively, and D represents the parameters used in the encoder process.
[0117] S45: Obtained from S44 Use the FCAM module to fuse RGB images and point cloud features.
[0118]
[0119]
[0120] Where q, k, and v represent query, key, and value respectively, and FCAM represents the FCAM module, satisfying...
[0121]
[0122] In the formula, i,j∈{rgb,o},i=j.
[0123] S46: A decoder composed of S44 and S45, which obtains features. Therefore, RGB images and point cloud features can be complementaryly fused to obtain F. rgb ,F o The feature maps are then concatenated along the channel dimension to obtain instance features. Finally, the MLP should generate global features for the instance.
[0124]
[0125] in Representing local features of an instance, This indicates concatenation along the channel dimension. This represents the global characteristics of the generated instance.
[0126] S5: The global shape features of the input 3D car model are extracted using GSENet based on ResNet-1 network, and then the PointNet++ network is used to extract features to obtain category-level features. The specific content of step S5 is as follows:
[0127] S51: Input the 3D car model input in step S12 into GSENet to extract the global shape G of the car category. s ∈R n×3 .
[0128] S52: Transfer the 3D model features G obtained in step S51 to... s The input is fed into the PointNet++ network to extract features, resulting in point cloud features.
[0129] S6: By inputting the obtained global shape features and instance features into the prior adaptive module, new class-level global features are obtained. Then, the attention matrix is obtained by fusing the class-level global features and instance global features using a Transformer-based fusion module. The specific content of step S6 is as follows:
[0130] S61: The instance local features F obtained in step S46 are... rgb ,F o and the global feature F from step S52 rIntroduced into the prior adaptive module, F r F o F rgb Using these as queries, keys, and values respectively, prior adaptive semantic features are obtained through multi-head attention.
[0131]
[0132] S = Concat(Y) (1) ,Y (2) ,...,Y (m) (17)
[0133] in m represents the index of multi-head attention. This represents the prior adaptive semantic features.
[0134] S62: Combine the obtained prior adaptive semantic features S and point cloud features F r The system concatenates the data to generate category-level local features, and finally uses an MLP layer to generate category-level global features based on the obtained category-level local features.
[0135]
[0136] in Represents category-level local features. This represents a category-level global feature.
[0137] S63: The category-level global features obtained in S62 and the instance-level global features obtained in S46 are fused using a Transformer-based fusion module, and finally concatenated along the channel dimension.
[0138] Generate global features
[0139]
[0140] S64: In order to make full use of local features and make pose estimation more accurate, the global features obtained in S62 are concatenated with the instance local features, and finally an attention matrix is generated using the fused features.
[0141]
[0142] in This represents the attention matrix.
[0143] S7: Introduce the attention matrix obtained in S6 into the Transformer-based attention module to transform the features obtained in S5 in both geometric space and feature space, obtaining the rotation angle, size, and translation. The specific content of step S7 is as follows:
[0144] S71: To enable the network to effectively handle the size and shape of the target object, the features F obtained in step S5 are processed. r MLP layers are applied in the feature space and geometric space to obtain features in both spaces.
[0145] S r =MLP(F r ) (twenty two)
[0146] S f =MLP(F r ) (twenty three)
[0147] in Let represent the prior point cloud in geometric space and the prior point cloud in feature space, respectively.
[0148] S72: In the obtained S r S f The feature map is obtained by applying an MLP layer. It also generates queries, keys, and values for attention modules in geometric space and feature space.
[0149]
[0150] in Representing queries, keys, and values in the feature space.
[0151] This represents the query, key, and value in the geometric space. The obtained query, key, and value are then divided into several headers to obtain...
[0152] S73: Based on the attention matrix obtained in step S6, combine it with the attention module in geometric space to obtain S... r The deformation yields the deformed S' r .
[0153]
[0154] Where D K ∈R (C / h) Indicates channel dimension, S represents r The deformation offset. This represents the deformed prior point cloud.
[0155] S74: Similar to step S73, in the feature space, combine it with the attention module to obtain S. f The deformation yields the deformed S' f .
[0156]
[0157] Where D K ∈R (C / h) Indicates channel dimension, S represents f The deformation offset. This represents the deformed prior point cloud.
[0158] S75: The deformed prior point cloud is connected with local features, and the rotation, translation and size of the object are generated using MLP layers.
[0159]
[0160] Where R,t,s represents the rotation angle, size, and translation of the predicted object.
Claims
1. A method for estimating the pose of an autonomous vehicle, characterized in that, The method includes: S1: Color images of the road during vehicle movement, obtained from the perspective of the car's camera; use the color images and 3D models of similar cars as input data for the model. S2: Using a Mask-RCNN-based network, obtain instance images of cars in the image and 3D point clouds of cars in the car depth image; S3: Use PSPNet and PointNet++ networks to extract features from the obtained car instance images and car point clouds, respectively, to obtain the original image features of the instances. and original point cloud features of the instance ; S4: The instance features extracted above are fused by the Transformer-based fusion module, the features are encoded using FSAM, and the different features are decoded and fused by combining the FSAM and FCAM modules to obtain the global features of the instance; The specific details of step S4 are as follows: S41: The Transformer-based fusion module consists of an encoder and a decoder. The encoder process uses the FSAM module to encode the feature maps, and the decoder process uses both the FSAM module and the FCAM module to fuse features from different sources. S42: Image features obtained from S3 and point cloud features It uses a series of MLPs to map features into queries, keys, and values required by the FSAM module; (1) (2) in , , These represent the query, key, and value, respectively. This indicates the parameters used during the encoder process; S43: Based on the parameters obtained in S42, use the FSAM module to generate coded features, and obtain... ;in (3) Where FSAM represents the FSAM module, satisfying... (4) in for The dimension; S44: Obtain Based on this, it is introduced into the decoder, and FSAM is used for feature extraction to obtain... ;in (5) (6) (7) Indicates the parameters used during the decoder process; S45: Obtained from S44 The FCAM module is used to fuse RGB images and point cloud features, where (8) (9) (10) in , , These represent query, key, and value, respectively. FCAM represents the FCAM module, which satisfies... (11) In the formula ; S46: A decoder composed of S44 and S45, which obtains features. , Therefore, complementary fusion of image and point cloud features is performed to obtain instance fused image features. Instance fusion of point cloud features The feature maps are then concatenated along the channel dimension to obtain the instance local features. Finally, MLP is applied to generate global features for the instances. ; (12) (13) (14) (15) in Representing local features of an instance, This indicates concatenation along the channel dimension. Represents the global characteristics of the generated instance; S5: Use GSENet to extract the global shape features G of the input 3D model. s The PointNet++ network was used to extract features, resulting in point cloud features. ; S6： S61: Fuse the instance image features obtained in step S46 Instance fusion of point cloud features and the point cloud features obtained from step S5 Introduced into the prior adaptive module; (16) (17) in , , ; Index representing multi-head attention: , For the number of heads to focus on, , , The first The query, key, and value of each element can be used to learn the weight matrix; S62: Obtain the prior adaptive semantic features and point cloud features The system concatenates the data to generate category-level local features, and finally uses an MLP layer to generate category-level global features based on these local features. (18) (19) in Represents category-level local features. Represents category-level global features; S63: The category-level global features obtained in S62 and the instance-level global features obtained in S46 are fused using a Transformer-based fusion module, and finally concatenated along the channel dimension to generate the global features. ; (20) S64: The global features obtained in S63 are concatenated with the instance fusion image features and the instance fusion point cloud features, and finally an attention matrix is generated using the fused features. (21) in Represents the attention matrix; S7: Introduce the attention matrix obtained in S64 into the Transformer-based attention module to generate the geometric space prior point cloud from the point cloud features obtained in step S5. and feature space prior point cloud The deformation is performed to obtain the rotation angle, size, and translation.
2. The method for estimating the pose of an autonomous vehicle according to claim 1, characterized in that: The specific details of step S1 are as follows: S11: Color images of the road during vehicle movement, captured from the perspective of a car camera; the image resolution (width and height) is uniformly set to... ; S12: Use a known 3D model of the car interior as input.
3. The method for estimating the pose of an autonomous vehicle according to claim 1, characterized in that, The specific details of step S2 are as follows: S21: Use the Mask-RCNN network to detect car objects in the image and obtain the instance mask; S22: Based on the mask, crop out the car object from the image and resize the cropped image. ; S23: Based on the depth image obtained from the Mask-RCNN network, crop out the corresponding positions of the car, generate a point cloud of the car based on the camera intrinsic parameter matrix, and randomly sample from the point cloud. Each point serves as a scene point cloud. .
4. The method for estimating the pose of an autonomous vehicle according to claim 1, characterized in that, The specific details of step S7 are as follows: S71: In order for the network to effectively process the size and shape of the target object, the point cloud features obtained in step S5 are processed. MLP layers are applied to both the feature space and the geometric space to obtain features in both spaces. (22) (23) in , These represent the prior point cloud in geometric space and the prior point cloud in feature space, respectively. S72: In the obtained , The feature map is obtained by applying an MLP layer. , It generates queries, keys, and values for attention modules in the geometric and feature spaces; (24) (25) in Representing queries, keys, and values in the feature space. Represents queries, keys, and values in geometric space; The obtained query, key, and value are divided into several headers to obtain... , ; S73: Based on the attention matrix obtained in step S6, combine it with the attention module in geometric space to... Deformation yields the deformed ; (26) (27) in Indicates channel dimension, express The deformation offset. Represents the deformed prior point cloud; S74: Similar to step S73, in the feature space, combine it with the attention module to... Deformation yields the deformed ; (28) (29) in Indicates channel dimension, express The deformation offset. Represents the deformed prior point cloud; S75: The deformed prior point cloud is connected with the instance fusion image features and the instance fusion point cloud features, and the rotation, translation and size of the object are generated using the MLP layer; (30) in It indicates the predicted rotation angle, size, and translation of an object.