Self-driving vehicle pose estimation method

Abstract

The invention provides an automatic driving vehicle pose estimation method. The method comprises the following steps: firstly, acquiring a road color picture by using an automobile camera, and inputting the road color picture into a Mask-RCNN-based network in combination with a 3D automobile model to obtain an automobile instance image and a 3D point cloud; then, features are extracted through a PSPNet network and a PointNet + + network, feature fusion is carried out through a Transform fusion module, an FSAM module and an FCAM module, and global features are obtained; meanwhile, global and category-level features of the 3D automobile model are extracted by using a ResNet-1 network GSENet and a PointNet + + network. And inputting the features into an adaptive module to obtain new global features, and fusing the new global features with instance global features to generate an attention matrix. And finally, deforming the priori point cloud by using an attention module and an attention matrix based on Transform so as to estimate the pose of the vehicle. According to the method, the local and global features of the automobile can be effectively extracted, and the accuracy of pose estimation is improved through feature fusion and the attention matrix.

Description

Technical Field

[0001] The present invention belongs to the research field of autonomous driving and computer vision, and specifically relates to a method for estimating the posture of an autonomous driving vehicle. Background Art

[0002] With the rapid development of artificial intelligence and deep learning technology in recent years, pose estimation methods based on deep learning have become increasingly important. Pose estimation refers to accurately inferring the position, orientation, and posture of a target object based on input image or video data. This technology has a wide range of applications in computer vision, robotics, augmented reality, and other fields.

[0003] The development of deep learning technology has brought great improvements to pose estimation. Traditional pose estimation methods usually rely on manually designed features and models, which perform poorly in complex scenes and changing situations. Deep learning-based methods can automatically learn features and patterns from large amounts of data, thus better adapting to complex scenes and changes.

[0004] In order to solve the problem of fitting the prior point cloud to the target object and perform shape deformation, it is crucial to ensure that sufficient information is available for the deformation process. Traditional object pose estimation methods include directly regressing the target final pose using RGB-D images and directly connecting RGB images and point cloud feature information to regress the object pose. However, these traditional methods ignore the difference between instance features and category features and the difference between RGB images and point cloud features, resulting in reduced accuracy of object fitting during object pose estimation. At the same time, performance will be affected in severely occluded scenes.

[0005] Therefore, the pose estimation method of the above technology ignores the differences between features, which will lead to insufficient extraction of object information in complex backgrounds. At the same time, the accuracy will decrease during the fitting process of the target. The accuracy in actual use needs to be improved, so it is urgently needed. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the present invention proposes a method for estimating the posture of an autonomous driving vehicle, which can better estimate the posture of objects in an image even in an unfamiliar environment.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0008] A method for estimating the posture of an autonomous driving vehicle, the key technology of which is to follow the following steps:

[0009] A method for estimating a posture of an autonomous driving vehicle, characterized in that the method comprises:

[0010] S1: A color picture of the road on which the vehicle is traveling, obtained from the perspective of the car camera. The color picture and the 3D model of the same type of car are used as the input data of the model;

[0011] S2: Obtain the instance image of the car in the image and the 3D point cloud of the car in the car depth image through the Mask-RCNN based network;

[0012] S3: Use PSPNet and PointNet++ networks to extract features from the obtained car instance image and car point cloud respectively;

[0013] S4: The instance features extracted above are fused through the Transformer-based fusion module, the features are encoded using FSAM, and the FSAM and FCAM modules are combined to decode and fuse different features to obtain the instance global features;

[0014] S5: The global shape features of the input 3D car model are extracted using the GSENet network based on the ResNet-1 network, and then the features are extracted using the PointNet++ network to obtain the category-level features;

[0015] S6: The new class-level global features are obtained by inputting the obtained global shape features and instance features into the prior adaptation module. Secondly, the attention matrix is ​​obtained by fusing the class-level global features and instance global features using the Transformer-based fusion module;

[0016] S7: The attention matrix obtained in S6 is introduced into the Transformer-based attention module to deform the features obtained in S5 in the geometric space and feature space to obtain the rotation angle, size, and translation.

[0017] Further: the specific content of step S1:

[0018] S11: A color image of a road being driven by a vehicle obtained from the perspective of a car camera. The image resolution width and height are uniformly set to W and H.

[0019] S12: Take a known 3D model of the car class as input.

[0020] Further: the specific content of step S2:

[0021] S21: Use the Mask-RCNN network to detect car objects in the image and obtain instance masks.

[0022] S22: Based on the mask, the car object in the image is cropped and the cropped image size is adjusted to I o ∈R H×W×3 .

[0023] S23: Based on the depth image obtained by the Mask-RCNN network, the corresponding position of the car is cropped, and the point cloud of the car is generated according to the camera intrinsic parameter matrix. N points are randomly sampled from the point cloud. o Points as scene point cloud

[0024] Further: the specific content of step S3:

[0025] S31: image I obtained in step S2 o and point cloud P o Input into PSPNet and PointNet++ networks respectively to extract features and obtain image features and point cloud features

[0026] Further: the specific content of step S4:

[0027] S41: The Transformer-based fusion module consists of an encoder and a decoder, where the encoder process uses the FSAM module to encode the feature map, and the decoder process uses the FSAM module and the FCAM module to fuse features from different sources.

[0028] S42: Image features obtained according to S31 and point cloud features F o ∈R N×d ,Through a series of MLPs, the features are mapped into the queries, keys, and values ​​required by the FSAM module.

[0029]

[0030] Where q, k, v represent query, key and value respectively, and E represents the parameters used in the encoder process.

[0031] S43: Based on the parameters obtained in S42, the FSAM module is used to generate coding features, and the

[0032]

[0033] FSAM stands for FSAM module, which satisfies

[0034]

[0035] where d k is the dimension of k.

[0036] S44: Get On the basis of Introduce the decoder and use FSAM for feature extraction to obtain

[0037]

[0038] Where q, k, v represent query, key and value respectively, and D represents the parameters used in the encoder process.

[0039] S45: According to S44 Use the FCAM module to fuse RGB images and point cloud features.

[0040]

[0041] Where q, k, v represent query, key, and value respectively, and FCAM represents FCAM module.

[0042]

[0043] In the formula, i,j∈{rgb,o},i=j.

[0044] S46: Decoder composed of S44 and S45 to obtain features Therefore, the RGB image and point cloud features can be complementary fused to obtain F rgb ,F o . And concatenate the feature maps according to the channel dimension to obtain instance features Finally, MLP should generate instance global features

[0045]

[0046] in represents the local features of the instance, represents channel dimension splicing, Represents the generated instance global features.

[0047] Further: the specific content of step S5:

[0048] S51: Input the 3D car model input in step S12 into GSENet to extract the global shape G of the car category s ∈R n×3 .

[0049] S52: The 3D model feature G obtained in step S51 is s Input into PointNet++ network to extract features and obtain point cloud features

[0050] Further: the specific content of step S6:

[0051] S61: The instance local feature F obtained in step S46 is rgb ,Fo and the global feature F obtained by step S52 r Introduced into the prior adaptive module, F r , F o , F rgb As query, key and value respectively, the prior adaptive semantic features are obtained by using multi-head attention.

[0052]

[0053] S = Concat(Y (1) ,Y (2) ,...,Y (m) ) (17)

[0054] in m represents the index of multi-head attention, Represents a priori adaptive semantic features.

[0055] S62: Adapt the obtained prior semantic features S and point cloud features F r The category-level local features are generated by splicing, and finally, based on the obtained category-level local features, the MLP layer is used to generate category-level global features.

[0056]

[0057] in represents the category-level local features, Represents category-level global features.

[0058] S63: The category-level global features obtained in S62 and the instance-level global features obtained in S46 are fused through a Transformer-based fusion module, and finally the global features are generated by splicing them in the channel dimension.

[0059]

[0060] S64: In order to make full use of local features and make pose estimation more accurate, the global features obtained in S62 are concatenated with the local features of the instance, and finally an attention matrix is ​​generated using the fused features.

[0061]

[0062] in represents the attention matrix.

[0063] Further: the specific content of step S7:

[0064] S71: In order to enable the network to effectively process the size and shape of the target object, the feature F obtained in step S5 is rApplying MLP layers in feature space and geometric space obtains features in both spaces.

[0065] S r =MLP(F r ) (twenty two)

[0066] S f =MLP(F r ) (twenty three)

[0067] in They represent the prior point cloud of the geometric space and the prior point cloud of the feature space respectively.

[0068] S72: After getting S r , S f Apply the MLP layer to get the feature map And generate queries, keys and values ​​for the attention module in geometric space and feature space.

[0069]

[0070] in represents the query, key and value of the feature space,

[0071] Represents the query, key and value of the geometric space. The obtained query, key and value are divided into several heads to obtain

[0072] S73: Based on the attention matrix obtained in step S6, combine it with the attention module in the geometric space to S r Deformation to obtain the deformed S' r .

[0073]

[0074] Where D K ∈R (C / h) represents the channel dimension, Indicates S r The deformation offset, Represents the deformed prior point cloud.

[0075] S74: Same as step S73, in the feature space, combine it with the attention module to S f Deformation to obtain the deformed S' f .

[0076]

[0077] Where D K ∈R (C / h) represents the channel dimension, Indicates S f The deformation offset, Represents the deformed prior point cloud.

[0078] S75: The deformed prior point cloud is connected with the local features, and the MLP layer is used to generate the rotation, translation and size of the object.

[0079]

[0080] Where R,t,s represents the predicted object's rotation angle, size, and translation.

[0081] The beneficial effects of the present invention are:

[0082] This patent proposes a method for posture estimation of autonomous driving vehicles that can more effectively extract local and global features of RGB images and point clouds, and use a feature fusion module to achieve complementarity and fusion between different features. At the same time, it can generate an attention matrix with the obtained features, deform the prior point cloud, better fit the target object, and ultimately improve the accuracy of posture estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 Flow chart of the method of the present invention

[0084] Figure 2 Schematic diagram of feature fusion module

[0085] Figure 3 Schematic diagram of the attention matrix generation module

[0086] Figure 4 FSAM module diagram

[0087] Figure 5 FCAM module schematic

[0088] Figure 6 Schematic diagram of the process of prior point cloud deformation

[0089] Figure 7 GSENet network process diagram

[0090] Figure 8 Schematic diagram of the prior adaptive module process DETAILED DESCRIPTION

[0091] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0092] When implementing: Figures 1 to 8 As shown, in this embodiment, the color image obtained during vehicle driving and the prior point cloud of the vehicle are used as the data set for pose estimation, specifically:

[0093] A method for estimating the pose of an autonomous driving vehicle, combining Figure 1 It can be seen that the feature includes the following steps:

[0094] 1. A method for estimating the pose of an autonomous driving vehicle, characterized in that the method comprises:

[0095] S1: A color picture of the road on which the vehicle is traveling is obtained from the perspective of the car camera. The color picture and the 3D model of the same type of car are used as input data of the model. The specific content of step S1 is:

[0096] S11: A color image of a road being driven by a vehicle obtained from the perspective of a car camera. The image resolution width and height are uniformly set to W and H.

[0097] S12: Take a known 3D model of the car class as input.

[0098] S2: Obtain the instance image of the car in the image and the 3D point cloud of the car in the car depth image through the Mask-RCNN based network. The specific content of step S2 is:

[0099] S21: Use the Mask-RCNN network to detect car objects in the image and obtain instance masks.

[0100] S22: Based on the mask, the car object in the image is cropped and the cropped image size is adjusted to I o ∈R H×W×3 .

[0101] S23: Based on the depth image obtained by the Mask-RCNN network, the corresponding position of the car is cropped, and the point cloud of the car is generated according to the camera intrinsic parameter matrix. N points are randomly sampled from the point cloud. o Points as scene point cloud

[0102] S3: Use PSPNet and PointNet++ networks to extract features from the obtained car instance image and car point cloud respectively. The specific content of step S3 is:

[0103] S31: image I obtained in step S2 o and point cloud P o Input into PSPNet and PointNet++ networks respectively to extract features and obtain image features and point cloud features F o ∈R N×d .

[0104] S4: The instance features extracted above are fused through the Transformer-based fusion module, the features are encoded using FSAM, and the FSAM and FCAM modules are combined to decode and fuse different features to obtain the instance global features. The specific content of step S4 is as follows:

[0105] S41: The Transformer-based fusion module consists of an encoder and a decoder, where the encoder process uses the FSAM module to encode the feature map, and the decoder process uses the FSAM module and the FCAM module to fuse features from different sources.

[0106] S42: Image features obtained according to S31 and point cloud features F o ∈R N×d ,Through a series of MLPs, the features are mapped into the queries, keys, and values ​​required by the FSAM module.

[0107]

[0108] Where q, k, v represent query, key and value respectively, and E represents the parameters used in the encoder process.

[0109] S43: Based on the parameters obtained in S42, the FSAM module is used to generate coding features, and the

[0110]

[0111] FSAM stands for FSAM module, which satisfies

[0112]

[0113] where d k is the dimension of k.

[0114] S44: Get On the basis of Introduce the decoder and use FSAM for feature extraction to obtain

[0115]

[0116] Where q, k, v represent query, key and value respectively, and D represents the parameters used in the encoder process.

[0117] S45: According to S44 Use the FCAM module to fuse RGB images and point cloud features.

[0118]

[0119]

[0120] Where q, k, v represent query, key, and value respectively, and FCAM represents FCAM module.

[0121]

[0122] In the formula, i,j∈{rgb,o},i=j.

[0123] S46: Decoder composed of S44 and S45 to obtain features Therefore, the RGB image and point cloud features can be complementary fused to obtain F rgb ,F o . And concatenate the feature maps according to the channel dimension to obtain instance features Finally, MLP should generate instance global features

[0124]

[0125] in represents the local features of the instance, represents channel dimension splicing, Represents the generated instance global features.

[0126] S5: The global shape features of the input 3D car model are extracted using the GSENet network based on the ResNet-1 network, and then the features are extracted using the PointNet++ network to obtain the category-level features. The specific content of step S5 is as follows:

[0127] S51: Input the 3D car model input in step S12 into GSENet to extract the global shape G of the car category s ∈R n×3 .

[0128] S52: The 3D model feature G obtained in step S51 is s Input into PointNet++ network to extract features and obtain point cloud features

[0129] S6: The new class-level global features are obtained by inputting the obtained global shape features and instance features into the prior adaptive module. Secondly, the attention matrix is ​​obtained by fusing the class-level global features and the instance global features using the Transformer-based fusion module. The specific content of step S6 is:

[0130] S61: The instance local feature F obtained in step S46 is rgb ,F o and the global feature F obtained by step S52 rIntroduced into the prior adaptive module, F r , F o , F rgb As query, key and value respectively, the prior adaptive semantic features are obtained by using multi-head attention.

[0131]

[0132] S = Concat(Y (1) ,Y (2) ,...,Y (m) ) (17)

[0133] in m represents the index of multi-head attention, Represents a priori adaptive semantic features.

[0134] S62: Adapt the obtained prior semantic features S and point cloud features F r The category-level local features are generated by splicing, and finally, based on the obtained category-level local features, the MLP layer is used to generate category-level global features.

[0135]

[0136] in represents the category-level local features, Represents category-level global features.

[0137] S63: The category-level global features obtained in S62 and the instance-level global features obtained in S46 are fused through a Transformer-based fusion module, and finally concatenated through the channel dimension

[0138] Generate global features

[0139]

[0140] S64: In order to make full use of local features and make pose estimation more accurate, the global features obtained in S62 are concatenated with the local features of the instance, and finally an attention matrix is ​​generated using the fused features.

[0141]

[0142] in represents the attention matrix.

[0143] S7: Introduce the attention matrix obtained in S6 into the attention module based on Transformer to deform the features obtained in S5 in the geometric space and feature space to obtain the rotation angle, size, and translation. The specific content of step S7 is:

[0144] S71: In order to enable the network to effectively process the size and shape of the target object, the feature F obtained in step S5 is r Applying MLP layers in feature space and geometric space obtains features in both spaces.

[0145] S r =MLP(F r ) (twenty two)

[0146] S f =MLP(F r ) (twenty three)

[0147] in They represent the prior point cloud of the geometric space and the prior point cloud of the feature space respectively.

[0148] S72: After getting S r , S f Apply the MLP layer to get the feature map And generate queries, keys and values ​​for the attention module in geometric space and feature space.

[0149]

[0150] in represents the query, key and value of the feature space,

[0151] Represents the query, key and value of the geometric space. The obtained query, key and value are divided into several heads to obtain

[0152] S73: Based on the attention matrix obtained in step S6, combine it with the attention module in the geometric space to S r Deformation to obtain the deformed S' r .

[0153]

[0154] Where D K ∈R (C / h) represents the channel dimension, Indicates S r The deformation offset, Represents the deformed prior point cloud.

[0155] S74: Same as step S73, in the feature space, combine it with the attention module to S f Deformation to obtain the deformed S' f .

[0156]

[0157] Where D K ∈R (C / h) represents the channel dimension, Indicates S f The deformation offset, Represents the deformed prior point cloud.

[0158] S75: The deformed prior point cloud is connected with the local features, and the MLP layer is used to generate the rotation, translation and size of the object.

[0159]

[0160] Where R,t,s represents the predicted object's rotation angle, size, and translation.

Claims

1. A method for estimating the pose of an autonomous driving vehicle, characterized in that: The method comprises: S1: A color picture of the road on which the vehicle is traveling, obtained from the perspective of the car camera. The color picture and the 3D model of the same type of car are used as the input data of the model; S2: Obtain the instance image of the car in the image and the 3D point cloud of the car in the car depth image through the Mask-RCNN based network; S3: Use PSPNet and PointNet++ networks to extract features from the obtained car instance image and car point cloud respectively; S4: The instance features extracted above are fused through the Transformer-based fusion module, the features are encoded using FSAM, and the FSAM and FCAM modules are combined to decode and fuse different features to obtain the instance global features; S5: The global shape features of the input 3D model are extracted using the ResNet-1 network GSENet, and the features are extracted using the PointNet++ network to obtain category-level features; S6: The new class-level global features are obtained by inputting the obtained global shape features and instance features into the prior adaptation module. Secondly, the attention matrix is ​​obtained by fusing the class-level global features and instance global features using the Transformer-based fusion module; S7: The attention matrix obtained in S6 is introduced into the Transformer-based attention module to deform the features obtained in S5 in the geometric space and feature space to obtain the rotation angle, size, and translation.

2. The method for estimating the pose of an autonomous driving vehicle according to claim 1, characterized in that: Specific content of step S1: S11: A color image of a road being driven by a vehicle obtained from the perspective of a car camera. The image resolution width and height are uniformly set to W and H. S12: Take the known 3D model of the car class as input.

3. The method for estimating the pose of an autonomous driving vehicle according to claim 1, characterized in that: Specific content of step S2: S21: Use the Mask-RCNN network to detect car objects in the image and obtain instance masks. S22: Based on the mask, the car object in the image is cropped and the cropped image size is adjusted to I o ∈R H×W×3 . S23: Based on the depth image obtained by the Mask-RCNN network, the corresponding position of the car is cropped, and the point cloud of the car is generated according to the camera intrinsic parameter matrix. N points are randomly sampled from the point cloud. o points as the scene point cloud P o ∈R No×3 .

4. The method for estimating the pose of an autonomous driving vehicle according to claim 1, characterized in that: Specific content of step S3: S31: image I obtained in step S2 o and point cloud P o Input into PSPNet and PointNet++ networks respectively to extract features and obtain image features and point cloud features F o ∈R N×d .

5. The method for estimating the pose of an autonomous driving vehicle according to claim 1, characterized in that: Specific content of step S4: S41: The Transformer-based fusion module consists of an encoder and a decoder, where the encoder process uses the FSAM module to encode the feature map, and the decoder process uses the FSAM module and the FCAM module to fuse features from different sources. S42: Image features obtained according to S31 and point cloud features F o ∈R N×d ,Through a series of MLPs, the features are mapped into the queries, keys, and values ​​required by the FSAM module. Where q, k, v represent query, key and value respectively, and E represents the parameters used in the encoder process. S43: Based on the parameters obtained in S42, the FSAM module is used to generate coding features, and the FSAM stands for FSAM module, which satisfies where d k is the dimension of k. S44: Get On the basis of Introduce the decoder and use FSAM for feature extraction to obtain Where q, k, v represent query, key and value respectively, and D represents the parameters used in the encoder process. S45: According to S44 Use the FCAM module to fuse RGB images and point cloud features. Where q, k, v represent query, key, and value respectively, and FCAM represents FCAM module. In the formula, i,j∈{rgb,o},i=j. S46: Decoder composed of S44 and S45 to obtain features Therefore, the RGB image and point cloud features can be complementary fused to obtain F rgb ,F o . And concatenate the feature maps according to the channel dimension to obtain instance features Finally, MLP should generate instance global features in represents the local features of the instance, represents channel dimension splicing, Represents the generated instance global features.

6. The method for estimating the pose of an autonomous driving vehicle according to claim 1, characterized in that: Specific content of step S5: S51: Input the 3D car model input in step S12 into GSENet to extract the global shape of the car category S52: The 3D model feature G obtained in step S51 is s Input into PointNet++ network to extract features and obtain point cloud features 7. The method for estimating the pose of an autonomous driving vehicle according to claim 1, characterized in that: Specific content of step S6: S61: The instance local feature F obtained in step S46 is rgb ,F o and the global feature F obtained by step S52 r Introduced into the prior adaptive module, F r , F o , F rgb As query, key and value respectively, the prior adaptive semantic features are obtained by using multi-head attention. S=Concat(And (1) ,AND (2) ,...,AND (m) ) (17) in m represents the index of multi-head attention, Represents a priori adaptive semantic features. S62: Adapt the obtained prior semantic features S and point cloud features F r The category-level local features are generated by splicing, and finally, based on the obtained category-level local features, the MLP layer is used to generate category-level global features. in represents the category-level local features, Represents category-level global features. S63: The category-level global features obtained in S62 and the instance-level global features obtained in S46 are fused through a Transformer-based fusion module, and finally the global features are generated by splicing them in the channel dimension. S64: In order to make full use of local features and make pose estimation more accurate, the global features obtained in S62 are concatenated with the local features of the instance, and finally an attention matrix is ​​generated using the fused features. in represents the attention matrix.

8. The method for estimating the pose of an autonomous driving vehicle according to claim 1, characterized in that: Specific content of step S7: S71: In order to enable the network to effectively process the size and shape of the target object, the feature F obtained in step S5 is r Applying MLP layers in feature space and geometric space obtains features in both spaces. S r =MLP(F r ) (22) S f =MLP(F r ) (23) in They represent the prior point cloud of the geometric space and the prior point cloud of the feature space respectively. S72: After getting S r , S f Apply the MLP layer to get the feature map And generate queries, keys and values ​​for the attention module in geometric space and feature space. in represents the query, key and value of the feature space, Represents the query, key and value of the geometric space. The obtained query, key and value are divided into several heads to obtain S73: Based on the attention matrix obtained in step S6, combine it with the attention module in the geometric space to S r Deformation to obtain the deformed S' r . Where D K ∈R (C / h) represents the channel dimension, Indicates S r The deformation offset, Represents the deformed prior point cloud. S74: Same as step S73, in the feature space, combine it with the attention module to S f Deformation to obtain the deformed S' f . Where D K ∈R (C / h) represents the channel dimension, Indicates S f The deformation offset, Represents the deformed prior point cloud. S75: The deformed prior point cloud is connected with the local features, and the MLP layer is used to generate the rotation, translation and size of the object. Where R,t,s represents the predicted object's rotation angle, size, and translation.

Images