Vehicle pose estimation method based on semantic feature point detection

Abstract

The present application relates to the technical field of image processing, and discloses a vehicle pose estimation method based on semantic feature point detection, comprising the following steps: step S1: constructing a virtual vehicle 3D model, and selecting points on the vehicle model as 3D key points; step S2: collecting vehicle pictures in real scene road pictures, detecting the 2D bounding box of the vehicle picture through the YOLOv8 model, and extracting the vehicle picture in the 2D bounding box; step S3: classifying the vehicle of the extracted vehicle picture through the VisionTransformer technology. The present application can realize efficient and accurate vehicle pose estimation in a complex actual road scene without additional parameters and manual annotation, and is suitable for multiple fields such as automatic driving, intelligent transportation, augmented reality, etc., and has a wide prospect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a vehicle pose estimation method based on semantic feature point detection. BACKGROUND

[0002] In modern cities, traffic monitoring systems have become an important means of maintaining traffic order and improving road safety. Vehicle pose estimation, as an important part of intelligent transportation and vehicle autonomous driving, has received widespread attention. The core goal of vehicle pose estimation is to accurately obtain the position information and attitude parameters of the vehicle, including the position coordinates, orientation, and rotation angle of the vehicle, through analysis of image or sensor data.

[0003] Traditional vehicle pose estimation methods are usually divided into two categories. One is based on deep learning technology. This type of technology mainly predicts vehicles by training deep learning models, which are usually divided into two categories. One class trains the model according to the labeled data to directly obtain the vehicle pose information; the other class detects the key points of the vehicle, estimates the pose information through perspective projection according to the camera parameters, or matches the vehicle model to estimate the pose.

[0004] The other is based on precise instruments such as sensors to estimate the pose, which fuses multiple sensors or filtering results to iteratively update the vehicle state according to the vehicle dynamic model to estimate the vehicle pose.

[0005] Traditional deep learning-based pose estimation methods have certain shortcomings:

[0006] ① The current deep learning method for pose estimation usually adopts a supervised method, which requires a large amount of samples for model training. It is very difficult to label three-dimensional pose information in 2D images, often requiring a lot of manpower and resources.

[0007] ② Once the deep learning model is trained, the types of vehicles that the model can detect are fixed. Vehicles that exceed the training samples will not be correctly identified.

[0008] ③ Key point detection methods require information support, such as camera parameters and actual width-height ratio.

[0009] The method based on precise instruments has the following shortcomings:

[0010] ① The vehicle pose estimation method based on precise instruments has the advantage of high precision, but also has the disadvantages of high cost, complex equipment setup and operation, sensitivity to environmental conditions, and possible impact on vehicle weight, energy consumption, and data delay.

[0011] The methods are often only directed to a single vehicle to be observed, and are limited by visibility, hardware failure and signal interference, and may cause inaccurate or failed pose estimation in specific cases. SUMMARY

[0012] To solve the technical problems in the prior art, the application provides a vehicle pose estimation method based on semantic feature point detection.

[0013] The application adopts the following technical scheme: a vehicle pose estimation method based on semantic feature point detection, comprising the following steps:

[0014] Step S1: a virtual vehicle 3D model is constructed, and points on the vehicle model are selected as 3D key points;

[0015] Step S2: vehicle pictures in real scene road pictures are collected, a 2D bounding box of the vehicle picture is detected through a YOLOv8 model, and the vehicle picture in the 2D bounding box is extracted;

[0016] Step S3: the vehicle in the extracted vehicle picture is classified through a VisionTransformer technology;

[0017] Step S4: a corresponding vehicle 3D model is selected according to the classified vehicle type;

[0018] Step S5: the YOLOv8 model is improved, the SWS layer is integrated into the convolutional layer in the backbone network and the Neck of the YOLOv8, and a BSAM layer is added after the SPPF layer of the backbone network;

[0019] Step S6: the improved YOLOv8 model is used to detect the vehicle 3D model, and the 3D key points are screened;

[0020] Step S7: the vehicle pose is solved through a multi-point perspective algorithm:

[0021] Firstly, the vehicle 3D model is rendered to obtain a synthetic image sample;

[0022] Then, the improved YOLOv8 model is trained using the synthetic sample;

[0023] According to the definition of the 3D semantic feature points, the 3D semantic feature points of the real vehicle image are labeled, the improved YOLOv8 model is fine-tuned, and the fine-tuned YOLOv8 model detects the 2D key points;

[0024] The screened 3D key points and the detected 2D key points are one-to-one corresponding, and the pose is solved through a multi-point perspective algorithm.

[0025] As a further improvement to the above solution, in step S1, a virtual vehicle 3D model is formed using 3D modeling software.

[0026] As a further improvement to the above scheme, the step S1 of selecting corner points as 3D key points is as follows:

[0027] From any vertex x0 on the virtual vehicle 3D model, there are k nearest neighbor points {x1, x2, ..., xk} within a sphere of radius R. k The covariance matrix of this point set is in

[0028] Calculate the eigenvalues ​​λ1, λ2, λ3 of M, where 0 ≤ λ1 ≤ λ2 ≤ λ3. It can reflect the curvature around the vertex. A threshold ε is set; if... Then x0 is determined to be a corner point.

[0029] As a further improvement to the above scheme, the 2D bounding box in step S2 includes the category of the detected target, the 2D coordinates of the center point of the 2D bounding box, and the length and width of the bounding box.

[0030] As a further improvement to the above scheme, the synthesized image sample in step S7 includes vehicle images, 3D feature points corresponding to 3D key points, and 2D feature point coordinate information.

[0031] Among them, 3D feature points are the three-dimensional coordinates of 3D key points in a world coordinate system established with the center of the virtual vehicle 3D model as the origin, and 2D feature points are the projected coordinates of 3D feature points on a 2D image.

[0032] As a further improvement to the above scheme, the training steps of the improved YOLOv8 model in step S7 are as follows:

[0033] Configure the conda environment required for YOLOv8 and install the relevant dependency libraries;

[0034] Change the 2D point annotation files in the synthesized samples to the format required by YOLOv8;

[0035] The dataset is divided into a training set, a validation set, and a test set in an 8:1:1 ratio.

[0036] In the YOLOv8 configuration file, change the learning rate, batch size, epoch size, and other training-related settings during model training.

[0037] Run the training script, specify the input folder and the output folder, the input folder contains the training set pictures, the verification set pictures and the 2D annotation file, and the output folder is used to save the weight output by the model.

[0038] As a further improvement of the above scheme, the step S7 defines the semantic features of the screened 3D key points and the corresponding 3D feature points and 2D feature points, and defines the name and type of the screened 3D key points and the corresponding 3D feature points and 2D feature points.

[0039] As a further improvement of the above scheme, the multi-point perspective algorithm in step S7 is:

[0040] Wherein, X i , Y i and Z i are the coordinates of the 3D key points, u i and v i are the coordinates of the 2D key points, and K is the camera intrinsic parameter, so as to realize the solution of R rotation matrix and t translation matrix.

[0041] As a further improvement of the above scheme, the YOLOv8 model includes Backbone, Neck and Head.

[0042] Compared with the prior art, the beneficial effects of the present application are:

[0043] The present application can realize efficient and accurate vehicle pose estimation in complex real road scenes without additional parameters and manual annotation, and is suitable for automatic driving, intelligent transportation, augmented reality and other fields, and has wide prospects. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 The flowchart of the vehicle pose estimation method based on semantic feature point detection provided by the present application;

[0045] Figure 2 The structure diagram of the YOLOv8 model provided by the present application;

[0046] Figure 3 The structure diagram of the improved YOLOv8 model provided by the present application;

[0047] Figure 4 The structure diagram of the Conv_SWS-1 layer-Conv_SWS-6 layer provided by the present application. DETAILED DESCRIPTION

[0048] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0049] Example 1:

[0050] Please combine Figure 1 The vehicle pose estimation method based on semantic feature point detection in this embodiment includes the following steps:

[0051] Step S1: Construct a virtual vehicle 3D model and select points on the vehicle model as 3D key points;

[0052] The process involves using 3D modeling software to create a virtual vehicle 3D model. This model can then be imported into software such as Unity, and corner points can be selected as 3D key points. The steps are as follows:

[0053] Select any vertex x0 on the virtual vehicle 3D model, and within a sphere of radius R, there are k nearest neighbor points {x1, x2, ..., xk}. k The covariance matrix of this point set is in

[0054]

[0055] Where x0, x1, x2, ..., x k Let M be the vertices on the virtual vehicle 3D model, and let M be the vertices passing through x0, x1, x2, ..., xn. k The covariance matrix calculated from the point set, where k is the number of nearest neighbors, and xi represents {x0, x1, ..., x...} k The three-dimensional coordinates of any point in {x0, x1, ..., x}, where u refers to the coordinates of any point in {x0, x1, ..., x}. k The average of these nearest neighbor sets, T, is denoted as the transpose of the matrix;

[0056] Calculate the eigenvalues ​​λ1, λ2, λ3 of M, where 0 ≤ λ1 ≤ λ2 ≤ λ3. It can reflect the curvature around the vertex. A threshold ε is set; if... Then x0 is determined to be a corner point; the threshold ε can reflect the curvature characteristics around the vertex, and can effectively distinguish vertices with curvature changes in local structures from those with relatively flat surfaces. For flat regions, the spatial distribution of vertices and their nearest neighbors is relatively uniform, resulting in smaller differences in the eigenvalues ​​of the covariance matrix; if The vertex is considered to have a large curvature around it, which meets the characteristics of a corner point, and is therefore selected as a 3D key point.

[0057] Wherein, λ1, λ2, λ3 are the eigenvalues calculated after the covariance matrix, λ1, λ2, λ3 represent the dispersion degree in different directions of the local geometry of the vertex, λ1 and λ3 represent the directions of the smallest and largest dispersion respectively;

[0058] Step S2: Collect the vehicle picture in the real scene road picture, detect the 2D bounding box of the vehicle picture through the YOLOv8 model, and extract the vehicle picture in the 2D bounding box;

[0059] Wherein, the 2D bounding box includes the category of the detection target, the 2D coordinates of the center point of the 2D bounding box, and the length and width of the bounding box;

[0060] Step S3: Classify the extracted vehicle picture through VisionTransformer technology, and divide it into sedan, SUV and truck;

[0061] Wherein, Vision Transformer (ViT) is an image classification model based on Transformer architecture. The network structure of ViT can be divided into several key parts. First, the input image is divided into fixed-size patches (such as 16x16 pixels), each patch is flattened and mapped to a fixed-dimensional embedding space through linear transformation. Next, position embedding is added to the embedding vector of each patch to provide the spatial position information of the patch in the image. Then, these patches are sent to the standard Transformer encoder, which includes multiple self-attention layers and feed-forward neural network layers. Self-attention layers can capture the global dependence between different patches, while feed-forward neural networks further process this information. The core of ViT is multiple stacked Transformer encoder layers, which improve the performance of image classification by capturing the relationship between patches in the image.

[0062] After processing by the Transformer encoder, ViT extracts a special [CLS] patch from the input sequence, which contains the global features of the entire image and serves as the basis for image classification. Next, the final representation of the [CLS] token is processed through a classification head to generate the final class prediction. Through the softmax function, the model can output the probability distribution of each class, and select the most likely class as the label of the image according to the probability;

[0063] Step S4: Select the corresponding vehicle 3D model according to the classified vehicle type;

[0064] Step S5: Improve the YOLOv8 model, integrate the SWS layer into the convolutional layer of the main network and Neck of YOLOv8, and add the BSAM layer after the SPPF layer of the main network;

[0065] Step S6: Detect the vehicle 3D model using the improved YOLOv8 model and screen the 3D key points;

[0066] Step S7: Solve the vehicle pose by the multi-point perspective algorithm:

[0067] First, render the vehicle 3D model to obtain the synthesized image samples;

[0068] Wherein, the vehicle 3D model rendering is to put the virtual 3D model into the virtual environment of unity for synthesis, and the synthesized image samples are the pictures of the vehicle at different positions and angles in the virtual environment;

[0069] Wherein, the synthesized image samples include vehicle pictures, 3D feature points corresponding to 3D key points and 2D feature point coordinate information;

[0070] Wherein, the 3D feature points are the three-dimensional coordinates of the 3D key points in the world coordinate system established with the center of the virtual vehicle 3D model as the origin, and the 2D feature points are the projection coordinates of the 3D feature points in the 2D image, and the 3D feature points and the 2D feature points are numbered to correspond to each other;

[0071] Then, use the synthesized samples to train the improved YOLOv8 model;

[0072] Wherein, the training steps of the improved YOLOv8 model are:

[0073] Configure the conda environment required by YOLOv8 and install the related dependent libraries;

[0074] Change the 2D point annotation file in the synthesized sample to the format required by YOLOv8;

[0075] Divide the data set into training set, validation set and test set according to the ratio of 8:1:1;

[0076] Change the learning rate, batch and epoch in the model training in the configuration file of YOLOv8 and other training related configurations;

[0077] Run the training script, specify the input folder and output folder, the input folder contains the training set pictures, validation set pictures and 2D annotation file, and the output folder is used to save the weight output by the model;

[0078] According to the definition of 3D semantic feature points, label the 3D semantic feature points of the real vehicle image, fine-tune the improved YOLOv8 model, and fine-tune the learning rate, batch_size and epoch parameters of the improved YOLOv8 model. The 2D key points detected by the improved YOLOv8 model are obtained;

[0079] wherein the semantic features of the screened 3D key points and the corresponding 3D feature points and 2D feature points are defined, and the names and types of the screened 3D key points and the corresponding 3D feature points and 2D feature points are defined;

[0080] The screened 3D key points and the detected 2D key points are one-to-one corresponding, and the pose is solved by a multi-point perspective algorithm.

[0081] wherein the multi-point perspective algorithm is:

[0082] wherein X i , Y i and Z i are the coordinates of the 3D key points, u i and v i are the coordinates of the 2D key points, and K is the camera intrinsic parameter, so as to solve the R rotation matrix and the t translation matrix;

[0083] wherein R and t represent the rotation matrix and the translation matrix respectively, R is a 3x3 matrix used to describe the rotation transformation from the world coordinate system to the camera coordinate system; it represents how the object rotates in the three-dimensional space, so that the coordinates of the object can be aligned with the coordinate system of the camera; t is a 3x1 vector representing the translation transformation from the world coordinate system to the camera coordinate system; it describes the displacement of the camera relative to the object; the combination of R and t is the pose of the object.

[0084] Embodiment 2:

[0085] As shown in Figure 2 , the YOLOv8 model includes Backbone, Neck and Head;

[0086] wherein the Backbone includes sequentially connected Conv layer, Conv-1 layer, C2f-1 layer, Conv-2 layer, C2f-2 layer, Conv-3 layer, C2f-3 layer, Conv-4 layer, C2f-4 layer and SPPF layer;

[0087] The Neck includes sequentially connected Upsample-1 layer, Concat-1 layer, C2f-5 layer, Upsample-2 layer, Concat-2 layer, C2f-6 layer, Conv-5 layer, Concat-3 layer, C2f-7 layer, Conv-6 layer, Concat-4 layer and C2f-8 layer;

[0088] The Head includes Detect-1 layer, Detect-2 layer and Detect-3 layer;

[0089] The C2f-2 layer is connected with the Concat-2 layer, the SPPF layer is connected with the Upsample-1 layer and the Concat-4 layer, the C2f-6 layer is connected with the Detect-1 layer, the C2f-7 layer is connected with the Detect-2 layer, and the C2f-8 layer is connected with the Detect-3 layer.

[0090] Embodiment 3

[0091] As shown in Figure 3 , the SWS (SimAM With Slicing) layer is integrated into the Conv-1 layer to the Conv-6 layer of the YOLOv8 model, and the BSAM (BiLebel Spatial Attention Module) layer is added after the SPPF layer of the backbone network to form an improved YOLOv8 model, so that the detection accuracy of the model is improved.

[0092] The structures of the Conv_SWS-1 layer to the Conv_SWS-6 layer are as shown in Figure 4 .

[0093] The BSAM layer is composed of a BRA module (Bi-level Routing Attention) and a spatial attention module. The spatial attention module weights the features at different positions in the spatial dimension, thereby capturing the relationship between different regions. The BRA module realizes more flexible content-aware computing allocation through a double-layer routing mechanism. The BRA module includes two levels of attention operations: Few-to-Many Attention and Many-to-Few Attention.

[0094] In the first level (Few-to-Many Attention), the number of query vectors is small, and the number of key vectors is large. Each query vector only performs attention calculation with part of the key vectors, which can reduce the amount of calculation and reduce the complexity. The second level (Many-to-Few Attention) ensures that each query vector performs attention calculation with all key vectors, thereby ensuring the expression ability of the model and capturing more feature information. The attention calculation formula in the BRA module is: wherein Q is a query vector, K is a key vector matrix, V is a value vector matrix, K^T represents the transpose of the key vector matrix, and C is a scalar factor used to avoid weight concentration and gradient disappearance.

[0095] The spatial attention module (SWS) is designed to enhance the spatial information of feature maps by focusing on the salient regions in the image. The core goal of this module is to highlight the areas of the image that are useful for the task by adaptively assigning weights to different spatial locations. The spatial attention module first receives a feature map from a convolutional network as input. The module compresses the spatial dimensions of the feature map through global average pooling and global max pooling, generating two single-channel images that capture the average response and maximum response of each location in the image, respectively. Then, these two pooled results are concatenated in the channel dimension to form a new feature map. A convolutional layer is applied to the concatenated feature map to generate a single-channel spatial attention map. To normalize the output and limit it to the range [0, 1], the spatial attention map is passed through a Sigmoid activation function. Finally, the generated attention map is multiplied element-wise with the input feature map, weighting different regions of the image. High-weight regions are enhanced, while low-weight regions are suppressed.

[0096] The SWS (SimAM With Slicing) layer is a SimAM module with slicing operation. SimAM (Simple Attention Mechanism) is a lightweight feature enhancement module that draws inspiration from the way humans process visual information. The brain assesses the importance of each pixel in an image by evaluating its relationship with surrounding pixels. In SimAM, this importance is represented by a 3-D weight. By calculating an Energy function, SimAM can measure the importance of each pixel and dynamically adjust the weights of the input image. Finally, by multiplying the calculated weights with the original input, an adjusted feature map is obtained.

[0097] SWS introduces slicing operation based on SimAM, dividing the feature map into multiple small blocks and calculating the average of pixel differences in each slice independently. Large targets have distinct texture features, which significantly affect the average value of the block they are in, resulting in reduced weighting enhancement in their region. However, when these slices are combined into a complete feature map, large targets can still maintain high recognizability and may even be further enhanced. In contrast, small targets have features that differ greatly from the local average of the slice they are in, so they will receive more weighting, allowing for significant enhancement of their features. By introducing slicing operation, the SWS module ensures that both large and small targets receive fair attention during feature enhancement. The reduced weighting of large targets helps to reduce their excessive influence in the feature map, while small targets receive more weighting, allowing their features to be highlighted better.

[0098] The application can realize efficient and accurate vehicle pose estimation in a complex actual road scene without additional parameters and manual labeling, and is suitable for multiple fields such as automatic driving, intelligent transportation, augmented reality and the like, and has wide prospects.

[0099] The above-mentioned embodiments are only preferred embodiments of the present application, and cannot be used to limit the protection scope of the present application, and any non-essential changes and replacements made by those skilled in the art on the basis of the present application shall fall within the protection scope of the present application.

Claims

1. A vehicle pose estimation method based on semantic feature point detection, characterized in that, The method comprises the following steps: Step S1: constructing a virtual vehicle 3D model and selecting points on the vehicle model as 3D key points; Step S2: collecting vehicle pictures in real scene road pictures, detecting 2D bounding boxes of the vehicle pictures through a YOLOv8 model, and extracting vehicle pictures in the 2D bounding boxes; Step S3: classifying the extracted vehicle pictures through a VisionTransformer technology; Step S4: selecting a corresponding vehicle 3D model according to the classified vehicle type; Step S5: improving the YOLOv8 model, integrating an SWS layer into convolutional layers in a backbone network and a Neck of the YOLOv8, the SWS layer being a SimAM module with a slicing operation, and adding a BSAM layer after an SPPF layer of the backbone network; Step S6: detecting the vehicle 3D model by using the improved YOLOv8 model and screening 3D key points; Step S7: solving a vehicle pose through a multi-point perspective algorithm; Firstly, the vehicle 3D model is rendered to obtain a synthetic image sample; Then, the improved YOLOv8 model is trained using the synthetic sample; According to the definition of 3D semantic feature points, the 3D semantic feature points of the real vehicle image are labeled, and the improved YOLOv8 model is fine-tuned, and the fine-tuned YOLOv8 model detects 2D key points; The screened 3D key points and the detected 2D key points are corresponded one by one, and the pose is solved through the multi-point perspective algorithm.

2. The semantic feature point detection based vehicle pose estimation method of claim 1, wherein, In the step S1, a 3D modeling software is used to model a virtual vehicle 3D model. 3.The semantic feature point detection based vehicle pose estimation method of claim 1, wherein, In the step S1, the corner points are selected as the 3D key points, and the steps are as follows: From any vertex on the virtual vehicle 3D model Within a spherical range of radius There are nearest neighbors The covariance matrix of this set of points is Where ; calculate eigenvalues , , ,and , It can reflect the curvature around the vertex, and a threshold can be set. ,like Then determine It is a corner point. 4.The semantic feature point detection based vehicle pose estimation method of claim 1, wherein, In the step S2, the 2D bounding box includes the category of the detected target, the 2D coordinates of the center point of the 2D bounding box, and the length and width of the bounding box.

5. The semantic feature point detection based vehicle pose estimation method of claim 1, wherein, In the step S7, the synthetic image sample includes a vehicle picture, 3D feature point coordinates corresponding to the 3D key points, and 2D feature point coordinates; The 3D feature points are three-dimensional coordinates of the 3D key points in a world coordinate system established with the center of the virtual vehicle 3D model as the origin, and the 2D feature points are projection coordinates of the 3D feature points on a 2D image.

6. The semantic feature point detection based vehicle pose estimation method of claim 1, wherein, In the step S7, the training steps of the improved YOLOv8 model are as follows: The conda environment required by YOLOv8 is configured, and the related dependent libraries are installed; The 2D point labeling file in the synthetic sample is changed to the format required by YOLOv8; The data set is divided into a training set, a validation set and a test set according to a ratio of 8:1:1; The learning rate, batch and epoch training related configurations in the model training are changed in the configuration file of YOLOv8; The training script is run, and the input folder and the output folder are specified, the input folder containing the training set pictures, the validation set pictures and the 2D labeling file, and the output folder being used to save the weight output by the model.

7. The semantic feature point detection based vehicle pose estimation method of claim 1, wherein, In the step S7, the semantic features of the screened 3D key points and the corresponding 3D feature points and 2D feature points are defined, and the names and types of the screened 3D key points and the corresponding 3D feature points and 2D feature points are defined.

8. The semantic feature point detection based vehicle pose estimation method of claim 1, wherein, In the step S7, the multi-point perspective algorithm is as follows: wherein, , and are coordinates of 3D keypoints, and are coordinates of 2D keypoints, is a camera intrinsic, thereby enabling and solving of a translation matrix.

9. The semantic feature point detection based vehicle pose estimation method of claim 1, wherein, The YOLOv8 model comprises a Backbone, a Neck, and a Head.

Images