Model matching-based vehicle 3D pose estimation method

Abstract

This invention relates to the field of artificial intelligence technology and discloses a model-matching-based method for estimating the 3D pose of a vehicle, comprising the following steps: Step 1, acquiring poses of a 3D vehicle model from different viewpoints, with each pose corresponding to an image, forming a pose image library; Step 2, acquiring the target RGB image, preprocessing the RGB image to obtain the image's normal map; Step 3, fusing the target's RGB image and normal map to perform a coarse estimation of the vehicle pose, thereby eliminating misleading information caused by vehicle symmetry. This invention can effectively identify the pose of vehicles in a scene, especially enabling pose estimation without additional parameters, and effectively reducing the risk of pose judgment errors due to vehicle symmetry.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for estimating the three-dimensional pose of a vehicle based on model matching. Background Technology

[0002] In accordance with the requirements of the National Road Traffic Safety Law on vehicle driving and traffic safety, the driving direction of vehicles on the road is fixed. In order to avoid the behavior of vehicles driving incorrectly on the road, this invention proposes a vehicle pose estimation method based on model matching.

[0003] Traditional vehicle pose estimation methods are generally divided into two types: one is based on deep learning technology. This type of technology mainly predicts vehicle pose by training a deep learning model, and it is usually divided into two categories. One type trains the model based on labeled data to directly obtain vehicle pose information; the other type detects vehicle key points and estimates pose information through perspective projection based on camera parameters, or estimates the pose by matching with a vehicle model.

[0004] Another method relies on precision instruments such as sensors to estimate vehicle pose. This method integrates the results from multiple sensors or filtering and iteratively updates the vehicle state based on the vehicle's dynamic model.

[0005] Traditional deep learning-based pose estimation methods have certain drawbacks:

[0006] Currently, deep learning methods for pose estimation typically employ supervised methods. This process requires a massive amount of data for model training, and annotating 3D pose information in 2D images is extremely difficult and often requires a significant amount of human and material resources.

[0007] Once a deep learning model is trained, the types of vehicles it can detect remain fixed. It will be unable to correctly identify vehicles outside the training sample.

[0008] Key point detection methods require information support, such as camera parameters and actual aspect ratio; due to the symmetry of vehicles, model matching methods are prone to mismatching symmetrical vehicles.

[0009] Methods based on precision instruments have the following drawbacks:

[0010] Vehicle pose estimation methods based on precision instruments have the advantage of high accuracy, but they also have drawbacks such as high cost, complex equipment setup and operation, sensitivity to environmental conditions, and potential impacts on vehicle weight, energy consumption, and data latency.

[0011] These methods are often limited to a single vehicle that needs to be observed, and are subject to limitations such as visibility, hardware failure, and signal interference, which may lead to inaccurate or failed pose estimation under certain circumstances. Summary of the Invention

[0012] To address the technical problems mentioned in the background section, this invention provides a vehicle 3D pose estimation method based on model matching.

[0013] This invention employs the following technical solution: a vehicle 3D pose estimation method based on model matching, comprising the following steps:

[0014] Step 1: Acquire poses of the 3D vehicle model from different viewpoints. Each pose corresponds to one image, forming a pose image library.

[0015] Step 2: Obtain the target RGB image, preprocess the RGB image, and obtain the normal map of the image;

[0016] Step 3: Fuse the RGB image and normal image of the target to make a rough estimate of the vehicle pose, thereby eliminating the misleading effect caused by vehicle symmetry.

[0017] Step 4: Extract the foreground from the target RGB image. Based on the rough pose estimation, input the result into a feature point detection and matching network and perform feature matching with the corresponding pose image database to obtain the matched feature points k. i , matching point m i and feature point k i Match score s i (i∈K, K is the number of images in the corresponding pose image library), as the matching result; for each matching result, the final matching score is calculated as Score=w1×cfd+w2×rc, where w1=0.5, w2=0.5, (num() is used to count the number of items, sum() is used to sum the results)

[0018] The pose of the image corresponding to the maximum value of the desired score is the final estimated pose.

[0019] The RGB image refers to the original image of the target, which is an unprocessed image.

[0020] The foreground extraction uses a Mask R-CNN segmentation model to eliminate the influence of the background on the matching effect and ensure the accuracy of the matching results. It includes the following steps:

[0021] The target image is fed into the Mask R-CNN network for semantic segmentation;

[0022] The target image is segmented based on the semantic information of the vehicle mask to obtain the foreground extracted image.

[0023] When the approximate pose of the vehicle is determined by coarse pose estimation, it is only compared with the pose image library of the corresponding orientation.

[0024] The pose image library corresponding to the orientation (assuming there are N images) is fed into the detection and matching network along with the target image to obtain N scores. The pose of the pose image with the highest score is considered to be consistent with the pose of the target image.

[0025] The pose information (pitch angle, rotation angle, and 3D bounding box information) of the saved pose image is obtained as the pose information of the target image.

[0026] The coarse estimation of the vehicle pose uses a classification model with the ResNet50 model as its skeleton, which includes the following steps:

[0027] The target image (the real image, i.e., the traditional RGB image) and the normal map obtained through Omnidata are fed into the same but different ResNet50 backbone network to obtain the features of the two images.

[0028] Features are concatenated and merged, such as A = [x1, x2, x3, ... x n B = [y1, y2, y3, ... y n After fusion, C = [x1, x2, x3, ... x n ,y1,y2,y3,…y n ], where A is the feature of the target image (RGB image), B is the feature of the normal map, and C is the fused feature. In other words, image fusion is image feature fusion.

[0029] Estimation method: The fused feature C is fed into the classification model for classification.

[0030] It is important to note that mismatches caused by symmetry occur because the matching network simply matches points in one image with points in the corresponding image that are close to them, without considering whether a point representing the front wheel is matched to a point representing the rear wheel in the symmetrical image.

[0031] Therefore, the orientation information of some vehicles in the real image was marked with different colors using the normal map obtained by Omnidata. Based on these color information and the texture information of the real image, the approximate orientation of the vehicle in the real image can be inferred.

[0032] The 3D pose acquisition is a 3D pose image of a 3D virtual model, and the workflow is as follows:

[0033] Use Unity software to obtain pose images;

[0034] Choose a pure white background to ensure that the background matches the foreground extraction result image later;

[0035] Add a 3D virtual vehicle model between a pure white background and the camera;

[0036] The positions of the camera and background are changed synchronously and symmetrically, while keeping the distance between the two sides and the vehicle constant, only the shooting angle is changed;

[0037] While saving the captured images, it also records the changes in the camera's pitch and rotation angles, as well as information about the 3D bounding box of the 3D virtual vehicle model.

[0038] The images are divided into 16 categories according to their degrees. Starting from each category, they are combined into pairs to form 8 categories for rough pose estimation, thus forming the final pose library.

[0039] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0040] This invention can effectively identify the pose of vehicles in a scene, especially to estimate the pose without additional parameters, and effectively reduce the risk of pose judgment errors caused by vehicle symmetry. Attached Figure Description

[0041] Figure 1 This is a flowchart of the estimation method proposed in this invention;

[0042] Figure 2 The original image and the processed normal map of the target of this invention;

[0043] Figure 3 The present invention relates to a 3D car model and the obtained pose library;

[0044] Figure 4 This is a schematic diagram of the target image obtained by fusing the original target image and the normal image of the present invention, and the corresponding pose library when matching the image in the same direction with the image in the same orientation.

[0045] Figure 5 This is a schematic diagram illustrating the process of matching the target image (after fusing the original image and normal map) with the corresponding pose library to an image in a symmetrical direction. Detailed Implementation

[0046] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0047] Example 1:

[0048] Reference Figure 1-5 The proposed model-matching-based vehicle 3D pose estimation method includes the following steps:

[0049] Step 1: Acquire poses of the 3D vehicle model from different viewpoints. Each pose corresponds to one image, forming a pose image library, which is then referenced. Figure 3 ;

[0050] Step 2: Obtain the target RGB image, preprocess the RGB image, and obtain the normal map of the image;

[0051] Step 3: Fuse the RGB image and normal image of the target to make a rough estimate of the vehicle pose, thereby eliminating the misleading effect caused by vehicle symmetry.

[0052] Step 4: Extract the foreground from the target RGB image. Based on the rough pose estimation, input the result into a feature point detection and matching network and perform feature matching with the corresponding pose image database to obtain the matched feature points k. i , matching point m i and feature point k i Match score s i (i∈K, K is the number of images in the corresponding pose image library), as the matching result; for each matching result, the final matching score is calculated as Score=w1×cfd+w2×rc, where w1=0.5, w2=0.5, (num() is used to count the number of items, sum() is used to sum the results)

[0053] The pose of the image corresponding to the maximum value of the desired score is the final estimated pose.

[0054] The RGB image refers to the original image of the target, which is an unprocessed image.

[0055] The foreground extraction uses a Mask R-CNN segmentation model to eliminate the influence of the background on the matching effect and ensure the accuracy of the matching results. It includes the following steps:

[0056] The target image is fed into the Mask R-CNN network for semantic segmentation;

[0057] The image is segmented based on the semantic information of the target image, which is a mask for vehicles (except for the mask part, which retains its original color, and the rest is set to white), and the foreground image is obtained.

[0058] When the approximate pose of the vehicle is determined by coarse pose estimation, it is only compared with the pose image library of the corresponding orientation.

[0059] The pose image library corresponding to the orientation (assuming there are N images) is fed into the detection and matching network along with the target image to obtain N scores. The pose of the pose image with the highest score is considered to be consistent with the pose of the target image.

[0060] The pose information (pitch angle, rotation angle, and 3D bounding box information) of the saved pose image is obtained as the pose information of the target image.

[0061] The coarse estimation of the vehicle pose uses a classification model with the ResNet50 model as its skeleton, which includes the following steps:

[0062] The target image (the real image, i.e., the traditional RGB image) and the normal map obtained through Omnidata are fed into the same but different ResNet50 backbone network to obtain the features of the two images.

[0063] Features are concatenated and merged, such as A = [x1, x2, x3, ... x n B = [y1, y2, y3, ... y n After fusion, C = [x1, x2, x3, ... x n ,y1,y2,y3,…y n ], where A is the feature of the target image (RGB image), B is the feature of the normal map, and C is the fused feature. In other words, image fusion is image feature fusion.

[0064] Estimation method: The fused feature C is fed into the classification model for classification.

[0065] It is important to note that mismatches caused by symmetry occur because the matching network simply matches points in one image with points in the corresponding image that are close to each other, without considering whether a point of the front wheel is matched to a point of the rear wheel in the symmetrical image. Figure 4 This is the situation we believe to be correct. Figure 5 The problem is caused by symmetry.

[0066] Therefore, the orientation information of some vehicles in the real image was marked with different colors using the normal map obtained by Omnidata. Based on these color information and the texture information of the real image, the approximate orientation of the vehicle in the real image can be inferred.

[0067] The 3D pose acquisition is a 3D pose image of a 3D virtual model (fbx file format). The workflow is as follows:

[0068] Use Unity software to obtain pose images;

[0069] Choose a pure white background to ensure that the background matches the foreground extraction result image later;

[0070] Add a 3D virtual vehicle model between a pure white background and the camera;

[0071] The positions of the camera and background are changed synchronously and symmetrically, while keeping the distance between the two sides and the vehicle constant, only the shooting angle is changed;

[0072] While saving the captured images, the camera's pitch and rotation angles (excluding the flip angle, which is assumed not to occur in normal scenes) and the 3D bounding box information of the 3D virtual vehicle model are also recorded.

[0073] The images are divided into 16 categories according to their degrees. Starting from each category, they are combined into pairs to form 8 categories for rough pose estimation, thus forming the final pose library.

[0074] Theoretically, the more images a solution produces, the more accurate the results will be, but this also depends on the performance of the matching model.

[0075] The above embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of protection of the present invention. Any non-substantial changes and substitutions made by those skilled in the art based on the present invention shall fall within the scope of protection claimed by the present invention.

Claims

1. A vehicle 3D pose estimation method based on model matching, characterized in that, Includes the following steps: Step 1: Acquire poses of the 3D vehicle model from different viewpoints. Each pose corresponds to one image, forming a pose image library. Step 2: Obtain the target RGB image, preprocess the RGB image, and obtain the normal map of the image; Step 3: Fuse the RGB image and normal image of the target to make a rough estimate of the vehicle pose, thereby eliminating the misleading effect caused by vehicle symmetry. Step 4: Extract the foreground from the target RGB image. Based on the rough pose estimation, feed the result into a feature point detection and matching network and perform feature matching with the corresponding pose image database to obtain the matched feature points. , matching point and feature points Match score , as the matching result; Calculate the final matching score for each matching result. ,in ; ； What is requested The pose of the image corresponding to the maximum value is the final estimated pose. When the approximate pose of the vehicle is determined by coarse pose estimation, it is only compared with the pose image library of the corresponding orientation. The pose image library corresponding to the orientation is assumed to have N images. Each pose image is fed into the detection and matching network along with the target image to obtain N scores. The pose image with the highest score is considered to be consistent with the pose image of the target image. The pose information of the saved pose image is obtained as the pose information of the target image; The coarse estimation of the vehicle pose uses a classification model with the ResNet50 model as its skeleton, which includes the following steps: The RGB image and the normal map obtained through Omnidata are fed into the same but different ResNet50 backbone network to obtain the features of the two images. Feature splicing and fusion, such as After fusion, Where A represents the features of the RGB image, B represents the features of the normal image, and C represents the fused features. In other words, image fusion is the fusion of image features. Estimation method: The fused feature C is fed into the classification model for classification; It is important to note that mismatches caused by symmetry occur because the matching network simply matches points in one image with points in the corresponding image that are close to each other, without considering whether a point of the front wheel is matched to a point of the rear wheel in the symmetrical image. Therefore, the orientation information of some vehicles in the real image is marked with different colors using the normal map obtained by Omnidata. Based on these color information and the texture information of the real image, the approximate orientation of the vehicle in the real image can be inferred. 3D pose acquisition obtains the 3D pose image of a 3D virtual model. The workflow is as follows: Use Unity software to obtain pose images; Choose a pure white background to ensure that the background matches the foreground extraction result image later; Add a 3D virtual vehicle model between a pure white background and the camera; The positions of the camera and background are changed synchronously and symmetrically, while keeping the distance between the two sides and the vehicle constant, only the shooting angle is changed; While saving the captured images, it also records the changes in the camera's pitch and rotation angles, as well as the 3D bounding box information of the 3D virtual vehicle model. The images are divided into 16 categories according to their degrees. Starting from each category, they are combined into pairs to form 8 categories for rough pose estimation, thus forming the final pose library.

2. The vehicle 3D pose estimation method based on model matching as described in claim 1, characterized in that, The RGB image refers to the original image of the target, which is an unprocessed image.

3. The vehicle 3D pose estimation method based on model matching as described in claim 1, characterized in that, The foreground extraction uses a Mask R-CNN segmentation model to eliminate the influence of the background on the matching effect and ensure the accuracy of the matching results. It includes the following steps: The target image is fed into the Mask R-CNN network for semantic segmentation; The target image is segmented based on the semantic information of the vehicle mask to obtain the foreground extracted image.

Images