An intelligent vehicle 3D target detection method suitable for uneven road surface scenes

Abstract

The application discloses a kind of 3D target detection methods of intelligent vehicle suitable for uneven road scene, vehicle camera sensor and vehicle MEMS sensor are respectively simultaneously obtained front view image data and vehicle acceleration information, the image feature map F of feature extraction is obtained to the front view image data obtained, simultaneously through the vehicle acceleration information obtained, the real-time roll angle and pitch angle of vehicle are calculated, then based on roll angle and pitch angle, rotation matrix A is calculated, transformation matrix M is obtained by rotation matrix and camera internal and external parameter matrix, the conversion of pixel coordinates is carried out to image feature map F by transformation matrix M, and image feature map F' is obtained, 2D feature is projected to 3D feature based on feature map F', finally, the result of 3D target detection is obtained by 3D target detection head based on 3D feature.The application can better eliminate the error of 3D target detection regression caused by uneven road of intelligent vehicle, improve the actual detection precision, improve the driving stability and safety of intelligent vehicle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent driving, and in particular to an intelligent vehicle 3D target detection method suitable for uneven road surface scenes. BACKGROUND

[0002] With the development of technology and economy and the wide application of intelligent technology, the field of intelligent driving has become a hot research direction at present. Among them, thanks to the maturity of computer vision technology, the field of intelligent driving perception has developed rapidly in recent years.

[0003] 3D target detection is an important branch of intelligent driving perception. The main task of 3D target detection is to output the class prediction and accurate position regression of the target object through the processing of sensor input data, so as to help the vehicle to avoid obstacles and plan routes. Therefore, it is crucial for intelligent driving to accurately perceive the surrounding environment and achieve accurate and reliable 3D target detection.

[0004] Although the 3D target detection technology based on vehicle-mounted laser radar sensors has achieved good performance, considering the actual manufacturing cost and industrial landing of intelligent driving vehicles, in recent years, 3D target detection based on vehicle-mounted camera sensors has received more and more attention.

[0005] Among them, monocular 3D target detection is one of the key research directions of 3D target detection, which is based on monocular camera data for 3D target detection.

[0006] However, most of the existing monocular 3D target detection methods are based on the premise of flat road, that is, it is usually assumed that the ego vehicle and the detected target vehicle are always on the same plane, and the target vehicle only has a heading angle relative to the camera on the ego vehicle, and the vehicle pitch angle and roll angle are both 0. However, in the actual intelligent driving environment, the road surface will not always be in flat conditions, especially on non-urban roads, uneven road surfaces can be found everywhere, at this time, the target vehicle may have changes in pitch angle and roll angle relative to the camera on the ego vehicle, if the 3D frame prediction is still carried out according to the ideal state assumption, errors will be introduced, resulting in obvious performance degradation of 3D frame detection accuracy and target position prediction. SUMMARY

[0007] The technical problem to be solved by the present application is to provide an intelligent vehicle 3D target detection method suitable for uneven road surface scenes, aiming to solve the problem of performance degradation caused by inaccurate 3D target detection frame regression when the road flatness and slope have slight fluctuations.

[0008] The technical problem to be solved by the present application is to provide an intelligent vehicle 3D target detection method suitable for uneven road surface scenes, aiming to solve the problem of performance degradation caused by inaccurate 3D target detection frame regression when the road flatness and slope have slight fluctuations.

[0009] A 3D target detection method for intelligent vehicles in uneven road surface scenarios, comprising the following steps:

[0010] Step 1), obtaining image data of a real-time front road of a vehicle-mounted camera as front view image data;

[0011] Step 2), obtaining real-time vehicle acceleration data measured by an accelerometer in a vehicle-mounted MEMS sensor;

[0012] Step 3), calculating real-time roll angle and pitch angle of the vehicle using the obtained real-time vehicle acceleration data;

[0013] Step 4), performing image feature extraction on the front view image data to obtain an image feature map F in pixel coordinates;

[0014] Step 5), calculating a rotation matrix A of the current vehicle coordinate system relative to the vehicle coordinate system when the vehicle is driving on a flat road using the obtained real-time roll angle and pitch angle of the vehicle, and obtaining a transformation matrix M for coordinate conversion of pixel points on the image feature map based on the rotation matrix A and camera internal and external parameters;

[0015] Step 6), performing pixel coordinate conversion on the image feature map F using the transformation matrix M to obtain a converted image feature map F';

[0016] Step 7), projecting 2D features of the image feature map F' into a 3D space to obtain 3D features;

[0017] Step 8), performing 3D target detection on the 3D features using a 3D target detection head to regress accurate 3D box positions and target class information.

[0018] As a further optimization scheme of the 3D target detection method for intelligent vehicles in uneven road surface scenarios, the real-time vehicle acceleration data measured by the accelerometer in the vehicle-mounted MEMS sensor in step 2) comprises a x , a y , a z , a x , a y , a z , respectively, being the acceleration of the vehicle in the x-axis, y-axis and z-axis of the vehicle coordinate system.

[0019] As a further optimization scheme of the 3D target detection method for intelligent vehicles in uneven road surface scenarios, the calculation of the roll angle θ r and the pitch angle θ p in step 3) is as follows:

[0020] θ r = arctan(a y / az )

[0021] θ p = arcsin(-a x ).

[0022] As a further optimization scheme of the intelligent vehicle 3D target detection method suitable for uneven road surface scenes, the rotation matrix A in step 4) is:

[0023]

[0024] The transformation matrix M in step 4) is:

[0025]

[0026] In the formula, i is a camera coordinate system constructed with the camera on the vehicle as the origin, j is a world coordinate system constructed with the ground as the xz plane, z is the height coordinate of the object center in the world coordinate system, P is the projection matrix of the 2D image feature to the world coordinate system, and it is:

[0027] P = KT

[0028] In the formula, K is the intrinsic matrix of the vehicle-mounted camera, and T is the extrinsic matrix of the vehicle-mounted camera.

[0029] As a further optimization scheme of the intelligent vehicle 3D target detection method suitable for uneven road surface scenes, the image feature extraction network in step 5) includes a ResNet-100 model and an FPN model, wherein the ResNet-100 model is used for image feature extraction, and the last 4 layers of the network are output to the FPN model for multi-scale feature fusion to obtain a feature map F.

[0030] As a further optimization scheme of the intelligent vehicle 3D target detection method suitable for uneven road surface scenes, the image feature map F is converted based on the pixel coordinates using the transformation matrix M in step 6), and the formula is:

[0031]

[0032] In the formula, is the pixel coordinates of each feature of the image feature map F, is the pixel coordinates of each feature of the converted image feature map F'.

[0033] As a further optimization scheme of the intelligent vehicle 3D target detection method suitable for uneven road surface scenes, the 2D features are projected to 3D space in step 7), and the projection matrix of the 3D features is P, and the projection relationship is:

[0034]

[0035] In the formula, x, y, z are coordinate values of points in the world coordinate system, and u, v are pixel coordinates of the image feature map F'.

[0036] As a further optimization scheme of the intelligent vehicle 3D target detection method suitable for uneven road surface scenes, the 3D target detection head in step 8) first compresses the 3D features along the z-axis to obtain the features in the bird's eye view, and then uses three parallel 1x1 convolutions to perform the 3D target detection task on the features in the bird's eye view, and respectively predicts the class, size and direction of each object.

[0037] Compared with the prior art, the above technical scheme has the following technical effects:

[0038] 1. The external disturbance of the vehicle-mounted camera caused by the slight roll and pitch of the vehicle due to the uneven road surface in the real intelligent vehicle driving scene is fully considered, so that the error of the 3D target detection regression caused by the disturbance can be better eliminated, and the actual detection accuracy is improved.

[0039] 2. The real-time vehicle acceleration information is obtained through the vehicle-mounted MEMS sensor, so that the real-time vehicle roll angle and pitch angle data can be directly inferred, and the image feature map transformation matrix M can be directly obtained, without the need to obtain M through the front and rear frame data or neural network prediction, so that the running efficiency of the 3D target detection model is improved, the model parameter amount is reduced, and the industrial landing and deployment of the model are more beneficial.

[0040] 3. The pixel coordinate conversion is directly performed on the basis of the image feature map after feature extraction, avoiding direct conversion on the original image, reducing the calculation amount of coordinate conversion, and improving the conversion efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 is a flowchart of the present application. DETAILED DESCRIPTION

[0042] The technical scheme of the present application will be further described in detail below with reference to the accompanying drawings:

[0043] The present application can be implemented in many different forms, and should not be considered limited to the embodiments described herein. On the contrary, these embodiments are provided to make the present disclosure thorough and complete, and to fully convey the scope of the present application to those skilled in the art. In the drawings, the components are enlarged for clarity.

[0044] Referring to Figure 1 As shown in the drawings, the present application discloses an intelligent vehicle 3D target detection method suitable for uneven road surface scenes, characterized in that it comprises the following steps:

[0045] Step 1), obtaining image data of a real-time front road of a vehicle-mounted camera as front-view image data;

[0046] Step 2), obtaining real-time vehicle acceleration data measured by an accelerometer in a vehicle-mounted MEMS sensor;

[0047] Step 3), calculating real-time roll angle and pitch angle of the vehicle by using the obtained real-time vehicle acceleration data;

[0048] Step 4), performing image feature extraction on the front-view image data to obtain an image feature map F in pixel coordinates;

[0049] Step 5), calculating a rotation matrix A of a current vehicle coordinate system relative to a vehicle coordinate system when the vehicle is driving on a flat road by using the obtained real-time roll angle and pitch angle of the vehicle, and obtaining a transformation matrix M for coordinate conversion of pixel points on the image feature map based on the rotation matrix A and internal and external parameters of the camera;

[0050] Step 6), performing pixel coordinate conversion on the image feature map F by using the transformation matrix M to obtain a converted image feature map F';

[0051] Step 7), projecting 2D features of the image feature map F' into a 3D space to obtain 3D features;

[0052] Step 8), performing 3D target detection on the 3D features by using a 3D target detection head to regress accurate 3D box positions and target category information.

[0053] The real-time vehicle acceleration data measured by the accelerometer in the vehicle-mounted MEMS sensor in the step 2) includes a x , a y , a z , a x , a y , a z , respectively representing accelerations of the vehicle on x-axis, y-axis and z-axis of the vehicle coordinate system.

[0054] The roll angle θ r in the step 3) is calculated as follows:

[0055]

[0056] The pitch angle θ p in the step 3) is calculated as follows:

[0057] θ p = arcsin(-a x )

[0058] The rotation matrix A in the step 4) is:

[0059]

[0060] The transformation matrix M in step 4) is:

[0061]

[0062] In the formula, i is a camera coordinate system constructed with the camera on the vehicle as the origin, j is a world coordinate system constructed with the ground as the xz plane, z is the height coordinate of the object center in the world coordinate system, P is a projection matrix of the 2D image feature to the world coordinate system, which is:

[0063] P = KT

[0064] In the formula, K is the intrinsic matrix of the vehicle-mounted camera, and T is the extrinsic matrix of the vehicle-mounted camera.

[0065] The image feature extraction network in step 5) includes a ResNet-100 model and an FPN model, wherein the ResNet-100 model is used for image feature extraction, and the last 4 layers of the network are output to the FPN model for multi-scale feature fusion to obtain a feature map F.

[0066] In step 6), the image feature map F is converted based on the pixel coordinates using the transformation matrix M, and the formula is:

[0067]

[0068] In the formula, is the pixel coordinates of each feature of the image feature map F, is the pixel coordinates of each feature of the converted image feature map F'.

[0069] In step 7), the 2D features are projected into the 3D space, and the projection matrix of the 3D features is P, and the projection relationship is:

[0070]

[0071] In the formula, x, y, and z are the coordinate values of the point in the world coordinate system, and u and v are the pixel coordinates of the image feature map F'.

[0072] In step 8), the 3D target detection head first compresses the 3D features along the z-axis to obtain the features in the bird's eye view, and then uses three parallel 1x1 convolutions to perform the 3D target detection task on the features in the bird's eye view, and respectively predicts the class, size and direction of each object.

[0073] As used herein, and unless otherwise indicated, all terms have their ordinary meanings. It should be understood that any definitions are to be used as elucidated herein and should not be taken in any restrictive sense unless expressly defined.

[0074] The above description is only specific embodiments of the present application. It should be understood that the above description is only a specific embodiment of the present application. It should be understood that any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for 3D object detection for intelligent vehicles in uneven road scenarios, the method comprising: The method comprises the following steps: Step 1), obtaining image data of a real-time front road of a vehicle-mounted camera as front-view image data; Step 2), obtaining real-time vehicle acceleration data measured by an accelerometer in a vehicle-mounted MEMS sensor; Step 3), calculating a real-time roll angle and a real-time pitch angle of the vehicle by using the obtained real-time vehicle acceleration data; Step 4), performing image feature extraction on the front-view image data to obtain an image feature map F in pixel coordinates; Step 5), calculating a rotation matrix A of a current vehicle coordinate system relative to a vehicle coordinate system when the vehicle is running on a flat road by using the obtained real-time roll angle and real-time pitch angle of the vehicle, and obtaining a transformation matrix M for coordinate conversion of pixel points on the image feature map based on the rotation matrix A and camera internal and external parameters; Step 6), performing pixel coordinate conversion on the image feature map F by using the transformation matrix M to obtain a converted image feature map F'; Step 7), projecting 2D features of the image feature map F' into a 3D space to obtain 3D features; Step 8), performing 3D target detection on the 3D features by using a 3D target detection head to regress accurate 3D frame positions and target category information. 2.The intelligent vehicle 3D target detection method for uneven road surface scenes of claim 1, wherein, The real-time vehicle acceleration data measured by the accelerometer in the vehicle-mounted MEMS sensor in step 2) includes a x , a y , a z , a x , a y , a z are the accelerations of the vehicle on the x-axis, y-axis and z-axis of the vehicle coordinate system, respectively. 3.The intelligent vehicle 3D target detection method for uneven road surface scenes of claim 2, wherein, The calculation of the roll angle θ r and the pitch angle θ p in said step 3) is as follows: θ r = arctan(a y / a z ) θ p = arcsin(-a x ).

4. The intelligent vehicle 3D object detection method suitable for uneven road surface scenarios of claim 3, wherein, The rotation matrix A in step 4) is: The transformation matrix M in step 4) is: In the formula, i is a camera coordinate system constructed with the camera on the vehicle as the origin, j is a world coordinate system constructed with the ground as the xz plane, z is the height coordinate of the object center in the world coordinate system, P is a projection matrix of the 2D image features to the world coordinate system, and P is: P = KT In the formula, K is an internal parameter matrix of the vehicle-mounted camera, and T is an external parameter matrix of the vehicle-mounted camera.

5. The intelligent vehicle 3D object detection method suitable for uneven road surface scenarios of claim 4, wherein, The image feature extraction network in step 5) comprises a ResNet-100 model and an FPN model, wherein the ResNet-100 model is used for image feature extraction, the last 4 layers of the network are output to the FPN model for multi-scale feature fusion to obtain the feature map F.

6. The intelligent vehicle 3D object detection method suitable for uneven road surface scenarios of claim 5, wherein, In step 6), the image feature map F is converted based on pixel coordinates by using the transformation matrix M, and the formula is: wherein is the pixel coordinate of each feature of the image feature map F, is the pixel coordinate of each feature of the transformed image feature map F'.

7. The intelligent vehicle 3D object detection method suitable for uneven road surface scenarios according to claim 6, wherein, In step 7), the 2D features are projected into a 3D space to obtain a projection matrix P of the 3D features, and the projection relationship is: In the formula, x, y, and z are coordinate values of a point in the world coordinate system, and u and v are pixel coordinates of the image feature map F'.

8. The intelligent vehicle 3D object detection method suitable for uneven road surface scenarios according to claim 7, wherein, In step 8), the 3D target detection head first compresses the 3D features along the z axis to obtain features in a bird's eye view, and then uses three parallel 1x1 convolutions to perform a 3D target detection task on the features in the bird's eye view to predict the category, size, and direction of each object.

Images