BEV feature mapping method and system suitable for vehicle body pose change and storage medium

Abstract

The invention discloses a BEV feature mapping method and system suitable for vehicle body pose change and a storage medium, and the method comprises the steps: carrying out the re-projection of a current image obtained by a vehicle-mounted around-view camera based on a real-time rotation matrix of a vehicle body relative to a road surface, and generating an equivalent image; extracting image features in the equivalent image, generating a feature map, and pooling the feature map along the vertical direction of the feature map; and on the basis of BEV grid feature indexes, mapping feature vectors in the pooled feature maps to BEV grid units to generate BEV features. According to the technical scheme, the imaging plane of the vehicle-mounted camera is converted to be perpendicular to the road surface through the real-time rotation matrix obtained through calculation, then the feature map of the image at the moment is pooled in the vertical direction, the feature index is established, the BEV features are generated, and the BEV feature generation accuracy can be improved when the vehicle body pose changes.

Description

Technical Field

[0001] The present application relates to the technical field of autonomous driving, and in particular, to a BEV feature mapping method applicable to vehicle body posture changes, a BEV feature mapping system applicable to vehicle body posture changes, and a computer-readable storage medium. Background Art

[0002] At present, the target detection technology based on the Birds Eye View (BEV) map has been widely used in vehicle autonomous driving solutions. This technology presents the surrounding environment of the vehicle body in the form of a bird's-eye view by fusing multi-sensor information, thereby greatly simplifying the development and application of subsequent technology stacks (such as decision planning and path planning control). To achieve BEV target detection, it is first necessary to obtain real-time environmental images taken by multiple on-board surround cameras, and then use a neural network to extract the features of these images and fuse them on the BEV plane. Finally, the fused features (BEV features) are input into the target detection network to achieve accurate recognition and positioning of various targets in the environment.

[0003] In order to obtain the above BEV features, after using a neural network to extract environmental image features from various perspectives, such as Lift-Splat-Shoot (LSS), these environmental image features can be reverse-projected into three-dimensional space based on the internal and external parameters of the on-board surround view camera, the body posture, and the depth probability distribution of the feature pixels to form a dense spatial point cloud. Then, multiple 2D-BEV grids are divided on the vehicle driving plane, and according to the projection coordinates of each point in the grid on the BEV plane, the feature information in the spatial point cloud is aggregated into the unit corresponding to the 2D-BEV grid to form BEV features, which can be used for subsequent target detection.

[0004] In the prior art, in order to reduce the amount of calculation in the process of mapping the point cloud to the BEV grid, the FastBEV method is proposed. This method assumes that the depth of the pixel points corresponding to the environmental image features is evenly distributed in the line of sight, and pre-establishes a query table based on the fixed internal and external parameters of the on-board surround view camera to store the correspondence between the point cloud and the BEV grid unit, thereby speeding up the real-time reasoning and mapping speed.

[0005] However, the relative position relationship between the vehicle body and the ground plane (body posture) is not constant during driving (such as when the vehicle goes uphill or downhill, accelerates or decelerates, or turns), and there are potholes of varying degrees on the road. Therefore, when the body posture changes, the correspondence between the pre-stored point cloud and the BEV grid unit will become invalid, causing the BEV feature to be inaccurate, which in turn affects the accuracy of target detection. Summary of the invention

[0006] The purpose of this application is to improve the accuracy of generating BEV features when the vehicle body posture changes.

[0007] The technical solution of the first aspect of the present application is: a BEV feature mapping method suitable for changes in vehicle body posture is provided, the method comprising: based on the real-time rotation matrix of the vehicle body relative to the road surface, reprojecting the current image acquired by the on-board surround view camera to generate an equivalent image; extracting image features in the equivalent image to generate a feature map, and pooling the feature map along the vertical direction of the feature map; based on the BEV grid feature index, mapping the feature vector in the pooled feature map to the BEV grid unit to generate a BEV feature.

[0008] In any of the above technical solutions, further, the method also includes: establishing a rectangular coordinate system for the vehicle body, and based on the acquired depth map and the internal and external parameters of the camera, mapping the road surface acquired by the vehicle-mounted surround view camera to the rectangular coordinate system of the vehicle body to generate a road point cloud; based on the road point cloud, generating a fitted road plane, and calculating the road surface normal vector of the fitted road surface plane; based on the fitted road surface plane and the road surface normal vector, establishing a road surface coordinate system, wherein the origin of the road surface coordinate system is the projection point of the origin of the vehicle body coordinate system on the fitted road surface plane, the z-axis of the road surface coordinate system is the road surface normal vector, and the x-axis of the road surface coordinate system is determined by the projection of the x-axis of the vehicle body coordinate system on the fitted road surface plane; based on the unit vectors of each coordinate axis in the road surface coordinate system, determining the real-time rotation matrix of the vehicle body relative to the road surface.

[0009] In any of the above technical solutions, further, the real-time rotation matrix A of the vehicle body relative to the road surface -1 The calculation formula is:

[0010] A -1 =A T

[0011]

[0012] Where A is the body rotation matrix, is the unit vector of each coordinate axis in the road coordinate system;

[0013] The coordinate transformation formula of the jth pixel in the equivalent image is:

[0014]

[0015] In the formula, z j is the pixel depth of the jth pixel in the equivalent image, (u j ,v j) is the pixel coordinate of the jth pixel in the equivalent image, K is the intrinsic parameter matrix of the vehicle-mounted surround view camera, R0 is the design rotation matrix when the imaging plane of the vehicle-mounted surround view camera is perpendicular to the road surface, R1 is the installation rotation matrix of the vehicle-mounted surround view camera, d(u i ,v i ) is the predicted depth value of the i-th pixel in the current image, (u i ,v i ) is the pixel coordinate of the i-th pixel in the current image.

[0016] In any of the above technical solutions, further, the method also includes: generating a BEV grid based on a preset size; calculating the number of the imaging plane corresponding to the center point coordinates of any BEV grid unit in the BEV grid and the projection coordinates on the imaging plane by projection based on the parameters of the on-board surround view camera, wherein the number of the imaging plane is determined by the number of the lens in the on-board surround view camera; establishing a BEV grid feature index based on the center point coordinates of the BEV grid unit, the number of the imaging plane and the horizontal projection coordinates in the projection coordinates;

[0017] Based on the BEV grid feature index, the feature vector in the pooled feature map is mapped to the BEV grid unit to generate the BEV feature, which specifically includes: obtaining the horizontal coordinates corresponding to the feature vector in the pooled feature map and the number of the shot corresponding to the pooled feature map; according to the horizontal coordinates and the number of the corresponding shot, searching the BEV grid feature index for the center point coordinates of the corresponding BEV grid unit to map the feature vector in the pooled feature map to the corresponding BEV grid unit to generate the BEV feature.

[0018] In any of the above technical solutions, further, the method also includes: when it is determined that the center point coordinates of any BEV grid unit correspond to the numbers of two imaging planes, the corresponding two feature vectors to be mapped are summed and pooled and then mapped to the corresponding BEV grid unit.

[0019] The technical solution of the second aspect of the present application is: a BEV feature mapping system suitable for changes in vehicle body posture is provided, the system comprising: an image processing unit, the image processing unit is configured to reproject the current image acquired by the on-board surround view camera based on the real-time rotation matrix of the vehicle body relative to the road surface to generate an equivalent image; a feature processing unit, the feature processing unit is configured to extract image features in the equivalent image, generate a feature map, and pool the feature map along the vertical direction of the feature map; a feature mapping unit, the feature mapping unit is configured to map the feature vector in the pooled feature map to the BEV grid unit based on the BEV grid feature index to generate a BEV feature.

[0020] In any of the above technical solutions, further, the system also includes a matrix calculation unit, which is configured to: establish a vehicle body rectangular coordinate system, and based on the acquired depth map and camera internal and external parameters, map the road surface acquired by the vehicle-mounted surround view camera to the vehicle body rectangular coordinate system to generate a road surface point cloud; based on the road surface point cloud, generate a fitted road surface plane, and calculate the road surface normal vector of the fitted road surface plane; based on the fitted road surface plane and the road surface normal vector, establish a road surface coordinate system, wherein the origin of the road surface coordinate system is the projection point of the origin of the vehicle body coordinate system on the fitted road surface plane, the z-axis of the road surface coordinate system is the road surface normal vector, and the x-axis of the road surface coordinate system is determined by the projection of the x-axis of the vehicle body coordinate system on the fitted road surface plane; based on the unit vectors of each coordinate axis in the road surface coordinate system, determine the real-time rotation matrix of the vehicle body relative to the road surface.

[0021] In any of the above technical solutions, further, the real-time rotation matrix A of the vehicle body relative to the road surface -1 The calculation formula is:

[0022] A -1 =A T

[0023]

[0024] Where A is the body rotation matrix, is the unit vector of each coordinate axis in the road coordinate system;

[0025] The coordinate transformation formula of the jth pixel in the equivalent image is:

[0026]

[0027] In the formula, z j is the pixel depth of the jth pixel in the equivalent image, (u j ,v j ) is the pixel coordinate of the jth pixel in the equivalent image, K is the intrinsic parameter matrix of the vehicle-mounted surround view camera, R0 is the design rotation matrix when the imaging plane of the vehicle-mounted surround view camera is perpendicular to the road surface, R1 is the installation rotation matrix of the vehicle-mounted surround view camera, d(u i ,v i ) is the predicted depth value of the i-th pixel in the current image, (u i ,v i ) is the pixel coordinate of the i-th pixel in the current image.

[0028] In any of the above technical solutions, further, the system also includes an index generation unit, which is configured to: generate a BEV grid based on a preset size; calculate the number of the imaging plane corresponding to the center point coordinates of any BEV grid unit in the BEV grid and the projection coordinates on the imaging plane by projection based on the parameters of the on-board surround view camera, wherein the number of the imaging plane is determined by the number of the lens in the on-board surround view camera; establish a BEV grid feature index based on the center point coordinates of the BEV grid unit, the number of the imaging plane and the horizontal projection coordinates in the projection coordinates;

[0029] The feature mapping unit generates BEV features and is specifically configured to: obtain the horizontal coordinates corresponding to the feature vectors in the pooled feature map, and the number of the shot corresponding to the pooled feature map; according to the horizontal coordinates and the number of the corresponding shot, search the center point coordinates of the corresponding BEV grid unit in the BEV grid feature index to map the feature vectors in the pooled feature map to the corresponding BEV grid unit to generate BEV features.

[0030] The technical solution of the third aspect of the present application is: a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, a BEV feature mapping method suitable for vehicle body posture changes as described in any one of the technical solutions of the first aspect of the present application is implemented.

[0031] The beneficial effects of this application are:

[0032] The technical solution in this application helps to improve the accuracy of generating BEV features when the body posture changes. In the process of generating BEV features, the imaging plane of the on-board camera is first transformed to an ideal state perpendicular to the road surface through the calculated real-time rotation matrix, thereby eliminating the geometric error caused by the change of the body posture and improving the accuracy of BEV representation. Then, along the vertical direction, the feature map of the image at this time is 2D pooled, and a feature index is established, and the pooled features are directly mapped to the BEV plane to generate BEV features, effectively reducing the computational complexity.

[0033] This application also establishes a BEV grid feature index, matches the center point of each BEV grid unit with the feature corresponding to the imaging plane of the vehicle camera, and can complete the construction of the index mapping at one time during initialization. In this way, during the real-time reasoning process, you only need to query the index mapping and complete the necessary feature pooling to obtain the complete BEV features, without having to recalculate the feature mapping every time you process, which significantly improves the computing efficiency and memory usage efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The advantages of the above and / or additional aspects of the present application will become apparent and easily understood in the description of the embodiments in conjunction with the following drawings, in which:

[0035] Figure 1 is a schematic flow chart of a BEV feature mapping method applicable to vehicle body posture changes according to an embodiment of the present application;

[0036] Figure 2 It is a schematic block diagram of a BEV feature map suitable for vehicle body posture changes according to an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0038] In the following description, many specific details are elaborated to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present application is not limited to the specific embodiments disclosed below.

[0039] Embodiment 1:

[0040] In a vehicle with an autonomous driving function, an inertial navigation unit IMU and a vehicle-mounted surround view camera are usually installed on the vehicle body. The vehicle-mounted surround view camera is usually composed of multiple, such as 6, vehicle-mounted cameras, so as to be able to obtain images around the vehicle in real time.

[0041] During the installation process, the imaging plane of each on-board camera is required to be perpendicular to the ground. However, due to installation errors and the errors of the on-board cameras themselves, it is usually difficult to achieve the perpendicularity of the above two planes. Therefore, it is necessary to calibrate the installed on-board cameras, obtain the internal and external parameters of the on-board cameras, and then transform the images (imaging planes) taken by each on-board camera to a plane perpendicular to the ground through coordinate transformation to ensure the accuracy of the point cloud data of the surrounding environment obtained through the image. Then, based on the posture of the vehicle on a flat road and the coordinate system of the on-board surround view camera, the correspondence between the point cloud and the BEV grid unit is established to generate the final BEV features.

[0042] However, the posture of the vehicle body is not constant during driving. For example, acceleration, deceleration, turning, etc. during driving will cause the relative position relationship between the vehicle body and the ground to change, making the imaging plane of the on-board camera no longer perpendicular to the ground. If the BEV feature is still generated in the above way, the BEV feature will be inaccurate.

[0043] like Figure 1 As shown, this embodiment provides a BEV feature mapping method suitable for vehicle body posture changes, the method comprising:

[0044] Step 1: Based on the real-time rotation matrix of the vehicle body relative to the road surface, the current image acquired by the on-board surround view camera is reprojected to generate an equivalent image;

[0045] Specifically, the real-time rotation matrix of the vehicle's current position relative to the road surface can be obtained by fitting the road surface and transforming the coordinate system. The imaging plane of the vehicle-mounted surround view camera is transformed through the real-time rotation matrix to ensure that the imaging plane of each vehicle-mounted camera is perpendicular to the road surface.

[0046] Therefore, in order to accurately obtain the real-time rotation matrix of the vehicle body relative to the road surface, this embodiment also shows a method for calculating the real-time rotation matrix, and the process specifically includes:

[0047] Step 101: Establishing a rectangular coordinate system of the vehicle body, and mapping the road surface acquired by the vehicle surround view camera to the rectangular coordinate system of the vehicle body based on the acquired depth map and the camera internal and external parameters, to generate a road surface point cloud;

[0048] Specifically, the installation point of the inertial navigation unit IMU, the center of gravity of the vehicle, the center point of the rear axle of the vehicle, etc. can be used as the origin O v , establish the body rectangular coordinate system (O v -X v -Y v -Z v ), where X v Point to the front of the car, Y v Pointing to the left side of the vehicle, Z v Perpendicular to X v -Y v At the same time, for each camera in the vehicle-mounted surround view camera, a corresponding camera coordinate system is established. Each camera coordinate system takes the optical center of the corresponding camera as the origin, the front of the field of view is the y-axis direction, the right side is the x-axis direction, and the z-axis is perpendicular to the xoy plane and points to the ground.

[0049] The depth map corresponding to the image taken by each on-board camera is obtained through the monocular depth prediction network Mono Depth Net. Then the PSPNet model is used to perform semantic segmentation on the image from the front-view on-board camera to extract the road surface area. For any pixel point in the image of the extracted road surface area, the calculation formula for converting it into the point cloud coordinate in the camera coordinate system is:

[0050]

[0051] In the formula, (x c ,y c , zc ) is the point cloud coordinate in the camera coordinate system, K is the intrinsic parameter matrix of the vehicle-mounted camera, and d(u, v) is the depth value corresponding to the pixel point (u, v).

[0052] After that, the external parameter matrix is ​​used to transform the point cloud coordinates in the camera coordinate system to the vehicle body rectangular coordinate system. The corresponding calculation formula is:

[0053]

[0054] In the formula, (x v ,y v , z v ) is the point cloud coordinate in the rectangular coordinate system of the vehicle body, and R and T are the external parameter matrices (rotation matrix and offset matrix) of the vehicle camera.

[0055] Using the above process, the image of the extracted road surface area can be converted into the corresponding road surface point cloud.

[0056] Step 102: Based on the road surface point cloud, a RANSAC plane fitting algorithm is used to generate a fitted road surface plane, and a road surface normal vector of the fitted road surface plane is calculated; the road surface equation corresponding to the road surface plane, i.e., the corresponding road surface normal vector n, can be set as:

[0057] z=ax+by+c

[0058]

[0059] Where a, b, and c are pavement fitting parameters.

[0060] Step 103: Based on the fitted road surface plane and the road surface normal vector, a road surface coordinate system is established, wherein the origin of the road surface coordinate system is the projection point of the origin of the vehicle body coordinate system on the fitted road surface plane, the z-axis of the road surface coordinate system is the road surface normal vector, and the x-axis of the road surface coordinate system is determined by the projection of the x-axis of the vehicle body coordinate system on the fitted road surface plane;

[0061] Specifically, the origin of the vehicle body coordinate system is projected onto the fitted road surface plane using the road surface normal vector n, and the projection point is recorded as r , which is the origin of the road surface coordinate system, corresponding to the coordinate (O r1 , O r2 , O r3 ) is calculated as:

[0062]

[0063] The projection point is denoted as O r As the origin, establish the road surface coordinate system, where the Z axis Zr of the road surface coordinate system is the road surface normal vector n, and then determine the X axis and Y axis of the road surface coordinate system. The specific process is:

[0064] Calculate the x-axis O in the vehicle body coordinate system v X v Projection on the road coordinate system and normalization, combined with the origin of the road coordinate system O r , we can get the O of the road surface coordinate system r X r Axis, the corresponding direction vector expression is:

[0065]

[0066] For the Y axis of the road coordinate system, it is perpendicular to the XOZ plane, so its corresponding vector satisfies:

[0067] O r Y r =O r Z r ×O r X r

[0068] Therefore, the road surface coordinate system O r Y r The direction vector expression of the axis is:

[0069]

[0070] Step 104: Based on the unit vectors of each coordinate axis in the road surface coordinate system, determine the real-time rotation matrix of the vehicle body relative to the road surface.

[0071] Specifically, for the convenience of expression, the direction vector expressions of the above three axes are recorded as: At this point, the rotation matrix converted from the road coordinate system to the vehicle body coordinate system can be obtained, that is, the vehicle body rotation matrix A is:

[0072]

[0073] Because the body rotation matrix A is an orthogonal matrix, the real-time rotation matrix A of the body relative to the road surface is -1 The calculation formula is:

[0074] A -1 =A T

[0075] After obtaining the real-time rotation matrix A of the vehicle body relative to the road surface -1 After that, the current image acquired by the on-board surround view camera can be reprojected to a plane perpendicular to the fitted road surface plane to generate an equivalent image, where the coordinate transformation formula of the j-th pixel point in the equivalent image is:

[0076]

[0077] In the formula, z j is the pixel depth of the jth pixel in the equivalent image, (u j , v j ) is the pixel coordinate of the jth pixel in the equivalent image, K is the intrinsic parameter matrix of the vehicle-mounted surround view camera, R0 is the design rotation matrix when the imaging plane of the vehicle-mounted surround view camera is perpendicular to the road surface, R1 is the installation rotation matrix of the vehicle-mounted surround view camera, which can be determined by camera calibration, d(u i , v i ) is the predicted depth value of the i-th pixel in the current image, (u i , v i ) is the pixel coordinate of the i-th pixel in the current image.

[0078] Through the above process, not only can the real-time rotation matrix between the vehicle body and the road surface be obtained to accurately reflect the relationship between the current posture of the vehicle and the road surface, but also the problem of the non-perpendicularity between the imaging plane of the on-board camera and the ground plane caused by errors can be optimized, which helps to improve the accuracy of BEV features.

[0079] Step 2: Extract image features from the equivalent image, generate a feature map, and pool the feature map along the vertical direction of the feature map;

[0080] Specifically, since the image (imaging plane) acquired by the vehicle-mounted camera is perpendicular to the ground (fitted road surface plane) through the above process, there is no need to adopt the method of "first calculating the dense spatial feature point cloud and then collecting the point cloud to the BEV grid" as in the LSS method. Therefore, the feature map obtained can be directly pooled in the vertical direction, and then mapped to the BEV grid according to the set mapping rules. The pooling operation can be maximum pooling, average pooling and other pooling methods, and the specific process will not be repeated here.

[0081] In any of the above embodiments, further, this embodiment shows a method for establishing a BEV grid feature index, and the process specifically includes:

[0082] Generate a BEV grid based on the preset size;

[0083] Based on the parameters of the on-board surround view camera, the number of the imaging plane corresponding to the coordinates of the center point of any BEV grid unit in the BEV grid and the projection coordinates on the imaging plane are calculated by projection, wherein the number of the imaging plane is determined by the number of the lens in the on-board surround view camera;

[0084] Specifically, it is assumed that the vehicle-mounted surround view camera in this embodiment is composed of 6 vehicle-mounted cameras, which are arranged in the front, right front, right rear, rear, left rear, and left front of the vehicle in a clockwise direction, and the corresponding numbers are 0-5, and each vehicle-mounted camera has an imaging plane, which can obtain an image and calculate a feature map. Therefore, the feature map corresponding to each vehicle-mounted camera after the transformation can be recorded as F i , i is the number of the feature map / image / imaging plane / vehicle camera, i=0,1,...,5.

[0085] Those skilled in the art will understand that, for any pixel in the image acquired by the vehicle-mounted camera, if its corresponding depth is known (such as predicted by the monocular depth prediction network Mono Depth Net), then, under the condition that the parameters of the vehicle-mounted camera are known, the three-dimensional coordinates of the pixel in the world / road coordinate system can be calculated by projection and coordinate system transformation, and the three-dimensional coordinates must be located in a unit in the BEV grid.

[0086] Therefore, through the inverse operation of the above process, the pixel points in the corresponding image at the center point coordinates of any BEV grid unit in the BEV grid can be calculated, and each pixel point corresponds to a position coordinate on the imaging plane of the vehicle-mounted camera, so the projection coordinates of the center point coordinates of any BEV grid unit in the BEV grid on the corresponding imaging plane can be calculated.

[0087] The BEV grid feature index is established based on the center point coordinates of the BEV grid unit, the number of the imaging plane, and the horizontal projection coordinates in the projection coordinates.

[0088] Specifically, since the above feature map is pooled in the vertical direction, and the feature vectors in the pooled feature map can only correspond one-to-one to the horizontal coordinates in the original feature map, the corresponding index form can be:

[0089] C bev [nx][ny]=C i [u]

[0090] Where (nx,ny) is the coordinate of the center point of the BEV grid unit, C bev [nx][ny] is the BEV feature with the center point coordinates (nx,ny), i is the number of the imaging plane, u is the horizontal projection coordinate in the projection coordinates on the imaging plane, C i [u] represents the feature vector with horizontal projection coordinate u in the pooled feature map corresponding to the i-th imaging plane.

[0091] Step 3: Based on the BEV grid feature index, map the feature vector in the pooled feature map to the BEV grid unit to generate the BEV feature.

[0092] Since the obtained feature map is directly pooled in the vertical direction in this embodiment, the feature vectors in the pooled feature map can only correspond one-to-one to the horizontal coordinates of the original feature map. Therefore, the above step 3 specifically includes:

[0093] Get the horizontal coordinates corresponding to the feature vectors in the pooled feature map, and the number of the shot corresponding to the pooled feature map;

[0094] According to the horizontal coordinates and the number of the corresponding lens, the center point coordinates of the corresponding BEV grid unit are found in the BEV grid feature index to map the feature vector in the pooled feature map to the corresponding BEV grid unit to generate BEV features.

[0095] In any of the above embodiments, further, considering that there may be overlapping fields of view between the various on-board cameras in the on-board surround view camera, in the process of establishing the BEV grid feature index and the feature vector mapping, the BEV grid unit located in the overlapping field of view will correspond to the two on-board cameras. Therefore, in order to ensure the accuracy of the BEV feature, the method further includes:

[0096] When it is determined that the coordinates of the center point of any BEV grid unit correspond to the numbers of two imaging planes, the corresponding two feature vectors to be mapped are summed and pooled, and then mapped to the corresponding BEV grid unit.

[0097] It should be noted that after performing the combined pooling operation, the pooled result can be divided by the number of corresponding imaging planes to obtain the mean of the two features, and then the mean is mapped to the corresponding BEV grid unit.

[0098] Embodiment 2:

[0099] like Figure 2 As shown, this embodiment provides a BEV feature mapping system 100 suitable for vehicle body posture changes, and the system 100 includes:

[0100] An image processing unit 10, the image processing unit 10 is configured to reproject a current image acquired by the vehicle-mounted surround view camera based on a real-time rotation matrix of the vehicle body relative to the road surface to generate an equivalent image;

[0101] A feature processing unit 20, the feature processing unit 20 is configured to extract image features in the equivalent image, generate a feature map, and pool the feature map along a vertical direction of the feature map;

[0102] The feature mapping unit 30 is configured to map the feature vector in the pooled feature map to the BEV grid unit based on the BEV grid feature index to generate the BEV feature.

[0103] In any of the above embodiments, further, the system further includes a matrix calculation unit 40, and the matrix calculation unit 40 is configured as follows:

[0104] Establish a rectangular coordinate system for the vehicle body, and based on the acquired depth map and camera internal and external parameters, map the road surface acquired by the on-board surround view camera to the rectangular coordinate system of the vehicle body to generate a road surface point cloud;

[0105] Based on the road surface point cloud, a fitted road surface plane is generated, and a road surface normal vector of the fitted road surface plane is calculated;

[0106] Based on the fitted road surface plane and the road surface normal vector, a road surface coordinate system is established, wherein the origin of the road surface coordinate system is the projection point of the origin of the vehicle body coordinate system on the fitted road surface plane, the z-axis of the road surface coordinate system is the road surface normal vector, and the x-axis of the road surface coordinate system is determined by the projection of the x-axis of the vehicle body coordinate system on the fitted road surface plane;

[0107] Based on the unit vectors of each coordinate axis in the road surface coordinate system, the real-time rotation matrix of the vehicle body relative to the road surface is determined.

[0108] In any of the above embodiments, further, the real-time rotation matrix A of the vehicle body relative to the road surface is -1 The calculation formula is:

[0109] A -1 =A T

[0110]

[0111] Where A is the body rotation matrix, is the unit vector of each coordinate axis in the road coordinate system;

[0112] The coordinate transformation formula of the jth pixel in the equivalent image is:

[0113]

[0114] In the formula, z j is the pixel depth of the jth pixel in the equivalent image, (u j ,v j ) is the pixel coordinate of the jth pixel in the equivalent image, K is the intrinsic parameter matrix of the vehicle-mounted surround view camera, R0 is the design rotation matrix when the imaging plane of the vehicle-mounted surround view camera is perpendicular to the road surface, R1 is the installation rotation matrix of the vehicle-mounted surround view camera, d(u i ,v i ) is the predicted depth value of the i-th pixel in the current image, (u i ,v i ) is the pixel coordinate of the i-th pixel in the current image.

[0115] In any of the above embodiments, further, the system further includes an index generating unit 50, and the index generating unit 50 is configured to:

[0116] Generate a BEV grid based on the preset size;

[0117] Based on the parameters of the on-board surround view camera, the number of the imaging plane corresponding to the coordinates of the center point of any BEV grid unit in the BEV grid and the projection coordinates on the imaging plane are calculated by projection, wherein the number of the imaging plane is determined by the number of the lens in the on-board surround view camera;

[0118] Establishing a BEV grid feature index based on the center point coordinates of the BEV grid unit, the number of the imaging plane, and the horizontal projection coordinates in the projection coordinates;

[0119] The feature mapping unit 30 generates BEV features, and is specifically configured as follows:

[0120] Get the horizontal coordinates corresponding to the feature vectors in the pooled feature map, and the number of the shot corresponding to the pooled feature map;

[0121] According to the horizontal coordinates and the number of the corresponding lens, the center point coordinates of the corresponding BEV grid unit are found in the BEV grid feature index to map the feature vector in the pooled feature map to the corresponding BEV grid unit to generate BEV features.

[0122] In any of the above embodiments, further, the feature mapping unit 30 is further configured to:

[0123] When it is determined that the coordinates of the center point of any BEV grid unit correspond to the numbers of two imaging planes, the corresponding two feature vectors to be mapped are summed and pooled and then mapped to the corresponding BEV grid unit.

[0124] Embodiment 3:

[0125] This embodiment further provides a computer-readable storage medium, in which a computer program is stored. When the stored computer program is executed by a processor, the BEV feature mapping method applicable to vehicle body posture changes of any one of the above-mentioned embodiments 1 is implemented.

[0126] Specifically, the computer program includes computer instructions, which are stored in a computer-readable storage medium; the processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method steps in any one of the above-mentioned embodiments 1.

[0127] The above-mentioned computer-readable storage medium may include random access memory (RAM), read-only memory (ROM), compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage or other magnetic storage devices, etc.

[0128] The above-mentioned processor may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0129] So far, various embodiments of the present application have been described in detail. In order to avoid obscuring the concept of the present application, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed herein.

[0130] Although some specific embodiments of the present application have been described in detail through examples, those skilled in the art should understand that the above examples are only for illustration rather than for limiting the scope of the present application.

[0131] The steps in this application can be adjusted in order, combined, and deleted according to actual needs.

[0132] Although the present application is disclosed in detail with reference to the accompanying drawings, it should be understood that these descriptions are merely exemplary and are not intended to limit the application of the present application. The scope of protection of the present application is defined by the appended claims and may include various modifications, alterations and equivalents made to the invention without departing from the scope and spirit of the present application.

Claims

1. A BEV feature mapping method suitable for vehicle body posture changes, characterized in that: The method comprises: Based on the real-time rotation matrix of the vehicle body relative to the road surface, the current image acquired by the on-board surround view camera is reprojected to generate an equivalent image; Extracting image features from the equivalent image to generate a feature map, and performing pooling on the feature map along a vertical direction of the feature map; Based on the BEV grid feature index, the feature vector in the pooled feature map is mapped to the BEV grid unit to generate the BEV feature.

2. The BEV feature mapping method applicable to vehicle body posture changes according to claim 1, characterized in that: The method further comprises: Establishing a rectangular coordinate system of the vehicle body, and based on the acquired depth map and camera internal and external parameters, mapping the road surface acquired by the on-board surround view camera to the rectangular coordinate system of the vehicle body to generate a road surface point cloud; Based on the road surface point cloud, generating a fitted road surface plane, and calculating a road surface normal vector of the fitted road surface plane; Based on the fitted road surface plane and the road surface normal vector, a road surface coordinate system is established, wherein the origin of the road surface coordinate system is the projection point of the origin of the vehicle body coordinate system on the fitted road surface plane, the z-axis of the road surface coordinate system is the road surface normal vector, and the x-axis of the road surface coordinate system is determined by the projection of the x-axis of the vehicle body coordinate system on the fitted road surface plane; The real-time rotation matrix of the vehicle body relative to the road surface is determined based on the unit vectors of each coordinate axis in the road surface coordinate system.

3. The BEV feature mapping method applicable to vehicle body posture changes as claimed in claim 2, characterized in that: The real-time rotation matrix A of the vehicle body relative to the road surface -1 The calculation formula is: A -1 =A T Where A is the body rotation matrix, is the unit vector of each coordinate axis in the road surface coordinate system; The coordinate transformation formula of the j-th pixel in the equivalent image is: In the formula, z j is the pixel depth of the jth pixel in the equivalent image, (u j ,v j ) is the pixel coordinate of the jth pixel in the equivalent image, K is the intrinsic parameter matrix of the vehicle-mounted surround view camera, R0 is the design rotation matrix when the imaging plane of the vehicle-mounted surround view camera is perpendicular to the road surface, R1 is the installation rotation matrix of the vehicle-mounted surround view camera, d(u i ,v i ) is the predicted depth value of the i-th pixel in the current image, (u i ,v i ) is the pixel coordinate of the i-th pixel in the current image.

4. The BEV feature mapping method applicable to vehicle body posture changes according to claim 1, characterized in that: The method further comprises: Generate a BEV grid based on the preset size; Based on the parameters of the vehicle-mounted surround view camera, the number of the imaging plane corresponding to the coordinates of the center point of any BEV grid unit in the BEV grid and the projection coordinates on the imaging plane are calculated by projection, wherein the number of the imaging plane is determined by the number of the lens in the vehicle-mounted surround view camera; Establishing the BEV grid feature index based on the center point coordinates of the BEV grid unit, the number of the imaging plane, and the horizontal projection coordinates in the projection coordinates; Based on the BEV grid feature index, the feature vector in the pooled feature map is mapped to the BEV grid unit to generate the BEV feature, specifically including: Obtaining a horizontal coordinate corresponding to a feature vector in the pooled feature map and a number of the shot corresponding to the pooled feature map; According to the horizontal coordinate and the corresponding number of the lens, the center point coordinates of the corresponding BEV grid unit are searched in the BEV grid feature index to map the feature vector in the pooled feature map to the corresponding BEV grid unit to generate the BEV feature.

5. The BEV feature mapping method applicable to vehicle body posture changes as claimed in claim 4, characterized in that: The method further comprises: When it is determined that the coordinates of the center point of any BEV grid unit correspond to the numbers of two imaging planes, the corresponding two feature vectors to be mapped are summed and pooled and then mapped to the corresponding BEV grid unit.

6. A BEV feature mapping system suitable for vehicle body posture changes, characterized in that: The system comprises: An image processing unit, wherein the image processing unit is configured to reproject a current image acquired by the vehicle-mounted surround view camera based on a real-time rotation matrix of the vehicle body relative to the road surface to generate an equivalent image; A feature processing unit, wherein the feature processing unit is configured to extract image features in the equivalent image, generate a feature map, and perform pooling on the feature map along a vertical direction of the feature map; A feature mapping unit is configured to map feature vectors in the pooled feature map to BEV grid cells based on the BEV grid feature index to generate BEV features.

7. The BEV feature mapping system suitable for vehicle body posture change according to claim 6, characterized in that: The system further comprises a matrix calculation unit, wherein the matrix calculation unit is configured to: Establishing a rectangular coordinate system of the vehicle body, and based on the acquired depth map and camera internal and external parameters, mapping the road surface acquired by the on-board surround view camera to the rectangular coordinate system of the vehicle body to generate a road surface point cloud; Based on the road surface point cloud, generating a fitted road surface plane, and calculating a road surface normal vector of the fitted road surface plane; Based on the fitted road surface plane and the road surface normal vector, a road surface coordinate system is established, wherein the origin of the road surface coordinate system is the projection point of the origin of the vehicle body coordinate system on the fitted road surface plane, the z-axis of the road surface coordinate system is the road surface normal vector, and the x-axis of the road surface coordinate system is determined by the projection of the x-axis of the vehicle body coordinate system on the fitted road surface plane; The real-time rotation matrix of the vehicle body relative to the road surface is determined based on the unit vectors of each coordinate axis in the road surface coordinate system.

8. The BEV feature mapping system suitable for vehicle body posture change according to claim 7, characterized in that: The real-time rotation matrix A of the vehicle body relative to the road surface -1 The calculation formula is: A -1 =A T Where A is the body rotation matrix, is the unit vector of each coordinate axis in the road surface coordinate system; The coordinate transformation formula of the j-th pixel in the equivalent image is: In the formula, z j is the pixel depth of the jth pixel in the equivalent image, (u j ,v j ) is the pixel coordinate of the jth pixel in the equivalent image, K is the intrinsic parameter matrix of the vehicle-mounted surround view camera, R0 is the design rotation matrix when the imaging plane of the vehicle-mounted surround view camera is perpendicular to the road surface, R1 is the installation rotation matrix of the vehicle-mounted surround view camera, d(u i ,v i ) is the predicted depth value of the i-th pixel in the current image, (u i ,v i ) is the pixel coordinate of the i-th pixel in the current image.

9. The BEV feature mapping system suitable for vehicle body posture change according to claim 6, characterized in that: The system further includes an index generating unit, wherein the index generating unit is configured to: Generate a BEV grid based on the preset size; Based on the parameters of the vehicle-mounted surround view camera, the number of the imaging plane corresponding to the coordinates of the center point of any BEV grid unit in the BEV grid and the projection coordinates on the imaging plane are calculated by projection, wherein the number of the imaging plane is determined by the number of the lens in the vehicle-mounted surround view camera; Establishing the BEV grid feature index based on the center point coordinates of the BEV grid unit, the number of the imaging plane, and the horizontal projection coordinates in the projection coordinates; The feature mapping unit generates the BEV feature and is specifically configured as follows: Obtaining a horizontal coordinate corresponding to a feature vector in the pooled feature map and a number of the shot corresponding to the pooled feature map; According to the horizontal coordinate and the corresponding number of the lens, the center point coordinates of the corresponding BEV grid unit are searched in the BEV grid feature index to map the feature vector in the pooled feature map to the corresponding BEV grid unit to generate the BEV feature.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the BEV feature mapping method applicable to vehicle body posture changes according to any one of claims 1 to 5 is implemented.

Images