Image inverse perspective transformation method, device, electronic device, and storage medium

Abstract

The present application discloses a method, device, electronic device, and storage medium for inverse perspective transformation of an image. The method includes: obtaining image data of a current frame and a previous frame, including an original image and a corresponding semantic segmentation image; obtaining inertial navigation data at corresponding moments of the two frames of image data, including the angle of the inertial navigation device relative to the ground plane and the vehicle posture; determining the angle of the inertial navigation device relative to the ground based on the inertial navigation data at corresponding moments of the two frames of image data; determining the height of the camera relative to the ground based on the two frames of image data and the vehicle posture; correcting the external parameters of the camera relative to the ground based on the angle of the inertial navigation device relative to the ground and the height of the camera relative to the ground, thereby performing IPM processing on the semantic segmentation image of the current frame. The present application combines inertial navigation data to determine the external parameters of the camera relative to the ground, is not affected by the external environment, and has a high output frequency, meeting real-time requirements. Based on this, the external parameters of the camera relative to the ground are corrected in real time, which greatly improves the IPM effect.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a method, device, electronic device, and storage medium for inverse perspective transformation of an image. Background Art

[0002] Inverse Perspective Mapping (IPM) refers to the process of projecting two-dimensional pixel points in the image plane onto the ground to obtain three-dimensional spatial points.

[0003] In the field of autonomous driving, visual semantic mapping or vehicle-assisted localization is often performed using ground landmarks such as lane markings, arrows, and stop signs in camera-captured images. Therefore, IPM plays a crucial role in this field. In scenarios like visual semantic mapping or vehicle-assisted localization, given the camera's internal and external parameters, IPM can be used to convert the two-dimensional pixels of the semantically segmented image into three-dimensional spatial points on the ground. This can then be used to construct a semantic map or to match and localize the image with an existing semantic map.

[0004] However, the existing IPM process has at least the following problems:

[0005] 1) IPM relies on the camera's external parameters relative to the ground. However, the unevenness of the road caused by the vehicle's ups and downs during driving can affect the IPM effect, such as the common ghosting problem.

[0006] 2) Using vanishing points to calculate vehicle undulation requires ensuring that lane lines on the actual road are parallel and that the straight line equation in the image is accurately fitted. However, actual road conditions may not always meet these requirements, and the accuracy of the straight line fitting is also affected by the semantic segmentation effect.

[0007] 3) The changes in the vehicle's ups and downs during driving will not only change the camera's angle relative to the ground, but also affect the camera's height from the ground. Therefore, the change in the camera's height from the ground also needs to be considered. Summary of the Invention

[0008] The embodiments of the present application provide a method, device, electronic device, and storage medium for inverse perspective transformation of an image, so as to adaptively adjust the parameters of the inverse perspective transformation and improve the effect of the inverse perspective transformation.

[0009] The embodiments of this application adopt the following technical solutions:

[0010] In a first aspect, an embodiment of the present application provides a method for inverse perspective transformation of an image, wherein the method comprises:

[0011] Acquire image data of a current frame and image data of a previous frame, wherein the image data includes an original image and a semantic segmentation image corresponding to the original image;

[0012] Acquire inertial navigation data at corresponding moments of the image data of the current frame and the image data of the previous frame, the inertial navigation data including an angle of the inertial navigation device relative to the ground plane and a vehicle posture output by the inertial navigation device;

[0013] determining an angle of the inertial navigation device relative to the ground based on the image data of the current frame and the inertial navigation data at a corresponding moment of the image data of the previous frame;

[0014] Determining the height of the camera relative to the ground based on the image data of the current frame and the image data of the previous frame, and the vehicle posture output by the inertial navigation device;

[0015] The extrinsic parameters of the camera relative to the ground are corrected according to the angle of the inertial navigation device relative to the ground and the height of the camera relative to the ground, and the semantic segmentation image of the current frame is inversely transformed using the corrected extrinsic parameters of the camera relative to the ground to obtain an image after inverse perspective transformation.

[0016] Optionally, before acquiring the inertial navigation data at the corresponding moments of the image data of the current frame and the image data of the previous frame, the method further includes:

[0017] Obtaining IMU measurement data from an inertial navigation device and the corresponding vehicle posture, wherein the IMU measurement data includes acceleration and angular velocity;

[0018] fusing the IMU measurement data using a Kalman filter to obtain an angle of the inertial navigation device relative to the ground plane;

[0019] The angle of the inertial navigation device relative to the ground plane and the corresponding vehicle posture are used as the inertial navigation data.

[0020] Optionally, determining the angle of the inertial navigation device relative to the ground based on the image data of the current frame and the inertial navigation data at corresponding moments of the image data of the previous frame includes:

[0021] Determining a relative translation between the current frame and the previous frame based on the vehicle posture at the time corresponding to the image data of the current frame and the vehicle posture at the time corresponding to the image data of the previous frame;

[0022] determining an angle between the ground plane and the ground based on the relative translation between the current frame and the previous frame;

[0023] The angle of the inertial navigation device relative to the ground at the moment corresponding to the image data of the current frame is determined according to the angle of the inertial navigation device relative to the ground plane at the moment corresponding to the image data of the current frame and the angle between the ground plane and the ground.

[0024] Optionally, determining the height of the camera relative to the ground based on the image data of the current frame and the image data of the previous frame, and the vehicle posture output by the inertial navigation device includes:

[0025] Track the original image of the previous frame and the original image of the current frame using a preset tracking algorithm to obtain a tracking result;

[0026] The height of the camera relative to the ground is determined according to the tracking result and the vehicle posture output by the inertial navigation device.

[0027] Optionally, determining the height of the camera relative to the ground according to the tracking result and the vehicle posture output by the inertial navigation device includes:

[0028] Get the external parameters of the camera relative to the vehicle;

[0029] Based on the external parameters of the camera relative to the vehicle, the relative posture of the camera in the current frame and the camera in the previous frame are determined using the vehicle posture at the time corresponding to the image data of the current frame and the vehicle posture at the time corresponding to the image data of the previous frame;

[0030] The height of the camera relative to the ground is determined according to the tracking result and the relative position of the camera in the current frame and the camera in the previous frame.

[0031] Optionally, the tracking result includes matched feature points, and determining the height of the camera relative to the ground based on the tracking result and the relative pose of the current frame camera and the previous frame camera includes:

[0032] Based on the relative pose of the current frame camera and the previous frame camera, triangulate the matched feature points using a triangulation algorithm to obtain three-dimensional feature points in the coordinate system of the current frame camera;

[0033] Perform plane fitting on the three-dimensional feature points in the coordinate system of the current frame camera to obtain the height of the camera relative to the ground.

[0034] Optionally, the tracking result includes matching feature points. After tracking the original image of the previous frame and the original image of the current frame using a preset tracking algorithm to obtain the tracking result, the method further includes:

[0035] Determining whether the number of the matched feature points is less than a preset number threshold;

[0036] When the number of the matched feature points is less than a preset threshold, feature extraction is performed on the original image of the current frame using the segmented image of the current frame so that the sum of the number of the extracted feature points and the matched feature points is not less than the preset threshold.

[0037] In a second aspect, an embodiment of the present application further provides a device for inverse perspective transformation of an image, wherein the device comprises:

[0038] A first acquisition unit is configured to acquire image data of a current frame and image data of a previous frame, wherein the image data includes an original image and a semantic segmentation image corresponding to the original image;

[0039] a second acquisition unit, configured to acquire inertial navigation data at corresponding moments of the image data of the current frame and the image data of the previous frame, the inertial navigation data including an angle of the inertial navigation device relative to the ground plane and a vehicle posture output by the inertial navigation device;

[0040] a first determining unit, configured to determine an angle of the inertial navigation device relative to the ground based on the image data of the current frame and the inertial navigation data at a corresponding moment of the image data of the previous frame;

[0041] a second determining unit, configured to determine a height of the camera relative to the ground based on the image data of the current frame and the image data of the previous frame, and a vehicle posture output by the inertial navigation device;

[0042] The inverse perspective transformation unit is used to correct the external parameters of the camera relative to the ground according to the angle of the inertial navigation device relative to the ground and the height of the camera relative to the ground, and use the corrected external parameters of the camera relative to the ground to perform an inverse perspective transformation on the semantic segmentation image of the current frame to obtain an image after inverse perspective transformation.

[0043] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0044] processor; and

[0045] A memory arranged to store computer executable instructions, which when executed cause the processor to perform any of the methods described above.

[0046] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple applications, the electronic device executes any of the aforementioned methods.

[0047] At least one of the above-mentioned technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: the inverse perspective transformation method of the image in the embodiments of the present application first obtains the image data of the current frame and the image data of the previous frame, the image data including the original image and the semantic segmentation image corresponding to the original image; then obtains the inertial navigation data at the corresponding moments of the image data of the current frame and the image data of the previous frame, the inertial navigation data including the angle of the inertial navigation device relative to the ground plane and the vehicle posture output by the inertial navigation device; then, the angle of the inertial navigation device relative to the ground is determined based on the inertial navigation data at the corresponding moments of the image data of the current frame and the image data of the previous frame; then, the height of the camera relative to the ground is determined based on the image data of the current frame and the image data of the previous frame, as well as the vehicle posture output by the inertial navigation device; finally, the external parameters of the camera relative to the ground are corrected according to the angle of the inertial navigation device relative to the ground and the height of the camera relative to the ground, and the semantic segmentation image of the current frame is inversely perspective transformed using the corrected external parameters of the camera relative to the ground to obtain an image after inverse perspective transformation. The inverse perspective transformation method of the image in the embodiment of the present application combines inertial navigation data to determine the angle of the inertial navigation device relative to the ground and the height of the camera relative to the ground. It is not affected by the external environment, and the inertial navigation data has a high output frequency, meeting real-time requirements. Based on this, real-time correction of the external parameters of the camera relative to the ground is performed, thereby improving the inverse perspective transformation effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0049] Figure 1 Schematic diagram of a flow chart of an inverse perspective transformation method of an image in an embodiment of the present application;

[0050] Figure 2 Schematic diagram of the structure of an image inverse perspective transformation device in an embodiment of the present application;

[0051] Figure 3 This is a structural diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0052] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0053] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0054] The embodiment of the present application provides a method for inverse perspective transformation of an image, such as Figure 1 As shown, a flowchart of a method for inverse perspective transformation of an image in an embodiment of the present application is provided. The method at least includes the following steps S110 to S150:

[0055] Step S110 , obtaining image data of a current frame and image data of a previous frame, wherein the image data includes an original image and a semantically segmented image corresponding to the original image.

[0056] When performing inverse perspective transformation on an image, the embodiment of the present application needs to first obtain the image data of the current frame and the image data of the previous frame. The image data here can specifically include the original road image directly output by the camera and the semantic segmentation image output after semantic segmentation processing. The semantic segmentation image can use a trained semantic segmentation model to extract lane lines, arrows, stop lines and other road signs contained in the original image. The object of the inverse perspective transformation of the embodiment of the present application is the semantic segmentation image of the current frame, and the purpose is to convert the two-dimensional pixel points of the road signs in the semantic segmentation image of the current frame into three-dimensional space points in the ground coordinate system.

[0057] The above-mentioned semantic segmentation model can be obtained based on the convolutional neural network training in the prior art, for example, U-NET or LaneNet network can be used. Of course, how to obtain the semantic segmentation image can be flexibly set by those skilled in the art according to actual needs, and no specific limitation is made here.

[0058] Step S120 , obtaining inertial navigation data at corresponding moments of the image data of the current frame and the image data of the previous frame, wherein the inertial navigation data includes an angle of the inertial navigation device relative to the ground plane and a vehicle posture output by the inertial navigation device.

[0059] The inertial navigation device of the embodiment of the present application may refer to a positioning device composed of an IMU (Inertial Measurement Unit) and RTK (Real-time kinematic). RTK is a dynamic differential technology based on GNSS (Global Navigation Satellite System), which can achieve centimeter-level positioning accuracy.

[0060] Since the inverse perspective transformation process needs to rely on the external parameters of the camera relative to the ground, the external parameters of the camera relative to the ground will change during the ups and downs of the vehicle. Among them, the external parameters with greater influence are mainly the pitch angle (pitch) and roll angle (roll) of the camera relative to the ground and the height of the camera relative to the ground, while the yaw angle (yaw) of the camera relative to the ground is not significantly affected. Therefore, the angle of the inertial navigation device relative to the ground plane obtained in the embodiment of the present application mainly refers to the pitch angle (pitch) and roll angle (roll) of the inertial navigation device relative to the ground plane.

[0061] The inertial navigation device can output data such as the angle of the inertial navigation device relative to the ground plane and the vehicle posture at a certain frequency. However, since the sampling frequency of the camera is different from that of the inertial navigation device, what needs to be obtained here is the inertial navigation data at the time corresponding to the image data of the current frame and the inertial navigation data at the time corresponding to the image data of the previous frame.

[0062] Step S130 : determining the angle of the inertial navigation device relative to the ground based on the image data of the current frame and the inertial navigation data at the corresponding moments of the image data of the previous frame.

[0063] The ground plane is a plane perpendicular to the direction of gravity. During the ups and downs of the vehicle, the ground actually seen is not consistent with the ground plane. The ultimate goal of the inverse perspective transformation is to convert the pixel points in the image to the ground. Therefore, the embodiment of the present application needs to convert the angle of the inertial navigation device relative to the ground plane and the vehicle posture at the corresponding moment of the image data of the current frame and the image data of the previous frame obtained in the above steps to the angle of the inertial navigation device relative to the ground plane.

[0064] Step S140 , determining the height of the camera relative to the ground based on the image data of the current frame and the image data of the previous frame, and the vehicle posture output by the inertial navigation device.

[0065] As the vehicle moves up and down, the height of the camera above the ground will also change. Therefore, it is necessary to use the image data of the current frame and the image data of the previous frame, combined with the vehicle posture output by the inertial navigation device, to calculate the height of the camera relative to the ground in real time.

[0066] Compared to the existing method of using vanishing points to calculate the pitch and roll angles of the camera relative to the ground, the inertial navigation data output by the inertial navigation device is not affected by the external environment, has higher accuracy and stability, and has a high output frequency, which can meet the needs of real-time correction.

[0067] It should be noted that there is no strict order between step S130 and step S140.

[0068] Step S150: Correcting the extrinsic parameters of the camera relative to the ground according to the angle of the inertial navigation device relative to the ground and the height of the camera relative to the ground, and performing an inverse perspective transformation on the semantic segmentation image of the current frame using the corrected extrinsic parameters of the camera relative to the ground to obtain an image after the inverse perspective transformation.

[0069] After obtaining the angle of the inertial navigation device relative to the ground, since the transformation between the inertial navigation device and the camera is rigid, the transformation relationship between the two can be calibrated in advance and is less affected by the ups and downs of the vehicle. Therefore, based on the angle of the inertial navigation device relative to the ground and the transformation relationship between the two, the angle of the camera relative to the ground can be converted. The angle of the camera relative to the ground in the current frame can be corrected using the angle of the camera relative to the ground, and the height of the camera relative to the ground in the current frame can be corrected using the height of the camera relative to the ground, thereby obtaining the corrected external parameters of the camera relative to the ground in the current frame.

[0070] Finally, based on the corrected external parameters of the current frame camera relative to the ground, the IPM is used to perform inverse perspective transformation on the semantic segmentation image of the current frame, thereby converting the two-dimensional pixel points of the road markings in the image into three-dimensional spatial points on the ground, which can be used for subsequent mapping or positioning scenarios.

[0071] The inverse perspective transformation method of the image in the embodiment of the present application combines inertial navigation data to determine the angle of the inertial navigation device relative to the ground and the height of the camera relative to the ground. It is not affected by the external environment, and the inertial navigation data has a high output frequency, meeting real-time requirements. Based on this, real-time correction of the external parameters of the camera relative to the ground is performed, thereby improving the inverse perspective transformation effect.

[0072] In some embodiments of the present application, before obtaining the inertial navigation data at the corresponding moments of the image data of the current frame and the image data of the previous frame, the method further includes: obtaining IMU measurement data of the inertial navigation device and the corresponding vehicle posture, the IMU measurement data including acceleration and angular velocity; fusing the IMU measurement data using a Kalman filter to obtain the angle of the inertial navigation device relative to the ground plane; and using the angle of the inertial navigation device relative to the ground plane and the corresponding vehicle posture as the inertial navigation data.

[0073] The inertial navigation data of the embodiments of the present application can be obtained as follows: first, the IMU measurement data of the inertial navigation device is obtained, mainly including the measured values ​​of acceleration and angular velocity. Then, the IMU measurement data is input into a Kalman filter for fusion processing to obtain the pitch angle and roll angle of the inertial navigation device relative to the ground plane. The specific fusion process can be determined based on existing technologies. For example, the prior values ​​of the pitch angle and roll angle can be calculated using the angular velocities of the x-axis and y-axis in the inertial navigation device coordinate system. The measured values ​​of the pitch angle and roll angle can then be calculated using the accelerations of the x-axis, y-axis, and z-axis. Finally, the measured values ​​of the pitch angle and roll angle can be used to update the prior values.

[0074] In addition, in order to improve the fusion accuracy, the IMU measurement data can be denoised using a first-order low-pass filter, and then the denoised IMU measurement data can be fused.

[0075] In some embodiments of the present application, obtaining the inertial navigation data at the corresponding moments of the image data of the current frame and the image data of the previous frame includes: obtaining the inertial navigation data within a preset time length; based on the image data of the current frame and the image data of the previous frame, performing time synchronization processing on the inertial navigation data within the preset time length to obtain the inertial navigation data at the corresponding moments of the image data of the current frame and the image data of the previous frame.

[0076] Since the sampling frequency of the camera is different from that of the inertial navigation device, and the sampling frequency of the inertial navigation device is often higher, there may be multiple inertial navigation data between two frames of image, and the timestamps may not be exactly the same. To ensure the accuracy of data processing, the inertial navigation data within a period of time can be time-synchronized based on the image time corresponding to the image data of the current frame and the image data of the previous frame. For example, the angle of the inertial navigation device relative to the ground plane and the vehicle posture can be interpolated using the interpolation method to obtain the inertial navigation data at the time corresponding to the two frames of image data.

[0077] In some embodiments of the present application, determining the angle of the inertial navigation device relative to the ground based on the image data of the current frame and the inertial navigation data at the corresponding moments of the image data of the previous frame includes: determining the relative translation between the current frame and the previous frame based on the vehicle posture at the corresponding moment of the image data of the current frame and the vehicle posture at the corresponding moment of the image data of the previous frame; determining the angle between the ground plane and the ground based on the relative translation between the current frame and the previous frame; determining the angle of the inertial navigation device relative to the ground at the corresponding moment of the image data of the current frame based on the angle of the inertial navigation device relative to the ground plane and the angle between the ground plane and the ground.

[0078] The vehicle posture at the corresponding moment of two adjacent frames of image data contains the specific position data of the vehicle. Therefore, based on the specific position of the vehicle at the corresponding moment of these two frames of image data, the relative translation between the two frames of image can be calculated. This relative translation can be specifically decomposed into the translation amounts in the three directions of x, y, and z in the vehicle coordinate system. Therefore, based on the relative translation between the two frames of image, the angle between the ground plane and the ground can be calculated. The angle of the inertial navigation device relative to the ground plane at the corresponding moment of the current frame of image data is subtracted from the angle between the ground plane and the ground to obtain the angle of the inertial navigation device relative to the ground at the corresponding moment of the current frame of image data.

[0079] Since the angle of the inertial navigation device relative to the ground plane in the embodiment of the present application includes the pitch angle and the roll angle of the inertial navigation device relative to the ground plane, they can be calculated separately:

[0080] 1) Calculate the angle delta_pitch between the ground plane and the ground based on the pitch angle of the inertial navigation device relative to the ground plane:

[0081] delta_pitch=arcsin(delta_z / delta_x);

[0082] 2) Calculate the angle delta_roll between the ground plane and the ground based on the roll angle of the inertial navigation device relative to the ground plane:

[0083] delta_roll=arcsin(delta_z / delta_y);

[0084] Finally, the pitch angle and roll angle of the inertial navigation device relative to the ground at the time corresponding to the image data of the current frame are subtracted from the above delta_pitch and delta_roll respectively to obtain the pitch angle and roll angle of the inertial navigation device relative to the ground at the time corresponding to the image data of the current frame.

[0085] In some embodiments of the present application, determining the height of the camera relative to the ground based on the image data of the current frame and the image data of the previous frame, and the vehicle posture output by the inertial navigation device includes: tracking the original image of the previous frame and the original image of the current frame using a preset tracking algorithm to obtain a tracking result; and determining the height of the camera relative to the ground based on the tracking result and the vehicle posture output by the inertial navigation device.

[0086] When calculating the height of the camera relative to the ground, embodiments of the present application can first utilize a preset tracking algorithm to track and match the original images of the current frame and the previous frame. The preset tracking algorithm can, for example, employ optical flow tracking, thereby obtaining matching feature points in the two image frames, i.e., successfully tracked feature points. Of course, the specific tracking algorithm employed can be flexibly selected by those skilled in the art based on practical needs, and is not specifically limited herein.

[0087] In some embodiments of the present application, determining the height of the camera relative to the ground based on the tracking results and the vehicle posture output by the inertial navigation device includes: obtaining external parameters of the camera relative to the vehicle; based on the external parameters of the camera relative to the vehicle, using the vehicle posture at the corresponding moment of the image data of the current frame and the vehicle posture at the corresponding moment of the image data of the previous frame to determine the relative posture of the current frame camera and the previous frame camera; determining the height of the camera relative to the ground based on the tracking results and the relative posture of the current frame camera and the previous frame camera.

[0088] The extrinsic parameters of the camera relative to the vehicle characterize the relative transformation relationship between the camera and the vehicle. They are less affected by the ups and downs of the vehicle and can therefore be calibrated in advance. Based on the relative transformation relationship between the camera and the vehicle and the vehicle pose at the corresponding moments of the two frames of image data, the relative pose between the current frame camera and the previous frame camera can be calculated. For example, based on the relative transformation relationship between the camera and the vehicle, the vehicle pose at the corresponding moments of the two frames of image data can be converted into the camera pose at the corresponding moments of the two frames of image data. Finally, the relative transformation between the camera poses at the corresponding moments of the two frames of image data is calculated to obtain the relative pose between the two frames of camera.

[0089] In some embodiments of the present application, the tracking results include matching feature points, and determining the height of the camera relative to the ground based on the tracking results and the relative position of the current frame camera and the previous frame camera includes: based on the relative position of the current frame camera and the previous frame camera, using a triangulation algorithm to triangulate the matching feature points to obtain three-dimensional feature points in the coordinate system of the current frame camera; and performing plane fitting on the three-dimensional feature points in the coordinate system of the current frame camera to obtain the height of the camera relative to the ground.

[0090] In multi-view geometry, a camera observes the same spatial point from two locations. The coordinates of the three-dimensional spatial point can be obtained by combining the two camera poses and the image observation point coordinates. This process is called triangulation. Based on this, after obtaining the relative pose of the current frame camera and the previous frame camera, the embodiment of the present application can triangulate the matching feature points in the two frames based on the relative pose of the two frames, thereby obtaining the three-dimensional spatial coordinates of these matching feature points in the coordinate system of the current frame camera.

[0091] Then, plane fitting is performed on these three-dimensional feature points in the coordinate system of the current frame camera, so that the fitted plane equation can be obtained. Specifically, the general expression of the plane equation is:

[0092] Ax+By+Cz+D=0(C≠0);

[0093] By inputting the coordinate values ​​of multiple three-dimensional feature points in the coordinate system of the current frame camera, the parameter values ​​of each parameter of the plane equation can be obtained, where the parameter D is the height of the camera relative to the ground.

[0094] In some embodiments of the present application, the tracking result includes matched feature points. After tracking the original image of the previous frame and the original image of the current frame using a preset tracking algorithm to obtain the tracking result, the method further includes: determining whether the number of the matched feature points is less than a preset number threshold; if the number of the matched feature points is less than the preset number threshold, using the segmented image of the current frame to extract features from the original image of the current frame, so that the sum of the extracted feature points and the matched feature points is not less than the preset number threshold.

[0095] Based on the above embodiments, it can be seen that the number of matching feature points in two frames of images will affect the fitting effect of the plane equation. If the number is too small, the fitting is inaccurate, which in turn affects the calculation accuracy of the camera's height relative to the ground.

[0096] Based on this, in order to ensure that there are sufficient feature points for each tracking match, after obtaining the feature points that match between the current frame and the previous frame, the embodiment of the present application can first determine whether the number of matched feature points is less than the preset number threshold. If it is less than the preset number threshold, it means that when the next tracking match is performed based on the matched feature points in the current frame, the number of feature points that can be used for matching is too small to meet the requirements of matching accuracy and plane fitting accuracy. Therefore, the embodiment of the present application can use the ground part in the segmented image of the current frame as a mask to extract features from the original image of the current frame, ensuring that the sum of the newly extracted feature points and the number of matched feature points is not less than the preset number threshold, thereby providing support for the next tracking match. In addition, the size of the preset number threshold can be flexibly set according to actual needs and is not specifically limited here.

[0097] For example, assuming that there are 100 feature points that match the original image of the current frame and the original image of the previous frame, and the preset threshold is 200, then the ground part in the segmented image of the current frame can be used as a mask to extract at least 100 more feature points from the original image of the current frame to ensure that the final number of feature points is not less than 200.

[0098] To facilitate understanding of the above embodiments of the present application, the various coordinate systems involved in the present application are uniformly defined, including:

[0099] 1) Vehicle coordinate system: defined as front upper left - xyz;

[0100] 2) Inertial navigation equipment (IMU+RTK) coordinate system: defined as front upper left - xyz:

[0101] 3) Camera coordinate system: defined as bottom-right-front-xyz;

[0102] 4) Ground coordinate system: On the ground directly below the camera's optical center, defined as front-upper-left-xyz.

[0103] Of course, the above definition of the coordinate system is only an exemplary description, and those skilled in the art can also make adaptive adjustments according to actual needs, and no specific limitation is made here.

[0104] In summary, the inverse perspective transformation of an image in the embodiment of the present application achieves at least the following technical effects:

[0105] 1) Calculating the camera's pitch and roll angles relative to the ground based on inertial navigation data is less affected by the external environment than using vanishing point calculations. This method also has a high output frequency and requires less computation for Kalman filter fusion, thus meeting real-time requirements.

[0106] 2) Only the original images between two frames of cameras are used for optical flow tracking, which reduces the computational complexity and meets the real-time requirements. In addition, feature extraction with masks ensures a sufficient number of feature points and improves the plane fitting effect.

[0107] 3) To address the issue of the camera's height relative to the ground changing in real time due to the vehicle's undulations during driving, plane fitting using the 3D spatial points in the camera's coordinate system in the current frame can accurately estimate the camera's height relative to the ground. However, methods based on vanishing points cannot perceive height changes.

[0108] 4) By calculating the pitch and roll angles of the camera relative to the ground and the height of the camera relative to the ground in real time, the extrinsic parameters of the camera relative to the ground can be corrected in real time, thereby achieving adaptive inverse perspective transformation and effectively improving the accuracy of inverse perspective transformation.

[0109] The embodiment of the present application also provides an image inverse perspective transformation device 200, such as Figure 2 , a schematic structural diagram of an image inverse perspective transformation apparatus according to an embodiment of the present application is provided. The apparatus 200 includes at least: a first acquisition unit 210, a second acquisition unit 220, a first determination unit 230, a second determination unit 240, and an inverse perspective transformation unit 250, wherein:

[0110] A first acquisition unit 210 is configured to acquire image data of a current frame and image data of a previous frame, wherein the image data includes an original image and a semantically segmented image corresponding to the original image;

[0111] A second acquisition unit 220 is configured to acquire inertial navigation data at a corresponding moment to the image data of the current frame and the image data of the previous frame, wherein the inertial navigation data includes an angle of the inertial navigation device relative to the ground plane and a vehicle posture output by the inertial navigation device;

[0112] A first determining unit 230 is configured to determine an angle of the inertial navigation device relative to the ground based on the image data of the current frame and the inertial navigation data at a corresponding moment of the image data of the previous frame;

[0113] a second determining unit 240, configured to determine a height of the camera relative to the ground based on the image data of the current frame and the image data of the previous frame, and the vehicle posture output by the inertial navigation device;

[0114] The inverse perspective transformation unit 250 is used to correct the external parameters of the camera relative to the ground according to the angle of the inertial navigation device relative to the ground and the height of the camera relative to the ground, and use the corrected external parameters of the camera relative to the ground to perform an inverse perspective transformation on the semantic segmentation image of the current frame to obtain an image after inverse perspective transformation.

[0115] In some embodiments of the present application, the device also includes: a third acquisition unit, used to obtain IMU measurement data of the inertial navigation device and the corresponding vehicle posture, the IMU measurement data including acceleration and angular velocity; a fusion unit, used to fuse the IMU measurement data using a Kalman filter to obtain the angle of the inertial navigation device relative to the ground plane; and an output unit, used to use the angle of the inertial navigation device relative to the ground plane and the corresponding vehicle posture as the inertial navigation data.

[0116] In some embodiments of the present application, the second acquisition unit 220 is specifically used to: acquire inertial navigation data within a preset time length; perform time synchronization processing on the inertial navigation data within the preset time length based on the image data of the current frame and the image data of the previous frame, and obtain the inertial navigation data at the corresponding moments of the image data of the current frame and the image data of the previous frame.

[0117] In some embodiments of the present application, the first determination unit 230 is specifically used to: determine the relative translation between the current frame and the previous frame based on the vehicle posture at the time corresponding to the image data of the current frame and the vehicle posture at the time corresponding to the image data of the previous frame; determine the angle between the ground plane and the ground based on the relative translation between the current frame and the previous frame; determine the angle of the inertial navigation device relative to the ground at the time corresponding to the image data of the current frame based on the angle of the inertial navigation device relative to the ground plane and the angle between the ground plane and the ground.

[0118] In some embodiments of the present application, the second determination unit 240 is specifically used to: track the original image of the previous frame and the original image of the current frame using a preset tracking algorithm to obtain a tracking result; and determine the height of the camera relative to the ground based on the tracking result and the vehicle posture output by the inertial navigation device.

[0119] In some embodiments of the present application, the second determination unit 240 is specifically used to: obtain the external parameters of the camera relative to the vehicle; based on the external parameters of the camera relative to the vehicle, use the vehicle posture at the corresponding moment of the image data of the current frame and the vehicle posture at the corresponding moment of the image data of the previous frame to determine the relative posture of the current frame camera and the previous frame camera; determine the height of the camera relative to the ground according to the tracking result and the relative posture of the current frame camera and the previous frame camera.

[0120] In some embodiments of the present application, the tracking results include matching feature points, and the second determination unit 240 is specifically used to: based on the relative posture of the current frame camera and the previous frame camera, use a triangulation algorithm to triangulate the matching feature points to obtain three-dimensional feature points in the coordinate system of the current frame camera; perform plane fitting on the three-dimensional feature points in the coordinate system of the current frame camera to obtain the height of the camera relative to the ground.

[0121] In some embodiments of the present application, the tracking result includes matched feature points, and the device also includes: a third determination unit, used to determine whether the number of the matched feature points is less than a preset number threshold; a feature extraction unit, used to use the segmented image of the current frame to extract features from the original image of the current frame when the number of the matched feature points is less than the preset number threshold, so that the sum of the extracted feature points and the number of the matched feature points is not less than the preset number threshold.

[0122] It can be understood that the above-mentioned inverse perspective transformation device of the image can implement the various steps of the inverse perspective transformation method of the image provided in the aforementioned embodiment. The relevant explanations about the inverse perspective transformation method of the image are applicable to the inverse perspective transformation device of the image and will not be repeated here.

[0123] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 3 At the hardware level, the electronic device includes a processor and, optionally, an internal bus, a network interface, and memory. The memory may include internal memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for its services.

[0124] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0125] The memory is used to store programs. Specifically, the program may include program code, which includes computer operating instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0126] The processor reads the corresponding computer program from the non-volatile memory into the internal memory and then runs it, forming an inverse perspective transformation device for the image at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0127] Acquire image data of a current frame and image data of a previous frame, wherein the image data includes an original image and a semantic segmentation image corresponding to the original image;

[0128] Acquire inertial navigation data at corresponding moments of the image data of the current frame and the image data of the previous frame, the inertial navigation data including an angle of the inertial navigation device relative to the ground plane and a vehicle posture output by the inertial navigation device;

[0129] determining an angle of the inertial navigation device relative to the ground based on the image data of the current frame and the inertial navigation data at a corresponding moment of the image data of the previous frame;

[0130] Determining the height of the camera relative to the ground based on the image data of the current frame and the image data of the previous frame, and the vehicle posture output by the inertial navigation device;

[0131] The extrinsic parameters of the camera relative to the ground are corrected according to the angle of the inertial navigation device relative to the ground and the height of the camera relative to the ground, and the semantic segmentation image of the current frame is inversely transformed using the corrected extrinsic parameters of the camera relative to the ground to obtain an image after inverse perspective transformation.

[0132] The above application Figure 1The method performed by the image inverse perspective transformation device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in the processor or by software instructions. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of this application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0133] The electronic device may also perform Figure 1 The method of performing the inverse perspective transformation of the image in the apparatus and realizing the inverse perspective transformation of the image in the apparatus Figure 1 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0134] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, can enable the electronic device to execute Figure 1 The method performed by the inverse perspective transformation device of an image in the embodiment shown is specifically used to perform:

[0135] Acquire image data of a current frame and image data of a previous frame, wherein the image data includes an original image and a semantic segmentation image corresponding to the original image;

[0136] Acquire inertial navigation data at corresponding moments of the image data of the current frame and the image data of the previous frame, the inertial navigation data including an angle of the inertial navigation device relative to the ground plane and a vehicle posture output by the inertial navigation device;

[0137] determining an angle of the inertial navigation device relative to the ground based on the image data of the current frame and the inertial navigation data at a corresponding moment of the image data of the previous frame;

[0138] Determining the height of the camera relative to the ground based on the image data of the current frame and the image data of the previous frame, and the vehicle posture output by the inertial navigation device;

[0139] The extrinsic parameters of the camera relative to the ground are corrected according to the angle of the inertial navigation device relative to the ground and the height of the camera relative to the ground, and the semantic segmentation image of the current frame is inversely transformed using the corrected extrinsic parameters of the camera relative to the ground to obtain an image after inverse perspective transformation.

[0140] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0141] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0142] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0143] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0144] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0145] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0146] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0147] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0148] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for inverse perspective transformation of an image, wherein: The method comprises: Acquire image data of a current frame and image data of a previous frame, wherein the image data includes an original image and a semantic segmentation image corresponding to the original image; Acquire inertial navigation data at corresponding moments of the image data of the current frame and the image data of the previous frame, the inertial navigation data including an angle of the inertial navigation device relative to the ground plane and a vehicle posture output by the inertial navigation device; determining an angle of the inertial navigation device relative to the ground based on the image data of the current frame and the inertial navigation data at a corresponding moment of the image data of the previous frame; Determining the height of the camera relative to the ground based on the image data of the current frame and the image data of the previous frame, and the vehicle posture output by the inertial navigation device; Correcting the extrinsic parameters of the camera relative to the ground according to the angle of the inertial navigation device relative to the ground and the height of the camera relative to the ground, and performing an inverse perspective transformation on the semantic segmentation image of the current frame using the corrected extrinsic parameters of the camera relative to the ground to obtain an image after inverse perspective transformation; Determining the angle of the inertial navigation device relative to the ground based on the image data of the current frame and the inertial navigation data at corresponding moments of the image data of the previous frame includes: Determining a relative translation between the current frame and the previous frame based on the vehicle posture at the time corresponding to the image data of the current frame and the vehicle posture at the time corresponding to the image data of the previous frame; determining an angle between the ground plane and the ground based on the relative translation between the current frame and the previous frame; The angle of the inertial navigation device relative to the ground at the moment corresponding to the image data of the current frame is determined according to the angle of the inertial navigation device relative to the ground plane at the moment corresponding to the image data of the current frame and the angle between the ground plane and the ground.

2. The method according to claim 1, wherein: Before acquiring the image data of the current frame and the inertial navigation data at corresponding moments of the image data of the previous frame, the method further includes: Obtaining IMU measurement data from an inertial navigation device and the corresponding vehicle posture, wherein the IMU measurement data includes acceleration and angular velocity; fusing the IMU measurement data using a Kalman filter to obtain an angle of the inertial navigation device relative to the ground plane; The angle of the inertial navigation device relative to the ground plane and the corresponding vehicle posture are used as the inertial navigation data.

3. The method according to claim 1, wherein: Determining the height of the camera relative to the ground based on the image data of the current frame and the image data of the previous frame, and the vehicle posture output by the inertial navigation device includes: Track the original image of the previous frame and the original image of the current frame using a preset tracking algorithm to obtain a tracking result; The height of the camera relative to the ground is determined according to the tracking result and the vehicle posture output by the inertial navigation device.

4. The method according to claim 3, wherein: Determining the height of the camera relative to the ground according to the tracking result and the vehicle posture output by the inertial navigation device includes: Get the external parameters of the camera relative to the vehicle; Based on the external parameters of the camera relative to the vehicle, the relative posture of the camera in the current frame and the camera in the previous frame are determined using the vehicle posture at the time corresponding to the image data of the current frame and the vehicle posture at the time corresponding to the image data of the previous frame; The height of the camera relative to the ground is determined according to the tracking result and the relative position of the camera in the current frame and the camera in the previous frame.

5. The method according to claim 4, wherein: The tracking result includes matched feature points, and determining the height of the camera relative to the ground based on the tracking result and the relative position of the current frame camera and the previous frame camera includes: Based on the relative pose of the current frame camera and the previous frame camera, triangulate the matched feature points using a triangulation algorithm to obtain three-dimensional feature points in the coordinate system of the current frame camera; Perform plane fitting on the three-dimensional feature points in the coordinate system of the current frame camera to obtain the height of the camera relative to the ground.

6. The method of claim 3, wherein: The tracking result includes matched feature points. After the original image of the previous frame and the original image of the current frame are tracked using a preset tracking algorithm to obtain the tracking result, the method further includes: Determining whether the number of the matched feature points is less than a preset number threshold; When the number of the matched feature points is less than a preset threshold, feature extraction is performed on the original image of the current frame using the segmented image of the current frame so that the sum of the number of the extracted feature points and the matched feature points is not less than the preset threshold.

7. An image inverse perspective transformation device, wherein: The device comprises: A first acquisition unit is configured to acquire image data of a current frame and image data of a previous frame, wherein the image data includes an original image and a semantic segmentation image corresponding to the original image; a second acquisition unit, configured to acquire inertial navigation data at corresponding moments of the image data of the current frame and the image data of the previous frame, the inertial navigation data including an angle of the inertial navigation device relative to the ground plane and a vehicle posture output by the inertial navigation device; a first determining unit, configured to determine an angle of the inertial navigation device relative to the ground based on the image data of the current frame and the inertial navigation data at a corresponding moment of the image data of the previous frame; a second determining unit, configured to determine a height of the camera relative to the ground based on the image data of the current frame and the image data of the previous frame, and a vehicle posture output by the inertial navigation device; an inverse perspective transformation unit, configured to correct an extrinsic parameter of the camera relative to the ground according to an angle of the inertial navigation device relative to the ground and a height of the camera relative to the ground, and perform an inverse perspective transformation on the semantic segmentation image of the current frame using the corrected extrinsic parameter of the camera relative to the ground to obtain an image after inverse perspective transformation; The first determining unit is specifically configured to: Determining a relative translation between the current frame and the previous frame based on the vehicle posture at the time corresponding to the image data of the current frame and the vehicle posture at the time corresponding to the image data of the previous frame; determining an angle between the ground plane and the ground based on the relative translation between the current frame and the previous frame; The angle of the inertial navigation device relative to the ground at the moment corresponding to the image data of the current frame is determined according to the angle of the inertial navigation device relative to the ground plane at the moment corresponding to the image data of the current frame and the angle between the ground plane and the ground.

8. An electronic device comprising: processor; as well as A memory arranged to store computer executable instructions, which when executed cause the processor to perform the method of any one of claims 1 to 6.

9. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, causes the electronic device to execute the method according to any one of claims 1 to 6.

Images