Methods and devices for providing vehicle visualization, as well as vehicles

Abstract

The present invention provides a method for providing vehicle visualization, the method comprising the following steps: receiving current camera extrinsic parameter values ​​for each of a plurality of vehicle-mounted cameras of a vehicle; receiving vehicle suspension data related to the vehicle's wheel suspension; and visualizing the vehicle using a predefined vehicle model, wherein the vehicle is visualized on a ground surface; wherein, based on the current camera extrinsic parameter values ​​of the vehicle-mounted cameras and the suspension data, the ground surface is modeled as being in contact with the vehicle's wheels.

Description

Technical Field

[0001] This invention relates to a method for providing vehicle visualization, an apparatus for providing vehicle visualization, and a vehicle. Background Technology

[0002] Modern vehicles can be equipped with surround-view systems to provide visualization of the vehicle's surroundings based on camera data from onboard cameras. To provide a more realistic appearance, the vehicle itself can be visualized. Vehicle visualization can include animations of various features such as wheels, steering wheel, headlights and taillights, doors, and the hood.

[0003] Limitations on animation may stem from predefined surface structures, i.e., meshes, which relate to the ground beneath the vehicle. For example, if the ground is modeled as a horizontal plane, it cannot reflect the actual structure of the ground surface in real time. Summary of the Invention

[0004] In view of the above, the purpose of this invention is to provide a more realistic vehicle visualization.

[0005] According to the present invention, a method for providing vehicle visualization according to claim 1 and an apparatus for providing vehicle visualization according to claim 11 are provided. Furthermore, the present invention also provides a vehicle according to claim 13.

[0006] The preferred features of the present invention are set forth in the dependent claims.

[0007] Therefore, according to a first aspect, the present invention provides a method for providing vehicle visualization, wherein, for each of a plurality of vehicle-mounted cameras, current camera extrinsic parameter values ​​of the vehicle-mounted camera are received. Vehicle suspension data related to the vehicle's wheel suspension is received. The vehicle is visualized using a predefined vehicle model. The vehicle is visualized on the ground. The ground surface is modeled based on the current extrinsic camera parameter values ​​of the vehicle-mounted cameras and based on the suspension data, such that the ground surface is in contact with the vehicle's wheels.

[0008] According to a second aspect, the present invention provides an apparatus for providing vehicle visualization, the apparatus comprising an interface and a computing unit. The interface is configured to receive current camera extrinsic parameter values ​​for each of a plurality of vehicle-mounted cameras, and to receive vehicle suspension data related to the vehicle's wheel suspension. The computing unit calculates the vehicle visualization using a predefined vehicle model, wherein the vehicle is visualized on a ground surface. The computing unit models the ground surface based on the current camera extrinsic parameter values ​​of the vehicle-mounted cameras and based on the suspension data, such that the ground surface is in contact with the vehicle's wheels.

[0009] According to a third viewpoint, the present invention provides a vehicle comprising a plurality of vehicle-mounted cameras and a device for providing vehicle visualization according to the present invention.

[0010] This invention provides a realistic simulation of vehicles, including vertical movement of vehicles on non-flat surfaces. Correspondingly, this invention provides better visualization of vehicles, constituting an improved human-machine interface.

[0011] With more realistic vehicle visualization, drivers can identify obstacles and uneven road structures and maneuver the vehicle accordingly. Furthermore, the visualization model can serve as input for driver assistance systems that can control vehicle functions based on visualization. In this way, driver assistance systems can, for example, automatically or semi-automatically accelerate, decelerate, or steer the vehicle.

[0012] According to the present invention, the camera's external parameters can be provided in matrix form as follows:

[0013] |RT|

[0014] |0 1|

[0015] Here, R refers to a 3x3 rotation matrix, and T (3x1) refers to a translation vector. Camera extrinsic parameters refer to the coordinate system transformation from 3D world coordinates to 3D camera coordinates. The camera extrinsic parameters define the center position and orientation of the vehicle-mounted camera in world coordinates. The translation vector T represents the origin of the world coordinate system in the form of camera coordinates.

[0016] According to the implementation method of the vehicle visualization method, for each vehicle-mounted camera, the difference between the current external parameter value of the vehicle-mounted camera and the predefined initial value of the external parameter of the vehicle-mounted camera is calculated. Based on the calculated difference between the current external parameter value of the vehicle-mounted camera and the predefined initial value of the external parameter of the vehicle-mounted camera, the current three-dimensional state of the vehicle is calculated. Based on the calculated current three-dimensional state of the vehicle, the ground surface is modeled.

[0017] According to one embodiment of the vehicle visualization method, the three-dimensional state of the vehicle is calculated using affine transformations in homogeneous coordinates. The affine transformations include rotations related to vehicle rotation and translations related to vehicle translation.

[0018] According to one embodiment of the vehicle visualization method, the vehicle's three-dimensional pose is calculated by applying multivariate interpolation to the current external parameter values ​​of the camera.

[0019] According to one embodiment of the vehicle visualization method, multivariate interpolation includes at least one of bilinear interpolation and bicubic interpolation.

[0020] According to one embodiment of the vehicle visualization method, the displacement of each wheel is calculated based on vehicle suspension data. The ground surface is then modeled based on the calculated vehicle wheel displacements.

[0021] According to one embodiment of the vehicle visualization method, for each wheel, the vehicle suspension data includes information about the current suspension height. For each wheel, displacement is calculated based on the difference between the current suspension height and a predefined initial suspension height.

[0022] According to one embodiment of the vehicle visualization method, the position of each wheel of the vehicle is calculated based on the vehicle's three-dimensional pose and / or the displacement calculated from the vehicle's wheels.

[0023] According to one embodiment of the vehicle visualization method, the ground surface is modeled using multivariate interpolation with calculated wheel positions.

[0024] According to one embodiment of the vehicle visualization method, the position multivariate interpolation calculated using the wheels includes at least one of bilinear interpolation and bicubic interpolation.

[0025] According to one embodiment of the vehicle visualization method, the visualization may include a visualization in the form of a bowl-shaped view.

[0026] Furthermore, according to one embodiment, the device for providing vehicle visualization also includes a display for outputting the calculated vehicle visualization.

[0027] In addition, according to one embodiment, the vehicle also includes at least one sensor configured to measure the current camera external parameters and suspension data of the vehicle-mounted camera, and to provide the measured current camera external parameters and suspension data of the vehicle-mounted camera to the device.

[0028] According to one embodiment, the vehicle includes at least four onboard cameras disposed around the vehicle. Attached Figure Description

[0029] To provide a more complete understanding of the present invention and its advantages, exemplary embodiments of the invention will be explained in more detail in the following description with reference to the accompanying drawings, in which the same reference numerals refer to the same parts, wherein:

[0030] Figure 1 This figure illustrates, in the form of a block diagram, the vehicle visualization device provided according to an embodiment of the present invention;

[0031] Figure 2 This diagram shows the rear of the vehicle, illustrating the vehicle's suspension height;

[0032] Figure 3 This image shows a top-down view of the vehicle;

[0033] Figure 4 This image shows a side view of the vehicle;

[0034] Figure 5 This figure shows a schematic diagram of a vehicle according to an embodiment of the present invention; and

[0035] Figure 6 This figure illustrates, in the form of a flowchart, a method for providing a panoramic view image according to an embodiment of the present invention. Detailed Implementation

[0036] The accompanying drawings are provided to better understand the invention and form part of this description. The drawings illustrate specific embodiments of the invention and, together with the related description, explain the principles of the invention. Other embodiments of the invention and many associated advantages become more apparent from the following detailed description.

[0037] Figure 1 The diagram shows a block illustration of a device 1 providing vehicle visualization. Device 1 includes an interface 11 that connects to a plurality of vehicle-mounted cameras 22 to 2n of the vehicle via cable or wireless connection. Here, n can be any integer greater than 2. Preferably, at least four vehicle-mounted cameras 22 to 2n are positioned around the vehicle. The vehicle-mounted cameras 22 to 2n may include a front-facing camera, a rear-facing camera, and at least one side-mounted camera on each side of the vehicle. The vehicle-mounted cameras 22 to 2n are positioned such that they provide a 360-degree field of view. Adjacent vehicle-mounted cameras 22 to 2n may have partially overlapping detection areas.

[0038] In addition, interface 11 is also connected to sensor 21 for measuring the external parameters of the vehicle-mounted cameras 22 to 2n. Sensor 21 may include at least one of a yaw rate sensor, an acceleration sensor, a position sensor, and similar sensors. Sensor 21 may provide the current camera external parameters in the form of a 4x4 matrix.

[0039] |RT|

[0040] |0 1| Formula (1)

[0041] Here, R refers to a 3x3 rotation matrix, and T refers to a 3x1 translation vector. The 4x4 matrix corresponds to the vehicle state.

[0042] The camera's external parameters can also be given in the form of (x, y, z, Rx, Ry, Rz), where x, y, and z correspond to the T vector, and Rx, Ry, and Rz correspond to the R matrix using the following formulas:

[0043] R = Rz(a)·Ry(b)·Rx(c).

[0044] In addition, sensor 21 measures vehicle suspension data related to the vehicle's wheel suspension and provides the measured suspension data to interface 11. The suspension data may include the suspension height of each wheel of the vehicle.

[0045] The vehicle-mounted cameras 22 to 2n provide corresponding camera images to interface 11.

[0046] Interface 11 provides camera images, suspension data, and camera extrinsic parameters of vehicle-mounted cameras 22 to 2n to the computing unit 12 of device 1. The computing unit 12 may include at least one of a processor, microprocessor, integrated circuit, application-specific integrated circuit (ASIC), or similar component. Furthermore, the computing unit 12 may also include at least one memory for storing the received camera extrinsic parameters, suspension parameters, and camera images, as well as for storing program instructions.

[0047] The computing unit 12 uses a predefined 3D model of the vehicle to calculate vehicle visualization. The predefined vehicle model may include features such as wheels, steering wheel, headlights and taillights, doors, and hood. The computing unit 12 can visualize the vehicle model on a ground surface.

[0048] The calculation unit 12 stores the initial camera external parameters and the initial suspension data. The algorithm executed by the calculation unit 12 is based on an estimate of the current vehicle body state using the difference between the current camera external parameters and the initial camera external parameters, as well as the difference between the current wheel suspension height and the initial wheel suspension height.

[0049] The calculation unit 12 generates the ground surface by making the vehicle's wheels contact the ground surface. The ground surface is calculated based on the current external parameter values ​​of the onboard camera and suspension data.

[0050] Calculation unit 12 calculates the difference between the current extrinsic parameter value of each of the 22 to 2n vehicle-mounted cameras and the predefined initial values ​​of the extrinsic parameters. Furthermore, calculation unit 12 also calculates the current three-dimensional pose of the vehicle based on the calculated difference between the current extrinsic parameter value and the predefined initial values ​​of the extrinsic parameters of the 22 to 2n vehicle-mounted cameras. The three-dimensional pose of the vehicle is calculated using affine transformations in homogeneous coordinates.

[0051] Affine transformations include rotations related to vehicle rotation and translations related to vehicle translation. The computation unit 12 can apply multivariate interpolation to the current camera extrinsic parameter values ​​to calculate the vehicle's three-dimensional pose.

[0052] The calculation unit 12 can perform multivariate interpolation as bilinear interpolation.

[0053] Specifically, the computing unit 12 can calculate the difference between the current camera extrinsic parameters and the initial camera extrinsic parameters to estimate the current three-dimensional pose of the vehicle. Using homogeneous coordinates, the vehicle pose is given by a 4x4 matrix of Equation (1) through a combination of affine transformations corresponding to rotation and translation.

[0054] Rz(a) is a 3x3 rotation matrix that rotates the vehicle by an angle 'a' along the vertical Z-axis. Rz(a) does not reflect any changes in vehicle attitude caused by uneven ground. An identity matrix can be used accordingly.

[0055] Ry(b) is a 3x3 rotation matrix rotated along the horizontal y-axis by the angle b from the camera's extrinsic parameters. The angle b can be calculated by bilinear interpolation of the extrinsic parameters of the vehicle camera corresponding to the vehicle camera rotated around the y-axis.

[0056] Rx(c) is a 3x3 rotation matrix that rotates along the x-axis to the angle c of the camera's extrinsic parameters. The angle c is calculated by bilinear interpolation of the extrinsic parameters of the vehicle camera corresponding to the vehicle camera rotating around the x-axis.

[0057] T is a 3x1 translation matrix (or vector), where the x and y displacements can be set to be the same, as they do not reflect any changes in vehicle attitude caused by uneven ground surfaces. The z-coordinate is calculated by applying bilinear interpolation to the extrinsic parameters of the vehicle camera corresponding to the vehicle camera rotating about the z-axis.

[0058] Furthermore, the calculation unit 12 can calculate the displacement for each wheel based on vehicle suspension data. The ground surface can be modeled based on the calculated wheel displacements of the vehicle. Here, the vehicle suspension data includes information on the current suspension height of each wheel. For each wheel, the displacement is calculated based on the difference between the current suspension height and a predefined initial suspension height. The position of each wheel of the vehicle is calculated based on the vehicle's three-dimensional attitude and the displacements calculated based on the vehicle wheels. The displacements can be calculated in the x, y, and z directions, while the angle of the wheel top is calculated using suspension displacement, which is the difference between the current and initial wheel suspension positions and directions. The bottom of each wheel of the predefined vehicle model is placed at a corresponding, possibly different, position and angle.

[0059] The surface is calculated to match the height of the wheels and the vehicle. The height of the ground plane mesh beneath the vehicle is estimated using bilinear interpolation of the heights of the four bottoms of the wheels.

[0060] Generally, the surface can be modeled using multivariate interpolation based on the calculated wheel positions.

[0061] The ground plane grid can be set at equal intervals under the vehicle.

[0062] The surface height of the external planar mesh is modeled to be consistent with zero height. Accordingly, simple interpolation can be applied.

[0063] In addition, the computing unit 12 can apply a stable filtering mechanism, such as Kalman filtering, to each calculated height, thereby making the visualization smoother and more stable.

[0064] Furthermore, the computing unit 12 can also generate a panoramic surround-view image of the vehicle. The vehicle can be visualized within the bowl-shaped structure. The vehicle's ground surface is generated according to the steps described above. The ground surface transforms into a wall-like portion at a greater distance. The computing unit 12 can project camera images received by the onboard cameras 22 to 2n onto the bowl-shaped structure. This generates a panoramic surround-view image of the vehicle, which can be displayed to the vehicle driver on a display screen 13 of a device such as the vehicle's dashboard.

[0065] Figure 2 This diagram shows the rear view of vehicle 3. It also shows the suspension height *s* of wheel 33 of vehicle 3. Additionally, camera 22 is located at the rear of vehicle 3 at a height *H*. A coordinate system is defined, where *x* represents the longitudinal axis, *y* represents the horizontal axis, and *z* represents the vertical axis.

[0066] Figure 3 The diagram shown is a top view of a vehicle 3 equipped with four vehicle-mounted cameras 22 to 25. The first vehicle-mounted camera 22 is located at the front of the vehicle 3, the second vehicle-mounted camera 23 is located on the first side of the vehicle 3, the third vehicle-mounted camera 24 is located on the second side of the vehicle 3, and the fourth vehicle-mounted camera 25 is located at the rear of the vehicle 3.

[0067] Figure 4 The image shown is a side view of vehicle 3.

[0068] Figure 5 The diagram shown is a block illustration of vehicle 3. The vehicle includes multiple onboard cameras 22 to 2n, particularly front-facing cameras, rear-facing cameras, and / or side-mounted cameras. Additionally, vehicle 3 includes sensors 21 for determining the external parameters of the onboard cameras 22 to 2n.

[0069] Sensor 21 provides external parameters of the camera to device 1 to provide a panoramic surround view image. Additionally, vehicle-mounted cameras 22 to 2n also provide corresponding camera images to device 1. Device 1 is configured according to one of the above embodiments. As described above, device 1 provides visualization of vehicle 3 using a predefined model of vehicle 3, wherein vehicle 3 is visualized on a ground surface. The ground surface contacts the wheels of vehicle 3.

[0070] Device 1 can provide the generated visual information of vehicle 3 to the display 31 of vehicle 3. Correspondingly, the visual information of vehicle 3 can be displayed to the driver of vehicle 3.

[0071] Furthermore, the device 1 can also provide the generated visual information of the vehicle 3 to the driver assistance system 32, which can be configured to control at least one driving function of the vehicle 3. For example, the driver assistance system 32 can accelerate, decelerate, or steer the vehicle 3 based on the visual information of the vehicle 3.

[0072] Figure 6 This is a flowchart illustrating the method for providing panoramic surround view images.

[0073] In the first method step S1, the current external parameter value of the camera is received, which corresponds to a plurality of vehicle-mounted cameras 22 to 2n of the vehicle 3, preferably corresponding to at least four vehicle-mounted cameras 22 to 2n.

[0074] In the second method step S2, suspension data related to the suspension of the wheel 33 of the vehicle 3 is received.

[0075] In the third method step S3, an affine transformation is used to calculate a three-dimensional pose of vehicle 3 in homogeneous coordinates. These affine transformations include rotations related to the rotation of vehicle 3 and translations related to the translation of vehicle 3. The three-dimensional pose of vehicle 3 is calculated by applying multivariate interpolation to the current camera extrinsic parameter values.

[0076] In the fourth method step S4, the displacement of each wheel is calculated based on vehicle suspension data. The suspension data may include current suspension height information for each wheel 33. The displacement may be calculated based on the difference between the current suspension height and a predefined initial suspension height.

[0077] In the fifth method step S5, the visualization of vehicle 3 is calculated, wherein vehicle 3 is visualized on a ground surface. The ground surface is modeled by making the wheels of vehicle 3 contact the ground. The ground surface is generated based on the calculated wheel displacements of vehicle 3.

[0078] The position of each wheel of vehicle 3 can be calculated based on the three-dimensional attitude of the vehicle and the displacement calculated based on the wheels of vehicle 3.

[0079] The calculated wheel positions can be used to model the ground surface using multivariate interpolation.

[0080] In the sixth method step S6, a panoramic surround view image can be generated using visualizations of vehicle 3 and camera images provided by onboard cameras 22 to 2n. To generate the panoramic surround view image, visualization information of vehicle 3 can be generated, particularly including a virtual bowl structure of the modeled ground surface. The panoramic surround view image can be generated by projecting camera images onto the virtual bowl structure surrounding the model of vehicle 3. The panoramic surround view image can be displayed to the driver of vehicle 3. Alternatively or as an additional measure, the panoramic surround view image can be used in a driver assistance system 32 for controlling the driving functions of vehicle 3.

[0081] Figure Labels

[0082] 1 device

[0083] 3 vehicles

[0084] 11 Interface

[0085] 12 Calculation Units

[0086] 13 Monitors

[0087] 21 Sensors

[0088] 22 to 2n vehicle-mounted cameras

[0089] 31 Monitors

[0090] 32 Driver Assistance Systems

[0091] 33 wheels

[0092] H Camera height

[0093] s Suspension height

[0094] S1 to S6 Method Steps

[0095] x, y, z coordinates

Claims

1. A method for providing visualization of a vehicle (3), the method comprising the following steps: For each of the multiple vehicle-mounted cameras (22 to 2n) in the vehicle (3), receive the current camera external parameter value of the vehicle-mounted camera (22 to 2n); Receive vehicle suspension data relating to the suspension of the wheels (33) of the vehicle (3); and The vehicle (3) is visualized using a predefined model of the vehicle (3), wherein, The vehicle (3) is visualized on the ground surface; The ground surface modeling method is based on the current external parameter values ​​of the vehicle-mounted cameras (22 to 2n) and the suspension data, so that the wheels (33) of the vehicle (3) are in contact with the ground surface. The displacement of each wheel (33) is calculated based on vehicle suspension data, and each wheel (33) is visualized with its own displacement. The ground surface is modeled and visualized based on the calculated displacement of the wheels (33) of the vehicle (3).

2. The method according to claim 1, further comprising the following steps: For each vehicle-mounted camera (22 to 2n), calculate the difference between the current camera extrinsic parameter value of the vehicle-mounted camera (22 to 2n) and the predefined initial camera extrinsic parameter value of the vehicle-mounted camera (22 to 2n); and The current three-dimensional pose of the vehicle (3) is calculated based on the difference between the current external parameter value of the vehicle-mounted camera (22 to 2n) and the predefined initial external parameter value of the vehicle-mounted camera (22 to 2n). in, Based on the calculated current three-dimensional attitude of the vehicle (3), the ground surface is modeled.

3. The method according to claim 2, wherein, The three-dimensional pose of the vehicle (3) is calculated in homogeneous coordinates using affine transformations, wherein the affine transformations include rotations related to the rotation of the vehicle (3) and translations related to the translation of the vehicle (3).

4. The method according to claim 3, wherein, The three-dimensional pose of the vehicle (3) is calculated by applying multivariate interpolation to the current external parameter values ​​of the camera.

5. The method according to claim 4, wherein, The multivariate interpolation includes at least one of bilinear interpolation and bicubic interpolation.

6. The method according to any one of claims 1 to 5, in, The vehicle suspension data includes current suspension height information for each wheel (33); as well as For each wheel (33), the displacement is calculated based on the difference between the current suspension height and the predefined initial suspension height.

7. The method according to any one of claims 2 to 6, wherein, The position of each wheel (33) of the vehicle (3) is calculated based on the three-dimensional attitude of the vehicle (3) and / or based on the calculated displacement of the wheel (33) of the vehicle (3).

8. The method according to claim 7, wherein, The ground surface was modeled using multivariate interpolation of the calculated wheel (33) positions.

9. The method according to claim 8, wherein, The multivariate interpolation of the calculated wheel (33) position includes at least one of bilinear interpolation and bicubic interpolation.

10. A device (1) for providing visualization of a vehicle (3), comprising: Interface (11) is used to receive the current camera external parameter value of each of the multiple vehicle cameras (22 to 2n) of the vehicle (3), and to receive vehicle suspension data related to the wheel (33) suspension of the vehicle (3). as well as The computing unit (12) is adapted to compute the visualization of the vehicle (3) using a predefined model of the vehicle (3), wherein the vehicle (3) is visualized on the ground surface. The computing unit (12) is adapted to model the ground surface so that the wheels (33) of the vehicle (3) can contact the ground surface based on the current camera extrinsic parameter values ​​of the vehicle-mounted cameras (22 to 2n) and the suspension data. The calculation unit (12) is adapted to calculate the displacement of each wheel (33) based on the vehicle suspension data, visualize the displacement of each wheel (33) with its own displacement, and model and visualize the ground surface based on the calculated displacement of the wheels (33) of the vehicle (3).

11. The apparatus (1) according to claim 10 further includes a display (13) for outputting a visualization of the calculated vehicle (3).

12. Vehicles (3), including Multiple vehicle-mounted cameras (22 to 2n); and The apparatus (1) according to claim 10 or 11.

13. The vehicle (3) according to claim 12, further comprising at least one sensor (21), said sensor being adapted to - Measure the current camera external parameters and suspension data of the vehicle-mounted camera (22 to 2n), and - Provide the device (1) with the measured current camera external parameters and suspension data of the vehicle-mounted camera (22 to 2n).

14. The vehicle (3) according to claim 12 or 13, comprising at least four vehicle-mounted cameras (22 to 2n) disposed around the vehicle (3).

Images