An image-based method for estimating vehicle sideslip angle

Abstract

A method for estimating the sideslip angle of a vehicle's center of mass based on an image relates to a method for estimating vehicle information. The relationship between the vehicle camera coordinate system, the image coordinate system, and the earth coordinate system is determined, and the camera parameters are calibrated to include the camera's focal length, rotation matrix, and translation vector. An inverse perspective transformation is performed on the image to parallelize previously parallel tracks on the ground. The image texture direction is processed to determine the direction of the camera's ground velocity. Coordinate transformation is then used to determine the camera's coordinates in the vehicle coordinate system, and the vehicle's center of mass velocity is calculated to ultimately determine the vehicle's sideslip angle. Using a vehicle camera to estimate the vehicle's sideslip angle based on an image eliminates the need for a vehicle dynamics model, resulting in robustness, versatility, and minimal computational burden.

Description

Technical Field

[0001] The present invention relates to a method for estimating automobile information, in particular to an image-based method for estimating the sideslip angle of the center of mass of an automobile, and belongs to the technical field of vehicle motion state estimation. Background Art

[0002] With the continuous advancement of automotive electronics, the demand for vehicle body state information from onboard controllers is increasing. The vehicle's yaw rate, longitudinal speed, and center-of-mass slip angle are important body motion parameters. The center-of-mass slip angle reflects the deviation between the vehicle's ground speed and its headway, and is crucial for controlling vehicle stability and other aspects of its operation.

[0003] Yaw rate, as an angular velocity, is easily measured using on-board sensors, and longitudinal vehicle speed can be estimated fairly accurately using wheel speed. However, the sideslip angle is difficult to measure directly and requires estimation. Currently, estimation of the sideslip angle relies primarily on vehicle dynamics models, particularly high-precision models. Consequently, current estimation methods generally suffer from drawbacks such as reliance on vehicle dynamics parameters, poor robustness, limited versatility, and a high computational burden.

[0004] Furthermore, the widespread availability of onboard cameras, such as reversing cameras and electronic rearview mirrors, provides a hardware foundation for camera-based estimation of the vehicle's center of mass sideslip angle. Therefore, it is necessary to propose an image-based method for estimating the vehicle's center of mass sideslip angle based on onboard cameras to address the common issues with current estimation methods. Summary of the Invention

[0005] To address the shortcomings of the background technology, the present invention provides an image-based method for estimating the sideslip angle of the center of mass of a vehicle. It uses an on-board camera to estimate the sideslip angle of the center of mass of the vehicle based on an image, does not require a vehicle dynamics model, has good robustness, strong versatility, and low computational burden.

[0006] To achieve the above object, the present invention adopts the following technical solution: a method for estimating the sideslip angle of the center of mass of a vehicle based on an image, comprising the following steps:

[0007] Step 1: Calibrate the vehicle camera parameters

[0008] Determine the relationship between the vehicle camera coordinate system, the image coordinate system, and the earth coordinate system. The origin of the vehicle camera coordinate system is at the vehicle camera. Its X-axis and Y-axis directions are consistent with the arrangement of the photosensitive component array of the vehicle camera. The Z-axis coaxially points to the front of the vehicle camera. The coordinates of the point in the vehicle camera coordinate system are expressed as (x C ,y C ,z C), the image coordinate system is the plane coordinate system of the image captured by the vehicle camera, and its X-axis and Y-axis directions are consistent with the pixel arrangement in the image. The coordinates of the point in the image coordinate system are expressed as (x I ,y I ), the origin of the geodetic coordinate system is on the ground directly below the vehicle-mounted camera, and its X-axis and Y-axis directions are respectively the projections of the X-axis and Y-axis of the vehicle-mounted camera coordinate system on the ground, and the Z-axis passes through the vehicle-mounted camera and points directly upward. The coordinates of the point in the geodetic coordinate system are expressed as (x G ,y G ,z G ),

[0009] The conversion relationship of the coordinates of a certain point from the earth coordinate system to the vehicle camera coordinate system is:

[0010]

[0011] Where R is the rotation matrix and T is the translation vector, which can be expressed as:

[0012]

[0013]

[0014] Among them, r ij (i=1,2,3,j=1,2,3) represents the elements in the rotation matrix R, t x , t y , t z They represent the x, y, and z coordinates of the origin of the vehicle-mounted camera in the geodetic coordinate system,

[0015] Due to t x =t y =0, and the conversion relationship from the vehicle camera coordinate system to the image coordinate system is:

[0016]

[0017] Where f is the focal length of the vehicle camera,

[0018] The internal parameters that need to be collected for vehicle camera calibration are the focal length of the vehicle camera, and the external parameters are the rotation matrix and translation vector;

[0019] Step 2: Inverse perspective transformation

[0020] Perform inverse perspective transformation on the image so that the parallel tracks on the ground are parallel. For a point on the ground, the coordinates in the geodetic coordinate system are (x G ,y G ,0), converted to the vehicle camera coordinate system is expressed as:

[0021]

[0022] The perspective transform is:

[0023]

[0024] After finishing, we can get:

[0025]

[0026]

[0027] According to the above formula, the coordinates of the points on the ground in the image coordinate system in the geodetic coordinate system can be obtained.

[0028] The inverse perspective transformation is:

[0029]

[0030] In the formula, (x' I ,y' I ) represents the coordinates after inverse perspective transformation;

[0031] Step 3: Image texture direction recognition

[0032] The image captured by the vehicle-mounted camera is processed after inverse perspective transformation to obtain the image texture direction θ, which is the direction of the vehicle-mounted camera's ground speed;

[0033] Step 4: Coordinate transformation

[0034] Determine the actual position of the vehicle camera relative to the vehicle body. Define the origin of the vehicle coordinate system at the center of mass of the vehicle, with the x-axis pointing to the front of the vehicle body and the y-axis pointing to the left of the vehicle body. Assume that the coordinates of the vehicle camera in the vehicle coordinate system are (a, b). Then the ground speed of the vehicle camera is:

[0035] (v x -br,v y +ar)

[0036] Among them, v x and v y are the components of the vehicle's center of mass velocity on the x-axis and y-axis, r is the yaw rate,

[0037] The direction of the vehicle-mounted camera's ground velocity is the image texture direction θ, then:

[0038] (v x -br)sinθ=(v y +ar)cosθ

[0039] Because a, b, v x , r and θ are known, we can solve for vy :

[0040] v y =tanθ(v x -br)-ar

[0041] The car's center of mass side slip angle

[0042] Compared with the existing technology, the beneficial effects of the present invention are: the present invention uses an on-board camera to propose an image-based method for estimating the sideslip angle of the center of mass of a vehicle. Compared with the current estimation method, the present invention does not require a vehicle dynamics model, has low R&D costs, good robustness, can be easily transplanted between various vehicle models, has strong versatility, does not rely on the vehicle's dynamic parameters, does not require calculation of vehicle dynamics, and has a small computational burden. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a schematic diagram of the vehicle-mounted camera coordinate system, image coordinate system, and geodetic coordinate system in the present invention;

[0044] Figure 2 is a schematic diagram of the trajectories of various points on the ground during the perspective transformation phase of the present invention;

[0045] Figure 3 is a schematic diagram of the trajectories of various points on the ground during the inverse perspective transformation stage of the present invention;

[0046] Figure 4 is the image captured by the vehicle-mounted camera and subjected to inverse perspective transformation in the present invention;

[0047] Figure 5 yes Figure 4 The result of canny edge detection on the image;

[0048] Figure 6 yes Figure 4 Schematic diagram of texture direction of the image in the figure;

[0049] Figure 7 It is a schematic diagram of the position of the vehicle-mounted camera relative to the vehicle body in the present invention. DETAILED DESCRIPTION

[0050] The technical solutions of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0051] like Figures 1 to 7 As shown, a method for estimating the sideslip angle of the center of mass of a vehicle based on an image includes the following steps:

[0052] Step 1: Calibrate the vehicle camera parameters

[0053] Combine Figure 1 As shown, the relationship between the vehicle-mounted camera coordinate system, the image coordinate system and the earth coordinate system is determined, and the origin of the vehicle-mounted camera coordinate system is at the vehicle-mounted camera. C X、 C Y and C Z represents its X axis, Y axis and Z axis respectively, where C X and C The Y direction is consistent with the arrangement of the photosensitive components array of the vehicle camera. C The Z axis points to the front of the vehicle camera, and the coordinates of the point in the vehicle camera coordinate system are expressed as (x C ,y C ,z C ), the image coordinate system is the plane coordinate system of the image captured by the vehicle-mounted camera, I X and I Y represents its X-axis and Y-axis respectively, I X and I The Y direction is consistent with the pixel arrangement in the image, and the coordinates of the point in the image coordinate system are expressed as (x I ,y I ), the origin of the geodetic coordinate system is at the ground directly below the vehicle-mounted camera, G X、 G Y and G Z represents its X axis, Y axis and Z axis respectively, where G The X direction is C The projection of X on the ground, G The Y direction is C The projection of Y on the ground, G Z passes through the vehicle-mounted camera and points upwards. The coordinates of the point in the geodetic coordinate system are expressed as (x G ,y G ,z G ).

[0054] The conversion relationship of the coordinates of a certain point from the earth coordinate system to the vehicle camera coordinate system is:

[0055]

[0056] Where R is the rotation matrix and T is the translation vector, which can be expressed as:

[0057]

[0058]

[0059] Among them, r ij(i=1,2,3,j=1,2,3) represents the elements in the rotation matrix R, t x , t y , t z They represent the x, y, and z coordinates of the origin of the vehicle-mounted camera in the geodetic coordinate system.

[0060] Since the vehicle-mounted camera coordinate system is defined directly above the earth coordinate system, t x =t y =0, and the conversion relationship from the vehicle camera coordinate system to the image coordinate system is:

[0061]

[0062] Where f is the focal length of the vehicle camera.

[0063] During the conversion process, the coordinates of a point in the geodetic coordinate system undergo a rotation transformation, a translation transformation, and a projection transformation. The corresponding parameters are the rotation matrix R, the translation vector T, and the focal length f of the vehicle-mounted camera. Therefore, the intrinsic parameters that need to be collected for vehicle-mounted camera calibration are the focal length f of the vehicle-mounted camera, and the extrinsic parameters are the rotation matrix R and the translation vector T.

[0064] Step 2: Inverse perspective transformation

[0065] Combine Figure 2 As shown in Figure 1, because translation does not change the direction of a vector, and projection causes parallel lines to converge due to perspective, trajectories that should be parallel on the ground will intersect at a point in the captured image, making it difficult to identify the texture direction. Therefore, we first need to perform an inverse perspective transformation on the image to make the trajectories that were originally parallel on the ground parallel in the captured image.

[0066] For a point on the ground, the coordinates in the geodetic coordinate system are (x G ,y G ,0), converted to the vehicle camera coordinate system is expressed as:

[0067]

[0068] The perspective transform is:

[0069]

[0070] From the above formula, we can conclude that there is a one-to-one correspondence between the points in the image coordinate system and the points on the ground. Therefore, we can rearrange the above formula to get:

[0071]

[0072]

[0073] According to the above formula, the coordinates of the point on the ground in the image coordinate system in the geodetic coordinate system can be obtained.

[0074] Perspective transformation due to x C and y C Divide up by z C This results in objects being larger when near and smaller when far, which further causes the parallel lines on the ground to intersect at one point on the image. By multiplying the inverse perspective transformation by z C Eliminating this effect can be expressed as:

[0075]

[0076] In the formula, (x' I ,y' I ) represents the coordinate after inverse perspective transformation, z C Available through x G and y G express:

[0077] z C =r 31 x G +r 32 y G +t z

[0078] Combined with x G and y G By x I and y I Find, finally z C Can be obtained by x I and y I To obtain.

[0079] Combine Figure 3 As shown, the image is finally processed through inverse perspective transformation to make the ground track that intersects at one point on the image parallel again;

[0080] Step 3: Image texture direction recognition

[0081] The image captured by the vehicle camera is transformed into the following after inverse perspective transformation: Figure 4 As shown, after processing, the image texture direction θ can be obtained as Figure 6 As shown, the obtained image texture direction θ is the direction of the vehicle-mounted camera's ground speed.

[0082] There are many methods for identifying image texture direction, such as edge detection-based methods, neural network-based methods, and consistency check-based methods. The present invention adopts an edge detection-based method involving a canny edge detection algorithm and two convolution kernels.

[0083] First, perform canny edge detection on the image after inverse perspective transformation, and the result is as follows Figure 5 As shown,

[0084] Then use the convolution kernel Convolution kernel Perform convolution and get W 1sum and W 2sum , W 2sum It represents the overall downward trend of the image edge, and W 1sum It represents the overall trend of the image edge moving left and right.

[0085] Finally, the image texture direction is expressed as

[0086] Step 4: Coordinate transformation

[0087] Combine Figure 7 As shown, determine the actual position of the vehicle camera relative to the vehicle body, define the origin of the vehicle body coordinate system at the vehicle center of mass, the x-axis points to the front of the vehicle body, and the y-axis points to the left of the vehicle body. Assume that the coordinates of the vehicle camera in the vehicle body coordinate system are (a, b), then the ground speed of the vehicle camera is:

[0088] (v x -br,v y +ar)

[0089] Among them, v x and v y are the components of the vehicle's center of mass velocity on the x-axis and y-axis, respectively, and r is the yaw rate.

[0090] According to the image texture direction, the direction of the vehicle camera's ground speed is the image texture direction θ, so:

[0091] (v x -br)sinθ=(v y +ar)cosθ

[0092] Because a, b, v x , r and θ are known, we can solve for v y :

[0093] v y =tanθ(v x -br)-ar

[0094] The vehicle's center of mass sideslip angle β is further obtained:

[0095]

[0096] The present invention provides an image-based method for estimating the sideslip angle of a vehicle's center of mass. In the first stage, the vehicle camera parameters are calibrated. The internal parameters of the vehicle camera reflect focal length information, while the external parameters reflect position and posture. After obtaining the internal and external parameters of the vehicle camera, the vehicle coordinate system can be linked to the image coordinate system. Each point on the ground in the vehicle coordinate system has a unique, reversible coordinate transformation with the corresponding point in the image coordinate system. Therefore, for a vehicle or a vehicle of the same model, the parameters of the vehicle camera only need to be calibrated once. In the second stage, an inverse perspective transformation is performed. Due to the perspective effect, parallel straight lines on the ground intersect at a point in the image, which is not conducive to direction identification. Based on the parameters of the vehicle camera, the perspective effect of the image is removed, so that parallel straight lines on the ground remain parallel in the image. In the third stage, image texture direction identification is performed. When the vehicle moves, points on the ground also move relative to the vehicle camera, leaving a trajectory within the exposure time of the vehicle camera. The trajectory of each point on the ground forms the image texture. The direction of the image texture reflects the direction of the ground's motion speed relative to the vehicle camera in the image coordinate system. The image texture direction can be identified based on image processing methods such as edge detection. In step four, coordinate transformation is performed. Based on the position of the vehicle camera relative to the center of mass, combined with the vehicle's yaw rate and longitudinal speed, the speed of the vehicle camera relative to the ground is calculated. Finally, the speed of the center of mass relative to the road surface is calculated based on this speed, and then the center of mass sideslip angle is calculated.

[0097] Generally speaking, as a car moves, its onboard camera captures real-time images of the ground. Due to the car's motion relative to the ground, each point on the ground produces a trajectory in the camera's image plane during the exposure time. These trajectories create a directional texture on the image, whose direction reflects the camera's velocity relative to the ground. Therefore, we first perform an inverse perspective transformation on the camera's image. Then, we extract the direction of the image texture. Combined with the camera's position relative to the ground and the car's body, we can determine the direction of the car's velocity relative to the ground, and further calculate the center of mass sideslip angle. This method requires no vehicle dynamics model, is highly versatile, has a low computational burden, and can be easily ported across various vehicle models, making it highly suitable for practical applications.

[0098] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other configurations without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations coming within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

[0099] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A method for estimating the sideslip angle of a vehicle's center of mass based on an image, characterized by: The following steps are involved: Step 1: Calibrate the vehicle camera parameters Determine the relationship between the vehicle camera coordinate system, the image coordinate system, and the earth coordinate system. The origin of the vehicle camera coordinate system is at the vehicle camera. Its X-axis and Y-axis directions are consistent with the arrangement of the photosensitive component array of the vehicle camera. The Z-axis coaxially points to the front of the vehicle camera. The coordinates of the point in the vehicle camera coordinate system are expressed as (x C ,y C ,z C ), the image coordinate system is the plane coordinate system of the image captured by the vehicle camera, and its X-axis and Y-axis directions are consistent with the pixel arrangement in the image. The coordinates of the point in the image coordinate system are expressed as (x I ,y I ), the origin of the geodetic coordinate system is on the ground directly below the vehicle-mounted camera, and its X-axis and Y-axis directions are respectively the projections of the X-axis and Y-axis of the vehicle-mounted camera coordinate system on the ground, and the Z-axis passes through the vehicle-mounted camera and points directly upward. The coordinates of the point in the geodetic coordinate system are expressed as (x G ,y G ,z G ), The conversion relationship of a point's coordinates from the earth coordinate system to the vehicle camera coordinate system is: Where R is the rotation matrix and T is the translation vector, which can be expressed as: Among them, r ij (i=1,2,3,j=1,2,3) represents the elements in the rotation matrix R, t x , t y , t z They represent the x, y, and z coordinates of the origin of the vehicle-mounted camera in the geodetic coordinate system, Due to t x =t y =0, and the conversion relationship from the vehicle camera coordinate system to the image coordinate system is: Where f is the focal length of the vehicle camera, The internal parameters that need to be collected for vehicle camera calibration are the focal length of the vehicle camera, and the external parameters are the rotation matrix and translation vector; Step 2: Inverse perspective transformation Perform inverse perspective transformation on the image so that the parallel tracks on the ground are parallel. For a point on the ground, the coordinates in the geodetic coordinate system are (x G ,y G ,0), converted to the vehicle camera coordinate system is expressed as: The perspective transform is: After finishing, we can get: According to the above formula, the coordinates of the points on the ground in the image coordinate system in the geodetic coordinate system can be obtained. The inverse perspective transformation is: In the formula, (x' I ,y' I ) represents the coordinates after inverse perspective transformation; Step 3: Image texture direction recognition The image captured by the vehicle-mounted camera is processed after inverse perspective transformation to obtain the image texture direction θ, which is the direction of the vehicle-mounted camera's ground speed; Step 4: Coordinate transformation Determine the actual position of the vehicle camera relative to the vehicle body. Define the origin of the vehicle coordinate system at the center of mass of the vehicle, with the x-axis pointing to the front of the vehicle body and the y-axis pointing to the left of the vehicle body. Assume that the coordinates of the vehicle camera in the vehicle coordinate system are (a, b). Then the ground speed of the vehicle camera is: (v x -br,v y +ar) Among them, v x and v y are the components of the vehicle's center of mass velocity on the x-axis and y-axis, r is the yaw rate, The direction of the vehicle-mounted camera's ground velocity is the image texture direction θ, then: (v x -br)sinθ=(v y +ar)cosθ Because a, b, v x , r and θ are known, we can solve for v y : v y =tanθ(v x -br)-ar The car's center of mass side slip angle 2. The method for estimating the vehicle's sideslip angle based on an image according to claim 1, wherein: The method for obtaining the image texture direction θ in step 3 adopts an edge detection method. First, the image after inverse perspective transformation is subjected to canny edge detection, and then the convolution kernel is used to obtain the image texture direction θ. Convolution kernel Perform convolution and get W 1sum and W 2sum , W 2sum It represents the overall downward trend of the image edge, and W 1sum Represents the overall trend of the image edge moving left and right, the image texture direction

Images