Road vertical profile detection using stable coordinate system

Abstract

The invention relates to road vertical profile detection using a stable coordinate system. In some embodiments, a first homography matrix created from two images of a road is decomposed to determine self-motion, and the self-motion is used to adjust a previous estimate of the road plane. The adjusted previous estimate of the road plane is combined with the current plane estimate to create a second homography matrix, and the second homography matrix is used to determine residual motion and vertical deviation in the road surface. In some embodiments, a plurality of road profiles each corresponding to a common portion of a road are adjusted in slope and offset by optimizing a function having a data item, a smoothness item, and a regularization item; and the adjusted road longitudinal sections are combined into a multi-frame road longitudinal section. In some embodiments, road profile information for a predetermined number of data points is transmitted in periodic data bursts, where each data burst has more than one data point.

Description

[0001] This application is a divisional application of the application with the filing date of February 28, 2016, application number 202111234124.2, and invention title "Road Vertical Profile Detection Using a Stable Coordinate System".

[0002] The application with the filing date of February 28, 2016, application number 202111234124.2, and invention title "Road Vertical Profile Detection Using a Stable Coordinate System" is a divisional application of the application with the filing date of February 28, 2016, application number 201680017324.3, and invention title "Road Vertical Profile Detection Using a Stable Coordinate System".

[0003] Cross - reference to related applications

[0004] This application claims priority to the following applications: U.S. Provisional Patent Application 62 / 120,929, entitled "Road Plane Profile Output in a Stabilized World Coordinate Frame", filed on February 26, 2015; U.S. Provisional Patent Application 62 / 131,374, entitled "Road Plane Output in a Stabilized World Coordinate Frame", filed on March 11, 2015; U.S. Provisional Patent Application 62 / 149,699, entitled "Road Plane Profile Output in a Stabilized World Coordinate Frame", filed on April 20, 2015; and U.S. Provisional Patent Application 62 / 238,980, entitled "Road Plane Output in a Stabilized World Coordinate Frame", filed on October 8, 2015; the disclosures of all applications are incorporated herein by reference.

[0005] Technologically related applications

[0006] This application is related to U.S. Application No. 14 / 554,500 (now U.S. Patent No. 9,256,791), entitled "Road Vertical Contour Detection", filed on November 26, 2014; and U.S. Application No. 14 / 798,575, entitled "Road Contour Vertical Detection", filed on July 14, 2015; both are incorporated herein by reference in their entirety. Background Technical field

[0007] This application relates to a driver assistance system and method for detecting vertical deviations of a road profile using a camera, and more particularly to a driver assistance system and method for detecting vertical deviations of a road profile using a stable coordinate system.

[0008] Description of Related Art

[0009] In recent years, camera-based driver assistance systems (DAS) have entered the market, including lane departure warning (LDW), automatic high beam control (AHC), traffic sign recognition (TSR), forward collision warning (FCW), and pedestrian detection.

[0010] Now refer to Figure 1 and Figure 2 , which illustrate a system 16 according to some embodiments. The system includes a camera or image sensor 12 mounted in a vehicle 18. The image sensor 12 that images the field of view in the forward direction provides image frames 15 in real time, and the image frames 15 are captured by an image processor 30. The processor 30 can be used to process the image frames 15 simultaneously and / or in parallel for multiple DAS / applications. The processor 30 can be used to process the image frames 15 to detect and identify images or portions of images in the forward field of view of the camera 12. The DAS can be implemented using specific hardware circuits (not shown) with on-vehicle software and / or software control algorithms in a memory 13. The image sensor 12 can be monochrome or black and white, i.e., without color separation, or the image sensor 12 can be color sensitive. By way of example in Figure 2 , according to some embodiments, the image frames 15 are used for pedestrian detection 20, TSR 21, FCW 22, and real-time detection 23 of the vertical profile of the road or deviations from the road plane.

[0011] In some embodiments, more than one camera can be installed in the vehicle. For example, the system can have multiple cameras pointing in different directions. The system can also have multiple cameras that point in the same or similar directions relative to the vehicle but are installed at different locations. In some embodiments, the system can have multiple cameras with partially or fully overlapping fields of view. In some embodiments, two side-by-side cameras can operate in stereo. The non-limiting examples discussed herein consider a single-camera system, but they can be similarly implemented in a multi-camera system, where some or all of the relevant images and frames can be captured by different cameras, or can be created from a synthesis of images captured from multiple cameras.

[0012] In some cases, the image frames 15 are divided among different driver assistance applications, and in other cases, the image frames 15 can be shared among different driver assistance applications.

[0013] Some existing methods for detecting the vertical deviation of road profiles using on-vehicle cameras are known.

[0014] Some previously known algorithms can be summarized as follows:

[0015] 1. Align the first pair of two consecutive frames captured by the on-vehicle camera using the homography matrix of the road. This gives the ego-motion (rotation R and translation T) between the frames. Figure 3A Thirty-three grid points tracked in the image and used to calculate the homography matrix are shown.

[0016] 2. Then select a second pair of frames, the current frame and the most recent previous frame representing the time point when the vehicle started its current movement after a minimum threshold distance. The link of the first pair of frames (consecutive frames) is used to create an initial guess of the homography matrix, then a more accurate homography matrix of the second pair of frames is calculated, and the reference plane is determined.

[0017] 3. Then project the path of the vehicle onto the image plane (as shown by the line in Figure 3C ). The strips along each path are used to calculate the residual motion, which gives a distribution map relative to the determined reference plane. Figure 3B The normalized correlation scores of a 31-pixel-wide strip along the path of the left wheel for vertical motion of ±6 pixels are shown. The small bend indicated by the arrow can be seen. This bend represents a small residual motion indicating a speed bump.

[0018] 4. Finally, the residual motion is converted to metric distances and heights and combined into a multi-frame model.

[0019] The results are shown in the upper figure in Figure 3C .

[0020] A previously known algorithm according to the above brief description is described in more detail in U.S. Patent No. 9,256,791.

[0021] In Figure 4A and Figure 4B an example of data generated according to the previous method is shown. Figure 4A and Figure 4B show the road profile from a series of frames aligned using the recovered rotation (R), translation (T), and plane normal (N). A speed bump can be seen near 40 meters, and strong divergence of the signal due to incorrect motion and plane normal can also be seen; the sampling shows a divergence of approximately 0.1 meters in height between the profiles.

[0022] Brief overview

[0023] A disadvantage of known methods such as the above method is the use of different reference planes at each time step. The reference plane used at each time step is related to the true ground plane of the road calculated based on the frames at each corresponding step. By recalculating this ground plane at each step, there may be small deviations between the reference ground planes in each step, making it troublesome to combine information from multiple frames at different time steps. For example, when there are speed bumps on the road, especially when the speed bumps or ridges occupy most of one or more frames, the main plane can combine parts of the ridges and parts of the road, resulting in inaccurate calculation of the vertical deviation of the road. In addition, small deviations in the road plane may cause new measurements to be poorly aligned. In theory, the ego-motion and plane information derived from the homography matrix can align new measurements; however, noise in the plane normal and ego-motion (especially translation) is not conducive to achieving accurate alignment. These problems can be seen in the example results shown in Figure 7A and Figure 7B which are described in more detail below.

[0024] Another disadvantage of some prior methods is that they cannot account for the effects of lens distortion (e.g., radial distortion) and rolling shutter. If the effects of lens distortion and rolling shutter are ignored, prior known methods may yield a qualitatively correct single result with a vertical deviation of the identified road plane, but the distance may be inaccurate, the ego-motion may be inaccurate, and the resulting profile may have a small upward curvature.

[0025] Therefore, there is a need for systems and methods for calculating the vertical deviation of a road surface using a consistent ground plane. As explained herein, an improved method can create a homography matrix based on a consistent ground plane at each time step, and then this homography matrix can be used to align frames and calculate residual motion and road profiles.

[0026] As used herein, the term "homography matrix" refers to an invertible transformation from a projective space to itself that maps lines to lines. In the field of computer vision, two images of the same planar surface in space are related by a homography matrix characteristic of a pinhole camera model. In some embodiments of the present disclosure that refer to the homography matrix, alternative models of the road surface can also be used in a similar manner; for example, in some embodiments, the road can be modeled as a quadric surface with a certain curvature.

[0027] Specifically, by way of example, for a given camera 12 with a height (1.25 m), a focal length (950 pixels), and vehicle motion (1.58 m) between frames, the motion of points on the road plane between two image frames 15a and 15b can be predicted respectively. Using a model of an almost flat surface for the motion of road points, the second image 15b can be warped towards the first image 15a. The following Matlab TM code will perform the initial warping step 501:

[0028]

[0029] where dZ is the forward motion of the vehicle 18, H is the height of the camera 12, and f is the focal length of the camera 12. P0 = (x0; y0) is the vanishing point of the road structure. Alternatively, initial calibration values can be used during installation of the system 16 in the vehicle 18, where x0 is the forward direction of the vehicle 18 and y0 is the horizontal line when the vehicle 18 is on a horizontal surface. The variable S is an overall scaling factor related to the image coordinates between two image frames 15a and 15b captured at different vehicle distances Z from the camera 12. The term "relative scaling variation" as used herein refers to the overall scaling variation of image coordinates that depends on the distance Z to the camera 12.

[0030] In addition, systems and methods are needed that take into account the effects of radial distortion and rolling shutter in order to accurately detect distances, accurately measure self-motion, and accurately generate a road profile without artificial upward curvature. In some embodiments, such systems and methods can take into account radial distortion and rolling shutter, but can use the initial image for point tracking (without adjustment for rolling shutter and radial distortion) in order to avoid computationally expensive image pre-warping.

[0031] Various driver assistance systems and computerized methods that can be installed in a host vehicle are provided herein for detecting vertical deviations of a road surface. When the host vehicle is moving, the method can be performed by a DAS installed in the host vehicle. The DAS can include a camera operatively connected to a processor.

[0032] 1) In some embodiments, a computerized road surface deviation detection method is performed by a driver assistance system installed in a vehicle, wherein the driver assistance system includes a camera operably connected to a processor. In some embodiments, the method includes: capturing a plurality of images by the camera, the plurality of images including a first image of the road surface captured at a first time and a second image of the road surface captured at a second time, the second time being after the first time; determining a first estimate of the planar normal of the road and a second estimate of the planar normal of the road based at least on the first image and the second image; creating a model of the road surface based at least on the first estimate of the planar normal and the second estimate of the planar normal; using the model of the road surface to determine a residual motion along a projected path of the vehicle; calculating a vertical deviation of the road surface based on the residual motion; and transmitting the vertical deviation data to a vehicle control system.

[0033] 2) The method as described in 1), wherein the model of the road surface is a homography matrix.

[0034] 3) The method as described in 1), wherein determining the first estimate of the planar normal of the road and the second estimate of the planar normal of the road includes:

[0035] calculating an initial homography matrix based at least on the first image of the road and the second image of the road,

[0036] decomposing the initial homography matrix to determine a self - motion and the second estimate of the planar normal of the road, and

[0037] adjusting a previous estimate of the planar normal to generate the first estimate of the planar normal, the adjustment being based at least on the determined self - motion.

[0038] 4) The method as described in 3), wherein calculating the initial homography matrix includes tracking points between the first image and the second image, wherein the points are distributed in the first image or the second image to correspond to positions on the road that are evenly spaced apart from each other.

[0039] 5) The method as described in 3), wherein calculating the initial homography matrix includes tracking points between the first image and the second image, wherein variable weights are assigned to the points based on their positions in the first image or the second image.

[0040] 6) The method as described in 1), wherein the vehicle control system is configured to adjust a suspension system of the vehicle in response to the calculated vertical deviation in the road surface.

[0041] 7) The method as described in 1), wherein the model of the road surface is based at least on a historical average of the planar normals.

[0042] 8) The method according to 7), wherein the first estimate of the plane normal is weighted more heavily in the model of the road surface than the historical average of the plane normal.

[0043] 9) The method according to 1), wherein the first estimate of the plane normal is weighted more heavily in the model of the road surface than the second estimate of the plane normal.

[0044] 10) The method according to 1), comprising:

[0045] calculating a vehicle path in world coordinates, and

[0046] projecting the vehicle path onto the second image using stable plane parameters determined according to the model of the road surface.

[0047] 11) The method according to 1), comprising:

[0048] correcting the coordinates of points in the plurality of images for at least one of radial distortion and rolling shutter.

[0049] 12) The method according to 11), wherein the correction is performed for at least one of radial distortion and rolling shutter after calculating the model of the road surface.

[0050] 13) In some embodiments, a driver assistance system is installed in a vehicle, the system comprising a processor, a camera operably connected to the processor, and a memory storing instructions that, when executed by the processor, cause the system to perform the following operations: capturing a plurality of images via the camera, the plurality of images including a first image of a road surface captured at a first time and a second image of the road surface captured at a second time, the second time being after the first time; determining at least a first estimate of a plane normal of the road and a second estimate of the plane normal of the road based at least on the first image and the second image; creating a model of the road surface based at least on the first estimate of the plane normal and the second estimate of the plane normal; determining a residual motion along a projected path of the vehicle using the model of the road surface; calculating a vertical deviation of the road surface based on the residual motion; and transmitting vertical deviation data to a vehicle control system.

[0051] 14) In some embodiments, a computer-readable storage medium stores instructions that, when executed by a processor operatively coupled to a driver assistance system installed in a host vehicle, cause the system to perform the following operations: capture a plurality of images via a camera operatively connected to the processor, the plurality of images including a first image of a road surface captured at a first time and a second image of the road surface captured at a second time, the second time being after the first time; determine a first estimate of a planar normal of the road and a second estimate of the planar normal of the road based at least on the first image and the second image; create a model of the road surface based at least on the first estimate of the planar normal and the second estimate of the planar normal; use the model of the road surface to determine a residual motion along a projected path of the vehicle; calculate a vertical deviation of the road surface based on the residual motion; and transmit the vertical deviation data to a vehicle control system. In some embodiments, the computer-readable storage medium is transient. In some embodiments, the computer-readable storage medium is non-transient.

[0052] 15) In some embodiments, a computerized road surface deviation detection method is performed by a driver assistance system installed in a vehicle, wherein the driver assistance system includes a camera operatively connected to a processor. In some embodiments, the method includes: selecting at least two road profiles, wherein each of the at least two road profiles includes a common portion of the road; optimizing a function including a data term, a smoothness term, and a regularization term to align a first profile from the at least two profiles; and combining the at least two road profiles into a multi-frame profile.

[0053] 16) The method as described in 15), wherein aligning the first profile from the at least two profiles includes:

[0054] determining an optimized slope value and an optimized offset value based on the optimization,

[0055] modifying the first profile from the at least two road profiles using the optimized slope value and the optimized offset value.

[0056] 17) The method as described in 15), wherein the at least two road profiles are generated using a stable world coordinate system.

[0057] 18) The method as described in 15), wherein combining the at least two road profiles includes calculating a median of the profiles.

[0058] 19) In some embodiments, a driver assistance system installed in a vehicle includes a processor, a camera operably connected to the processor, and a memory storing instructions that, when executed by the processor, cause the system to perform the following operations: select at least two road profiles, wherein each of the at least two road profiles includes a common portion of the road; optimize a function including a data term, a smoothness term, and a regularization term to align a first road profile from the at least two road profiles; and combine the at least two road profiles into a multi-frame road profile.

[0059] 20) In some embodiments, a computer-readable storage medium stores instructions that, when executed by a processor operably coupled to a driver assistance system installed in a host vehicle, cause the system to perform the following operations: select at least two road profiles, wherein each of the at least two road profiles includes a common portion of the road; optimize a function including a data term, a smoothness term, and a regularization term to align a first road profile from the at least two road profiles; and combine the at least two road profiles into a multi-frame road profile. In some embodiments, the computer-readable storage medium is transient. In some embodiments, the computer-readable storage medium is non-transient.

[0060] 21) In some embodiments, a method of transmitting road profile information in a driver assistance system installed in a vehicle is performed, wherein the driver assistance system includes a camera and a data interface operably connected to a processor. In some embodiments, the method includes: selecting a first distance in front of a wheel of the vehicle; sampling a predetermined number of data points from a road profile along a projected path of the vehicle, wherein the road profile is at least partially based on an image captured by the camera, and wherein the data points are sampled along a segment of the path having endpoints corresponding to the selected first distance, and wherein one or more data points are generated based on a stable reference plane; and transmitting, via the data interface, road profile information corresponding to the predetermined number of data points in a predetermined number of data transmissions, wherein the number of data transmissions is less than the number of data points.

[0061] 22) The method as described in 21) includes:

[0062] selecting a frequency of transmitting data via the data interface, wherein the selected frequency defines a frequency of transmitting the predetermined number of data transmissions.

[0063] 23) The method as described in 21) includes:

[0064] estimating a second distance traveled by the vehicle in a time period equal to the reciprocal of the selected frequency, wherein the data points are sampled along a segment of the path corresponding to the estimated second distance that exceeds the selected first distance.

[0065] 24) The method according to 23), wherein estimating the second distance includes determining that the second distance will be traversed during the time period at a known current speed of the vehicle.

[0066] 25) The method according to 21), wherein the road profile information transmitted through the data interface is configured such that the road profile information corresponding to the respective data points can be read separately by the receiving system and is associated with the respective portions of the road represented by the road profile.

[0067] 26) The method according to 21), wherein selecting the first distance in front of the wheels of the vehicle includes taking into account system latency.

[0068] 27) The method according to 21), wherein one or more of the data points are associated with respective confidence values.

[0069] 28) The method according to 21), wherein the interval of the data points is determined such that it estimates that the vehicle will travel from one data point to the next data point within a predetermined time period.

[0070] 29) The method according to 21), wherein one or more of the data points are generated at least in part by combining road profile information from multiple frames.

[0071] 30) In some embodiments, a driver assistance system is installed in a vehicle, the system including a processor, a camera operatively connected to the processor, and a memory storing instructions that, when executed by the processor, cause the system to perform the following operations: select a first distance in front of the wheels of the vehicle; sample a predetermined number of data points from a road profile along a projected path of the vehicle, wherein the road profile is at least partially based on an image captured by the camera, and wherein the data points are sampled along a segment of the path having endpoints corresponding to the selected first distance, and wherein one or more data points are generated based on a stable reference plane; and transmit, through a data interface, road profile information corresponding to the predetermined number of data points in a predetermined number of data transmissions, wherein the number of data transmissions is less than the number of data points.

[0072] 31) In some embodiments, a computer-readable storage medium stores instructions that, when executed by a processor operatively coupled to a driver assistance system installed in a host vehicle, cause the system to perform the following operations: select a first distance in front of a wheel of the vehicle; sample a predetermined number of data points from a road profile along a projected path of the vehicle, wherein the road profile is at least partially based on an image captured by a camera operatively connected to the processor, and wherein the data points are sampled along a segment of the path having endpoints corresponding to the selected first distance, and wherein one or more data points are generated according to a stable reference plane; and transmit road profile information corresponding to the predetermined number of data points in a predetermined number of data transmissions via a data interface, wherein the number of data transmissions is less than the number of data points. In some embodiments, the computer-readable storage medium is transient. In some embodiments, the computer-readable storage medium is non-transient.

[0073] 32) In some embodiments, a computerized road surface deviation detection method is performed by a driver assistance system installed in a vehicle, wherein the driver assistance system includes a camera operatively connected to a processor. In some embodiments, the method includes: capturing, via the camera, a plurality of images depicting a portion of a road; determining whether features of the images are attributable to shadows cast by tree branches; processing the features to determine a vertical deviation of the portion of the road based on a determination that the features of the images are not attributable to shadows cast by tree branches; and reducing the consideration of the features when determining the vertical deviation of the portion of the road based on a determination that the features of the images are attributable to shadows cast by tree branches.

[0074] 33) The method as described in 32), wherein reducing the consideration of the features when determining the vertical deviation of the portion of the road includes ignoring the features when determining the vertical deviation of the portion of the road.

[0075] 34) The method as described in 32), wherein determining whether the features of the images are attributable to shadows cast by tree branches includes determining whether points on the road are detected as changing in height over time.

[0076] 35) The method as described in 32), wherein determining whether the features of the images are attributable to shadows cast by tree branches includes determining whether the height of points on the road is detected as changing in sign over time.

[0077] 36) The method as described in 32), wherein determining whether the features of the images are attributable to shadows cast by tree branches includes determining whether the height of points on the road is detected as changing by more than a threshold height amount within a predetermined time period.

[0078] 37) The method as described in 32), wherein determining whether a feature of the image is attributable to a shadow cast by a tree branch includes determining whether the detected movement is not towards the focus of expansion.

[0079] 38) The method as described in 32), wherein determining whether a feature of the image is attributable to a shadow cast by a tree branch includes measuring a gradient to determine the softness of an edge.

[0080] 39) In some embodiments, a driver assistance system is installed in a vehicle, the system including a processor, a camera operably connected to the processor, and a memory storing instructions that, when executed by the processor, cause the system to perform the following operations: capturing, via the camera, a plurality of images depicting a portion of a road; determining whether a feature of an image is attributable to a shadow cast by a tree branch; processing the feature to determine a vertical deviation of the portion of the road based on a determination that the feature of the image is not attributable to a shadow cast by a tree branch; and reducing the consideration of the feature when determining the vertical deviation of the portion of the road based on a determination that the feature of the image is attributable to a shadow cast by a tree branch. In some embodiments, the computer-readable storage medium is transient. In some embodiments, the computer-readable storage medium is non-transient.

[0081] 40) In some embodiments, a computer-readable storage medium stores instructions that, when executed by a processor operably coupled to a driver assistance system installed in a host vehicle, cause the system to perform the following operations: capturing, via a camera operably connected to the processor, a plurality of images depicting a portion of a road; determining whether a feature of an image is attributable to a shadow cast by a tree branch; processing the feature to determine a vertical deviation of the portion of the road based on a determination that the feature of the image is not attributable to a shadow cast by a tree branch; and reducing the consideration of the feature when determining the vertical deviation of the portion of the road based on a determination that the feature of the image is attributable to a shadow cast by a tree branch.

[0082] 41) In some embodiments, a computerized road surface deviation detection method is performed by a driver assistance system installed in a vehicle, wherein the driver assistance system includes a single camera operably connected to a processor. In some embodiments, the method includes: capturing, via the camera, a plurality of images depicting a portion of a road; generating a road profile at least in part based on one or more of the plurality of images of the road; associating a confidence value with a point in the road profile; and transmitting data corresponding to the points in the road profile and the confidence values associated with the points in the road profile to a vehicle control system.

[0083] 42) The method as described in 41), wherein the data corresponding to the points in the road profile includes the height of the road.

[0084] 43) The method according to 41), wherein the data corresponding to the points in the road profile includes the distance in front of the vehicle.

[0085] 44) The method according to 41), wherein associating a confidence value with a point in the road profile includes determining the confidence value based on a forward / backward verification performed between two of the plurality of images.

[0086] 45) The method according to 41), wherein associating a confidence value with a point in the road profile includes: determining the confidence value based on the curvature of a score matrix associated with a comparison of two images spaced apart in time.

[0087] 46) The method according to 41), comprising:

[0088] transmitting a multi-frame confidence value by means of the data corresponding to the points in the road profile and the confidence values associated with the points in the road profile, wherein the multi-frame confidence value is determined based on a plurality of height measurements from different images of the points in the road profile and the differences between the height measurements.

[0089] 47) In some embodiments, a driver assistance system is installed in a vehicle, the system including a processor, a single camera operatively connected to the processor, and a memory storing instructions that, when executed by the processor, cause the system to perform the following operations: capturing, by the camera, a plurality of images depicting a portion of a road; generating a road profile based at least in part on one or more of the plurality of images of the road; associating a confidence value with a point in the road profile; and transmitting, to a vehicle control system, data corresponding to the points in the road profile and the confidence values associated with the points in the road profile.

[0090] 48) In some embodiments, a computer-readable storage medium stores instructions that, when executed by a processor operatively coupled to a driver assistance system installed in a host vehicle, cause the system to perform the following operations: capturing, by a single camera operatively connected to the processor, a plurality of images depicting a portion of a road; generating a road profile based at least in part on one or more of the plurality of images of the road; associating a confidence value with a point in the road profile; and transmitting, to a vehicle control system, data corresponding to the points in the road profile and the confidence values associated with the points in the road profile. In some embodiments, the computer-readable storage medium is transient. In some embodiments, the computer-readable storage medium is non-transient.

[0091] 49) In some embodiments, a computerized road surface deviation detection method is performed by a driver assistance system installed in a vehicle, wherein the driver assistance system includes a camera operably connected to a processor and having a rolling shutter and radial distortion. In some embodiments, the method includes: capturing, by the camera, a plurality of images depicting a portion of a road; compensating for the effects of the rolling shutter and radial distortion; generating, at least based on the plurality of images depicting the portion of the road, a longitudinal profile of the portion of the road; and transmitting information about the longitudinal profile of the portion of the road to a vehicle control system.

[0092] 50) The method according to 49), wherein compensating for the effects of the rolling shutter includes:

[0093] projecting an image point onto a 3D point on the road,

[0094] adjusting a forward coordinate according to the speed of the vehicle, and

[0095] projecting the adjusted 3D point back into the image.

[0096] 51) In some embodiments, a driver assistance system is installed in a vehicle, the system including a processor, a camera having a rolling shutter and radial distortion, and a memory storing instructions, the camera being operably connected to the processor, the instructions, when executed by the processor, causing the system to perform the following operations: capturing, by the camera, a plurality of images depicting a portion of a road; compensating for the effects of the rolling shutter and radial distortion; generating, at least based on the plurality of images depicting the portion of the road, a longitudinal profile of the portion of the road; and transmitting information about the longitudinal profile of the portion of the road to a vehicle control system.

[0097] 52) In some embodiments, a non - transitory computer - readable storage medium stores instructions that, when executed by a processor operably coupled to a driver assistance system installed in a host vehicle, cause the system to perform the following operations: capturing, by a camera operably connected to the processor and having a rolling shutter and radial distortion, a plurality of images depicting a portion of a road; compensating for the effects of the rolling shutter and radial distortion; generating, at least based on the plurality of images depicting the portion of the road, a longitudinal profile of the portion of the road; and transmitting information about the longitudinal profile of the portion of the road to a vehicle control system.

[0098] The above - mentioned and / or other aspects will become apparent from the following detailed description when considered in conjunction with the accompanying drawings. Brief Description of the Drawings

[0099] The following disclosure is made with reference to the accompanying drawings, in which:

[0100] Figure 1 and Figure 2Illustrated is a system including a camera or image sensor installed in a vehicle according to some embodiments.

[0101] Figure 3A Illustrated are points covering a view of a road according to some embodiments, where these points can be tracked to calculate a homography matrix.

[0102] Figure 3B Illustrated is a normalized correlation score along a path of a vehicle according to some embodiments.

[0103] Figure 3C Illustrated is a path of a vehicle projected onto a road image according to some embodiments.

[0104] Figure 4A - Figure 4B Illustrated is an unstable road profile according to some embodiments.

[0105] Figure 5 Shown is a flowchart illustrating a method for correcting radial distortion and rolling shutter according to some embodiments.

[0106] Figure 6A - Figure 6D Shown is a flowchart illustrating a method for detecting a deviation of a road surface according to some embodiments.

[0107] Figure 7A - Figure 7B Illustrated is a stable road profile according to some embodiments.

[0108] Figure 8 Shown is a flowchart illustrating a method for creating a multi-frame road profile according to some embodiments.

[0109] Figure 9A - Figure 9B Illustrated is a stable road profile with fine-tuning according to some embodiments.

[0110] Figure 10A - Figure 10B Illustrated is a stable road profile with fine-tuning and adjustment for ridge detection according to some embodiments.

[0111] Figure 11A - Figure 11C Illustrated is a multi-frame road profile according to some embodiments.

[0112] Figure 12 Shown is a flowchart illustrating a method for generating data according to 1D sampling according to some embodiments.

[0113] Figure 13A - Figure 13C Illustrated is a result of 1D road profile sampling according to some embodiments.

[0114] Figure 14A Illustrated is a predicted path of a wheel covering a scene of a road according to some embodiments.

[0115] Figure 14B The predicted path of the wheel is illustrated, which shows the predicted vertical deviation of the road on the y-axis of each figure. Detailed description

[0116] Reference will now be made in detail to the features of the present disclosure, examples of which are illustrated in the accompanying drawings, wherein like reference numerals always refer to like elements. These features are described below to explain the technology disclosed herein with reference to the accompanying drawings.

[0117] Before explaining the features of the technology in detail, it should be understood that the technology taught herein does not limit its application to the details of the design and arrangement of components set forth in the following description or illustrated in the drawings. The technology taught herein is capable of incorporating other features or being practiced or carried out in various ways. Also, it should be understood that the language and terminology employed herein are for the purpose of description and should not be regarded as limiting.

[0118] In this document, lowercase coordinates (x, y) are used to represent image coordinates; and uppercase coordinates (X, Y, Z) are used to represent world coordinates using a right-handed coordinate system, where, relative to the vehicle-mounted camera, X is forward, Y is to the left, and Z is upward.

[0119] By way of introduction, various embodiments of the technology described herein are useful for accurately detecting road shapes such as the vertical cross-section of a road using a camera 12 mounted in a host vehicle 18 as described above. Figure 1 Using the systems and methods provided herein, ridges and / or holes such as speed bumps, curbs, and manhole covers can be accurately detected, where there is a vertical deviation of as little as one centimeter from the road plane. The systems and methods disclosed herein can be similarly applied to cameras 12 for forward, side, and rear views. The various methods described herein can accurately estimate a planar (or biquadratic) model of the road surface and then calculate small deviations from the planar (or biquadratic) model to detect ridges and holes.

[0120] Correction of radial distortion and rolling shutter

[0121] Now refer to Figure 5 , which shows a flowchart of a method 500 for correcting radial distortion and rolling shutter in a road image of an image frame 15 such as described above. Figure 2 In some embodiments, correction of both radial distortion and rolling shutter may be advantageous for accurately detecting distances, accurately measuring self-motion, and accurately generating a road longitudinal section without artificial upward curvature.

[0122] In some embodiments, at step 502, the coordinates of inlier points in each captured image can be first corrected for radial distortion using a standard model and default lens parameters.

[0123] In some embodiments, after the coordinates of the in-bounds value points are corrected, at step 504, each point in the image can be compensated for rolling shutter according to the following steps. First, at step 506, a reference line y is selected in the image t0 as t0. Then, at step 508, the time offset of the reference line is given for the y coordinate of the image according to the following equation:

[0124] δ t = (y – y t0 ) * t 线

[0125] Next, at step 510, assuming a default road model ([0, 0, 1] is beneficial for normal vehicle speeds and calibrated camera heights), the points are projected onto 3D points on the road. Then, at step 512, the X (forward) coordinate is adjusted by v * 8 t , where v is the vehicle speed. Finally, at step 514, the 3D points are projected back into the image.

[0126] Generation of a single-frame profile in a stable coordinate system

[0127] Now referring to Figure 6A - 6D , Figure 6A - 6D is a flowchart of method 600, which creates a homography matrix based on a consistent ground plane at each time step, and then this homography matrix can be used to align the frames and calculate the residual motion and the road profile. In some embodiments, the techniques according to method 600 can be superior to previously known techniques because method 600 can allow the analysis of multiple frames relative to the same reference frame, rather than allowing the analysis of each frame relative to a unique reference frame calculated uniquely according to that corresponding frame itself. As explained in further detail below, in some embodiments, method 600 enables a stable world coordinate system to be calculated based at least on a combination of the following: (a) a reference plane calculated according to the current frame, (b) reference planes calculated according to one or more previous frames, and (c) a default reference plane based at least in part on the assumption of a fixed position of the camera relative to the vehicle. In some embodiments, the combination of these factors can result in a stable world coordinate system that can be used to compare frames corresponding to different time steps in order to calculate the residual motion and identify vertical deviations of the road surface. In some embodiments, generating a profile in a stable road world coordinate system may be more desirable or beneficial for a receiving system (such as an OEM vehicle control system that controls the vehicle suspension); by generating the profile in a stable coordinate system, the information can be more easily applied by the control system without having to be transformed afterwards to the road coordinated by the control system.

[0128] Note that, in some embodiments, another type of model of the road surface may be used in a similar manner without using a homography matrix. For example, in some embodiments, the road surface may be modeled as a quadric surface with a certain curvature. For the sake of brevity, in the description of the exemplary method 600 herein, the present disclosure will relate to the use of a homography matrix.

[0129] As further explained in detail below, one aspect of method 600 may provide the creation of a homography matrix that is used to warp one image towards another image in order to calculate residual motion and detect vertical deviations of the road surface. In some embodiments, the creation of the homography matrix used in this warping process may include a combination of information related to a previous estimate of the plane normal, a current estimate of the plane normal, and an average estimate of distance and normal based on a historical time period (e.g., the previous 20 seconds) that the vehicle has driven through. In some embodiments, the reconstructed homography matrix as explained below may be understood to be based not only on the information collected in the images that the system uses when determining whether there are any vertical deviations in the road surface, but also on a predetermined speculation about the road plane and / or historical calculations about the road plane. In some embodiments, such a homography may be calculated and used to analyze frames to detect vertical deviations of the road surface according to the following steps.

[0130] In step 602, in some embodiments, an initial guess H0 of the homography matrix for the road is used to track points between two consecutive frames. Although the method has been described above with reference to Figure 5 explained the method, in some embodiments, this initial step 602 may be performed without correcting for radial distortion and rolling shutter. These are point correspondences between the pre-warped image I1w and the image I2.

[0131] In some embodiments, the points may be evenly placed in each image, which gives more points closer to the vehicle; or the points may be evenly spaced in distance, which requires more points concentrated mainly in the image (farther from the vehicle). In some embodiments, the points may be evenly vertically spaced in the image, but greater weight may be given to points that are farther away in the least squares calculation of the homography matrix. For example, when performing least squares (e.g., solving Ax = b), each row of A may be weighted equally, or certain rows may be given more weight (e.g., solving wAx = wb, where w is a diagonal matrix of weights).

[0132] In step 604, in some embodiments, the inverse matrix of H0 is used to obtain the point correspondences between images I1 and I2.

[0133] In step 606, in some embodiments, the coordinates of the points in each image are corrected for radial distortion and compensated for rolling shutter. In some embodiments, these corrections and compensations can be performed according to the techniques explained above with reference to Figure 5 as described.

[0134] In step 608, in some embodiments, the homography matrix H of the road is found using Random Sample Consensus (RANSAC) on the corrected points. RANSAC is an example of a robust algorithm that rejects outliers. Iteratively Reweighted Least Squares (IRLS) and other robust algorithms can also be used.

[0135] In step 610, in some embodiments, the same inlier / outlier classification can be used on the uncorrected points to calculate the homography matrix H that aligns the road in the distorted image d .

[0136] In step 612, in some embodiments, the homography matrix H can be decomposed into the self - motion (rotation R and translation T) between the frame and N 当前 and D 当前 (the plane normal and the distance to the plane, respectively). Note that R and T represent the vehicle motion and are thus unique, while the plane N and D are somewhat arbitrary and will therefore have significantly more noise.

[0137] In step 614, in some embodiments, the previous estimate of the plane (N 当前 and D 当前 ) is adjusted based on the self - motion (rotation R and translation T) and the new plane estimate of N 先前 , D 先前 . The adjustment values for N 新 and D 新 can be calculated according to the following equation

[0138] (N 新 , D 新 ) = 0.8 * update R,T (N 先前 , D 先前 ) + 0.1 * (N 当前 , D 当前 ) + 0.1 * history(N 当前 , D 当前 ) (1)

[0140] where the function history() is the average of the K last values of (N 当前 and D 当前 ), and the function update R,T is given by:

[0141] N 更新 = R * N先前 (2)

[0142] D 更新 D = T * N 更新 + D 先前 (3)

[0143] Note that the values 0.8, 0.1, and 0.1 in Equation (1) are merely exemplary and any other set of three values that sum to 1.0 may be used.

[0144] In step 616, in some embodiments, R and T are calculated by linking according to the following sub - steps. First, for each pair of adjacent frames between the current frame and the previous far frame, a matrix can be created according to the following: 远 and T 远 . First, for each pair of adjacent frames between the current frame and the previous far frame, a matrix can be created according to the following:

[0145]

[0146] Next, the matrix C i is multiplied to obtain C 远 :

[0147]

[0148] Then finally, the upper 3x3 matrix of the product C 远 is taken as R 远 , and the top three elements of the right column are taken as T 远 .

[0149] In step 618, in some embodiments, then a new homography matrix H 远 and T 远 is created based on the calculated values for R 远 ,

[0150]

[0151] where T represents the transpose, and where the prime represents the second image (e.g., 'T is in the coordinate system of the second image).

[0152] In step 620, in some embodiments, the linking process can be repeated using the homography matrix of the distorted image. The linked Rd 近 and Td 远 from the distorted matrix Hd 远 (e.g., according to a similar equation as shown above with reference to step 618) and N 更新 and D 更新 from the undistorted image can be used to construct the homography matrix Hd 远 .

[0153] In step 622, in some embodiments, based on information from the vehicle, such as information indicating the vehicle's speed, velocity, direction, acceleration, etc., a vehicle path is calculated in world coordinates (e.g., X, Y, Z); this information can be obtained from sources other than a camera-based navigation system, such as other sensors equipped on the vehicle that can determine its speed, velocity, direction, acceleration, and other state information.

[0154] In step 624, in some embodiments, the vehicle path is projected onto an undistorted image using stable plane parameters, and then the coordinates are distorted according to radial distortion and rolling shutter to obtain path coordinates (x, y) in the image. In some embodiments, due to radial distortion and rolling shutter, the path is projected onto the current image in two steps; first, the path is projected onto virtual undistorted image coordinates; then, the coordinates of the path are distorted to give distorted coordinates. Alternatively, in some embodiments, reasonably accurate results can be obtained simply by projecting the vehicle path onto the distorted image (especially when the lens has low radial distortion).

[0155] In step 626, in some embodiments, the homography matrix Hd 远 warp the strip from image I 远 towards I2, where I2 is the current image, and I 远 is the previous image. Note that in some embodiments, I 远 may not be the image immediately preceding I2. In some embodiments, a minimum distance that the vehicle may need to travel (e.g., about 0.7 m) is between I 远 and I2. For example, in a system running at 18 frames per second, if the vehicle is traveling at 45 KMH or more, I 远 can be the previous image; if the vehicle is traveling between 22.5 KMH and 45 KMH, I 远 can be the image before the previous image. The sub-pixel alignment of the rows in the two strips gives the residual motion (d y ) of the points along the vehicle path relative to the stable road plane. In some embodiments, the sub-pixel alignment can be performed according to the techniques discussed in U.S. Patent No. 9,256,791 titled "Road Vertical Contour Detection" filed on November 26, 2014, and U.S. Application No. 14 / 798,575 titled "Road Contour Vertical Detection" filed on July 14, 2015.

[0156] In step 628, in some embodiments, the inverse matrix of Hd 远 can be used to make the points (x, y + d y)Undistort back from the distorted image to obtain the distorted image Id 远 The matching points in

[0157] In step 630, the points (x, y) and the matching points are undistorted with respect to the undistorted image coordinates. As a result of step 628, there is a sub-pixel match between all the points (x, y) along the path of the distorted I2 and the corresponding points in Id 远 In order to obtain accurate numbers, these coordinates can be undistorted in both images (for radial distortion and rolling shutter), and use H 远 Utilize I 远 The coordinates of (instead of the image itself) to re-distort. In some embodiments, the process can include manipulating only two vectors of the image coordinates with approximately 200 points per path, rather than the actual image; thus, the process may be computationally inexpensive.

[0158] In step 632, in some embodiments, use H 远 To distort the matched undistorted points, and subtract the undistorted points (x, y) to obtain the true residual motion. For example, the y coordinate can be subtracted to obtain the undistorted d y .

[0159] In step 634, in some embodiments, the residual motion and the undistorted points (x, y) are used to calculate the distance and height of the longitudinal section of a single frame. In some embodiments, this calculation can be performed according to the techniques discussed in U.S. Patent No. 9,256,791 titled "Road Vertical Contour Detection" filed on November 26, 2014 and U.S. Application No. 14 / 798,575 titled "Road Contour Vertical Detection" filed on July 14, 2015. Figure 7A And Figure 7B Show exemplary results of the longitudinal section of a single frame generated using the stable plane according to the above method. Figure 7A Shows a wider part of the longitudinal section, while Figure 7B Shows an image magnification on the area before the ridge detected in the road. For Figure 7A And Figure 7B The exemplary longitudinal section, only the forward translation along the plane is used for alignment. During and after the ridge, the longitudinal section is stable. Note that within a forward travel of 90m, the vertical axis ranges from -0.15m to 0.15m. Note that the sampling range in height is reduced to 0.05 meters, which is significantly less than the range of the longitudinal section height shown in Figure 4A And Figure 4B .

[0160] Multi-frame profile generation in a stable coordinate system

[0161] In some embodiments, multiple consecutive image frames captured near the same time point may all include a single common portion of a road that is part of the image. In some embodiments, it may be advantageous to correlate the image data and / or profile data obtained from each respective frame and combine it with other images and / or profile data corresponding to the same portion of the road obtained from other frames for analysis. Accordingly, techniques are provided for combining road profiles corresponding to overlapping regions of a road. In some embodiments, the techniques described below with respect to method 800 may be used to align the respective profiles corresponding to the same portion of the road such that slight misalignments between the multiple profiles are not inaccurately interpreted as vertical deviations (e.g., ridges or depressions) in the contour of the road. When the same stable world coordinate system is used to compute and represent the profiles, it is significantly helpful to ensure that differences between the profiles do not inaccurately indicate vertical road contour deviations, and the corrections described below with respect to method 800 may further reduce inaccurate indications of vertical road contour deviations. In some embodiments, the techniques of method 600 and method 800 may be used separately, although when they are used together, results with minimal inaccurate road contour deviations may be produced.

[0162] Now refer to Figure 8 , which shows a flow chart of a method for aligning multiple frames (e.g., frames representing different moments in time as a vehicle traverses a portion of a road) with a global model to combine road profiles corresponding to overlapping portions of the road.

[0163] Let T z be the forward motion of a vehicle along a path. In some embodiments, since T z is much smaller than the length of the profile, there may be many measurements over time for the same point along the road. In other words, there are many consecutive frames captured by a camera in which the same portion of the road is visible in the respective frames. Accordingly, in some embodiments, it would be advantageous to derive a multi-frame model that uses all the measurements for the same points being combined. One approach is to obtain all the single-frame profiles and plot them by shifting forward translation T z . They will be fairly well aligned by this method, but there will be small vertical shifts and variations in the slope. Simply averaging the heights for each point along the path can smooth the ridges, so it is crucial to account for these shifts in height and slope. Accordingly, in some embodiments, it is desirable to align the current single-frame profile P 当前 with the global model using slope and offset values (a, b). In some embodiments, this alignment may be accomplished according to the following steps.

[0164] In step 802, in some embodiments, k can be selected as the number of the most recent longitudinal sections to be used for analysis. Then the current longitudinal section data P from 5m to 12m is selected 当前 and k adjusted previous single-frame longitudinal sections that fall in the same region P k in which all the P have been aligned according to forward motion in the same region k .

[0165] In step 804, stochastic gradient descent can be used to minimize the cost function E over the slope and offset values (a, b), where E includes a data term, a smoothness term, and a regularization term. In some embodiments, the data term D can be any norm, not just L_1. The data term D can also be a saturated norm, such as Sum(min(abs(P_ 当前 ), threshold)). In some embodiments, the smoothness term S can also be a norm or a saturated norm. In some embodiments, the regularization term R can be attributed to a penalty designed to avoid large slope and offset terms (such as terms including the squared or absolute values of a and b, respectively). In some embodiments, E can be given by the following equation:

[0166] E a,b = λ1D + λ2S + λ3R (7)

[0167] where the data term D considering the difference from zero is given by:

[0168] D = ∑(abs(P 当前 )) (8)

[0169] The smoothness term considering the difference between the adjusted current longitudinal section and the previously adjusted single-frame longitudinal section P k :

[0170]

[0171] And the regularization term is designed to avoid large slope and offset terms:

[0172] R = a 2 + b 2 (10)

[0173] Note that the offsets and slopes of P in the above equations have been shifted by (a, b). Each P 当前 has been shifted by appropriate forward motion and by the slope and pitch (a k , b k , b k)Adjusted previous single-frame measurement results. In some embodiments, stochastic gradient descent uses the gradient computed for a random point along the profile and the value of E(a,b) to adjust the values of (a,b). In some embodiments, a fixed sequence of points rather than truly random points can be used, which can avoid unpredictable random results with a large number of unfavorable outliers and may not be reproducible.

[0174] In some embodiments, each of the K latest profiles can be adjusted towards the current profile.

[0175] In step 806, in some embodiments, the optimal values of (a,b) can be applied to profile P 当前 , and the median can be computed between the adjusted P 当前 and all the adjusted profiles P k . For example, at each distance along the profile, the system can have measurements from P 当前 and can also have a number (e.g., 5) of previous measurement results P k . The system can compute the median of those 6 values (excluding invalid values in some embodiments). In the described embodiments, this gives a multi-frame profile for the current frame.

[0176] In Figure 9A , Figure 9B , Figure 10A - 10B and Figure 11A - 11C shown in the example, the median calculation is performed between the adjusted P 当前 and all the adjusted profiles P k over all 140 values from 5m to 12m at an interval of 0.05m. Figure 9A - 9B Shows a single-frame profile after fine-tuning of the A and B values (e.g., step 804). Figure 9A Shows the road profile formed using a stable plane after fine alignment, while Figure 9B shows an enlargement of the area of the road before the retarder ridge for the same data. Note that the sampling is more closely distributed in height, but the height of the ridge decreases. Comparing Figure 7A and Figure 7B with Figure 9A and Figure 9B : In Figure 7A and Figure 7B , the profile range in the flat section is about 0.05m; while in Figure 9A and Figure 9B , the range is about half of that. However, note that in Figure 7A and Figure 7B , the ridge profiles are all higher than about 0.08m, while in Figure 9A andFigure 9B Among them, some vertical sections are lower than 0.08 m. Figure 10A and 10B shows a single-frame vertical section (at two different zoom levels), where A and B are calculated considering the retarder ridge as explained below by ignoring the data items regarding the retarder ridge. In Figure 10A and Figure 10B In, while maintaining a range tighter than Figure 7A and Figure 7B in, this problem of pushing down the retarder ridge such as 9A and Figure 9B is alleviated. Figure 11A - Figure 11C shows multi-frame vertical sections generated according to method 800 as described above at various zoom levels.

[0177] 1D sampling

[0178] In some embodiments, transmitting data on the road vertical section of a portion of the road ahead of the vehicle may require transmitting a large number of values. For example, the road vertical section output of 5 meters to 12 meters ahead of the vehicle sampled every 0.05 meters requires 140 values; for longer vertical sections, the number of required values is obviously larger. The large number of values that may be required can be problematic for a Controller Area Network (CAN) and may require considerable computing resources at the receiving end. For example, in some embodiments, 200 data points can be transmitted per wheel per frame, which may require transmitting more than 1 KB of data per frame. In some embodiments, transmitting this amount of data per frame may be infeasible and / or computationally too expensive for the receiving system.

[0179] In addition, some DAS and self-driving systems need to receive data at a predetermined frame rate (such as 100 Hz), such that the system requires one data point corresponding to the height of the road (at a predetermined distance ahead of the vehicle) every 10 milliseconds. In some embodiments, sending data to the system every 10 milliseconds may be infeasible because it may monopolize the data channel in the vehicle, such that other information cannot be sent through the same data channel.

[0180] Therefore, there is a need for systems and methods for transmitting a computationally efficient amount of data regarding the road vertical section in such a way that: (a) the total amount of data sent is computationally manageable, and (b) data transmission does not monopolize the data channel at all time points.

[0181] In some embodiments, a data format is provided for sending information about a road profile, where the road height is output at a specific distance in front of the wheels. In some embodiments, the distance is a fixed distance (e.g., 7 meters), and in some other embodiments, the distance can be determined dynamically. For example, the distance in front of the wheels can be determined dynamically based on the speed of the vehicle, with the reference distance increasing at higher speeds. In some embodiments, the distance can be set based on the distance covered by the vehicle within a given amount of time (e.g., the distance covered by the vehicle in 0.5 seconds at its current speed). Sending only the data corresponding to a certain fixed distance in front of the wheels, rather than transmitting the entire profile or the profile data related to the road height along the entire known wheel path at each time step, allows for the transmission of less total data, which can save computational resources and bandwidth.

[0182] In some embodiments, a data format is also provided in which multiple data points are transmitted substantially simultaneously, rather than just one data point, to effectively multiply the frame rate. For example, if the frame rate is 10 Hz, in some embodiments, the system can simulate a 100 Hz data output by sending 10 data points at a time in a burst at a true frequency of 10 Hz. After transmission, the receiving component can split the 10 data points and query them one by one as needed successively. In some embodiments, all the data points corresponding to a single frame can be transmitted in a single data transfer, and in some embodiments, they can be transmitted in multiple transfers with a number less than the number of data points per frame (e.g., a 7 - data - point - per - frame is sent in only two CAN messages). In some embodiments, each burst can send fewer than 10 data points. In some embodiments, each burst can send 16 data points.

[0183] Now note Figure 12 , Figure 12 is a flowchart showing an exemplary method 1200 for generating data according to the 1D sampling technique discussed above.

[0184] In some embodiments, at step 1202, the system can select the distance in front of the wheels of the vehicle to which the data to be transmitted should correspond. The distance in front of the wheels can be predetermined by the vehicle control system to which the data is to be transmitted. In some embodiments, selecting the distance in front of the wheels can include taking into account system latency such that the system will receive the data when the corresponding data corresponds to the target distance in front of the wheels. For example, if the target distance is 7 meters, the system can select 71 centimeters as the distance in front of the wheels such that the system latency results in the system receiving the data when the corresponding data point is at 7 meters (rather than 71 centimeters) in front of the wheels.

[0185] In some embodiments, at step 1204, the system may select the number of data points to be sent per burst. For example, as discussed above, the system may send 10 data points per burst. In some embodiments, the system may send multiple data transmissions per burst, such as two CAN messages per frame. In some embodiments where multiple transmissions are sent per burst, the system may select the number of data points to be sent in each transmission within the data burst (which may be the same or different).

[0186] In some embodiments, at step 1206, the system may select the frequency at which to send data bursts. For example, the system may send a data burst (e.g., 10 data points) every 100 milliseconds. The frequency of sending data bursts may be determined by the bandwidth of the data channel in the vehicle, or may alternatively be pre-determined by the receiving vehicle control system.

[0187] In some embodiments, at step 1208, the system may determine the distance the host vehicle is expected to travel within the time corresponding to the frequency at which data is to be sent. For example, the system may determine the distance the host vehicle is expected to travel within 100 milliseconds. In some embodiments, the distance the vehicle is expected to travel may be determined by sampling the vehicle's current speed and / or acceleration (from existing image data and / or from other sensors and systems in the vehicle).

[0188] In some embodiments, at step 1210, the system may sample the road profile at n points along the projected path of the wheel, where n is the number of data points per burst. In some embodiments, the points at which the road profile is sampled may be spaced apart between a selected distance in front of the wheel and that distance plus the estimated distance the vehicle will travel before sending the next data burst (e.g., may span, or may be evenly spaced therebetween). In some embodiments, the road profile may be sampled at points along a path segment that extends beyond the selected distance in front of the wheel plus the estimated distance the vehicle will travel before sending the next data burst; in this way, there may be some overlap between road portions corresponding to data bursts from different frames, creating redundancy which, in some embodiments, may improve the accuracy of vertical deviation calculations. In some embodiments, one or more data points may be associated with corresponding confidence values; other techniques regarding confidence values will be discussed below.

[0189] In some embodiments, data points may be generated based on a stable reference plane. In some embodiments, data points may be generated at least in part by combining cross-sectional information from multiple frames.

[0190] In some embodiments, the vehicle control system can receive all data points substantially simultaneously in a single data burst and can split the data points as needed to read them individually during the period before the next data burst arrives. For example, the system can receive a data burst containing 10 data samples and can split the burst and read one sample every 10 milliseconds until a second data burst (with 10 additional data points) is received 100 milliseconds later. In this way, the data burst can simulate sending one data point every 10 milliseconds even though the data is only transmitted once every 100 milliseconds.

[0191] For example, if a data point corresponding to 7 meters in front of the wheel is needed, the system can estimate how far the vehicle will move every 10 milliseconds and can sample a known road profile with 10 data points, starting at the point 7 meters in front of the wheel and continuing the sampling. Using Matlab TM Notation:

[0192] Z = 7 + [0:0.1:0.9]*dZ / 10 (11)

[0193] where dZ is the distance traveled between frames (100 ms). Thus, in the case where the vehicle travels 5 centimeters in 10 milliseconds, the transmitted data points can correspond to 700 centimeters, 705 centimeters, 710 centimeters, 715 centimeters, etc., up to 745 centimeters. After 100 milliseconds, a second pulse of 10 additional data points can be sent, and at this time the vehicle will have moved forward approximately 0.5 meters. When the vehicle is shifted forward by approximately 0.5 meters, the next 10 data points will be approximately evenly spaced from the previous 10 data points in real-world coordinates, resulting in a continuous path of data points every 5 centimeters, and the vehicle's driver assistance software can query this continuous path every 10 milliseconds regarding the point 7 meters in front of the current vehicle. In this way, by sending a burst of data points once, the system can simulate sending one data point at a higher frame rate without continuously monopolizing the data channel in the vehicle.

[0194] In some embodiments, if 10 data points are transmitted according to the above method, but each group of 10 data points is sampled from a different road profile, there may be a small interruption each time the data points switch from one profile to the next. To avoid this problem, a smooth transition from one road profile to the next can be generated by sampling 20 data points between, for example, 7m and 2*dZ:

[0195] Z = 7 + [0:0.1:1.9]*dZ / 10 (12)

[0196] Then, the current top 10 measurement results are averaged with the previous second 10 measurement results, gradually increasing the weight of the current measurement results and decreasing the weight of the previous frame measurement results. For example:

[0197] w = [0; 0.1; 0.9] * 10 / 9; (13)

[0198] P 输出 = w * P 当前 (1:10) + (1 - w) * P 先前 (11:20) (14)

[0199] The results of the 1D distribution graphs according to the methods and examples explained above are shown in Figure 13A - Figure 13C . Figure 13A A 1D sample distribution graph is shown, where 10 samples from each frame are shown as segments. Figure 13A The gap at 50 meters in is due to being invalid on the frame. Figure 13B Shows an enlargement of the area before the detected deceleration ridge, and Figure 13C Shows an enlargement of the area centered on the deceleration ridge itself. Figure 14A and Figure 14B Show examples of the system results of the example explained above with reference to 1D sampling and Figure 13A - Figure 13C . Figure 14A Shows the predicted path and residual motion overlaid on the road image, where the residual motion d y Is graphically represented as the deflection of a curve in the x - direction, where a deflection to the right indicates a ridge; it can be seen that the calculated deflection starts at the beginning of the ridge, increases towards the middle of the ridge, and then decreases on the far side of the ridge. In the example shown, the deflection is scaled by 30 times to make small deflections visible graphically. Figure 14B Represents the same drawn line as shown in Figure 14A But is shown in metric coordinates in a cross - sectional view. Figure 14B The top two curves in show the current left and right distribution graphs from 5 meters to 20 meters sampled at 5 - centimeter intervals from 5 meters to 10 meters and at 10 - centimeter intervals from 10 meters to 20 meters; the y - axis is from - 0.2m to 0.2m, and the detected ridge is approximately 0.08m high. Figure 14B The bottom two curves in show the left 1D samples and right 1D samples, which show the cumulative distribution graphs from 7 meters in front of the vehicle to the vehicle itself, where the vertical line represents the point 7 meters in front of the wheel. The segments to the right of the vertical line are the parts calculated for the current frame: from 7m to (7 + dZ)m. The parts to the left of the vertical line are calculated in the previous frame. Over time, the ridge will move to the left, and when it reaches zero, the wheel actually touches the ridge, and the car can react.

[0200] In some embodiments, the interval of the 1D sampled data points can be determined according to alternative techniques. For example, in some embodiments, the data points can be spaced in such a way that the data points are separated by the distance the vehicle is expected to travel in a fixed time. For example, the vehicle control system can be configured to process one data point every 10 milliseconds regardless of the vehicle speed. In this case, the system can estimate the position where the vehicle is expected to be located (e.g., estimate the corresponding positions the vehicle will be at for each time point, with the time points spaced 10 milliseconds apart) at an interval of 10 milliseconds (e.g., according to the current speed of the vehicle), and can place data points corresponding to each of these estimated positions.

[0201] In an exemplary system, bursts are output at 18 bursts per second (every 55 milliseconds), and the data points are spaced to be sampled at every 10 milliseconds at the distance covered by the vehicle. Thus, a burst can include enough samples to cover the time between bursts (55 milliseconds), e.g., by outputting 7 samples covering 60 milliseconds, such that there is some overlap in the road portion covered by each burst.

[0202] In another exemplary system, each burst includes 10 data points, and the data points are spaced at a distance corresponding to 5.5 milliseconds. In this system, 10 samples cover 49.5 milliseconds, leaving 5.5 milliseconds until the second burst arrives 55 milliseconds after the first burst, so there is no data point overlap.

[0203] Calculating System Latency

[0204] In some embodiments, the DAS or the autonomous driving system may need data corresponding to points at a certain distance in front of the vehicle (e.g., 7 meters). However, in some embodiments, due to system latency, when processing the image data and the controller receiving the data, the vehicle has moved, and the data reflects points at a distance less than the desired distance in front of the vehicle. Therefore, systems and methods are needed that take into account system latency when processing image data and transmitting the road profile and other data to the vehicle controller.

[0205] In some embodiments, if the latency T L is known, the sampling distance of the data points can be adjusted as follows:

[0206] Z = 7 + [0: 0.1: 1.9] * dZ / 10 + v * T L (15)

[0207] where v is the vehicle speed.

[0208] Based on the above adjustment of the latency, the longitudinal profile can be sampled at a distance adjusted for the latency to give a 1D distribution map at the desired distance in front of the wheel, correcting for the time when the signal is received by the controller. In some embodiments, the value T L The latency calculated by the controller or the latency of any other component of the system can also be considered.

[0209] In some embodiments, the system latency may not be fixed and may thus be unknown to the road longitudinal profile estimation system. In some embodiments, if the data output includes a timestamp of the image, the vehicle controller can use the timestamp to estimate the latency from when the image was captured to when the data is being received by the controller. The controller can then correct for variations in the system latency accordingly. In some embodiments, it may be advantageous if the road longitudinal profile system oversamples the longitudinal profile and outputs data at a more densely spaced interval than is required by the receiving controller or system. The controller or system can then determine the actual latency and select the data to use. If a 2 millisecond interval is used instead of a 10 millisecond interval, the controller can correct to better than 1 millisecond (e.g., the error is less than half a millisecond). Since all the values selected by the controller are calculated by the road longitudinal profile system, it is easy to reproduce the results offline, e.g., in testing. In some embodiments, since the controller does not interpolate and the numbers output by the controller are the values calculated by the road longitudinal profile system, it is possible to understand where the value comes from.

[0210] Ignore data items regarding the retarder ridge

[0211] In some embodiments, using multi-frame alignment techniques such as those described above with reference to method 600 may tend to "push down" the height of the detected retarder ridge because the algorithm tends to seek to push the entire longitudinal profile (including the road in front of the retarder ridge and the retarder ridge itself) down to zero. To counteract this tendency, in some embodiments, systems and methods are provided that are capable of ignoring data items regarding values above a certain threshold height. In some embodiments, the threshold height can be predetermined or determined dynamically. For example, in an (a, b) optimization, the system can ignore data items for all values above 3.5 centimeters.

[0212] In some embodiments, the system can determine a potentially detected retarder ridge by fitting the longitudinal profile to a straight line. If the longitudinal profile does not fit a straight line (e.g., if the mean square error is large), then it is suspected that the longitudinal profile contains a retarder ridge.

[0213] Confidence value

[0214] In some embodiments, it is advantageous to calculate one or more confidence values associated with a road profile and / or a road image. In some embodiments, confidence values associated with a single frame or multiple frames may be calculated. In some embodiments, confidence values associated with corresponding points in an image or corresponding points in a road profile may be calculated.

[0215] Single-frame confidence

[0216] In some embodiments, the confidence value of a single frame may be calculated based on the following.

[0217] In some embodiments, the first step in calculating the confidence is to perform forward / backward verification using a correlation score (as shown in Figure 3B ). In Figure 3B , there are 13 columns, and each column represents the score for a given offset (-6 to 6) for that row. In Figure 3B , darker shading represents a higher correlation score, while lighter shading represents a lower correlation score. In some embodiments, the correlation score may be represented by different colors, such as red for a high correlation score, yellow for a low correlation score, and green for an even lower correlation score. The residual motion of a point from Image 2 to Image 1 should be equal in magnitude, and the residual motion from Image 1 to Image 2 should be opposite in sign. The latter can be determined by looking at the diagonal in the score matrix from Image 2 to Image 1. If the forward / backward motion does not match within 1 pixel, the confidence can be determined to be zero.

[0218] In some embodiments, if the forward / backward motion matches, the confidence can be determined to be the curvature of the score matrix. When the image texture is obvious, a higher confidence score can thus be determined. For example, in some embodiments, good texture can give a curvature score on the order of 0.25.

[0219] In some embodiments, a confidence value of zero may be assigned to regions near the ends of the strip.

[0220] In some embodiments, the determined confidence value can be used in a multi-frame analysis process, such as the process explained above with reference to Method 800. Two thresholds regarding confidence can be used: first, a high confidence for selecting points in AB optimization; and second, a much lower threshold can be used to determine which points are used for the median. For example, a curvature score higher than 0.1 can be used to determine which points can be used for stochastic descent optimization (e.g., step 804), while a curvature score of 0.01 can be used to determine whether a point should be considered in the median (e.g., step 806).

[0221] Multi-frame confidence

[0222] Multiple frames of confidence can be output from the system based on the differences between multiple height measurements and the height used in median calculation. In some embodiments, for each point along the path, samples between zero and five are used to calculate the median output based on the single-frame confidence of the points in the current frame and the previous four frames. Let N be the number of samples above the threshold. The multi-frame confidence can be calculated as follows:

[0223] In some embodiments, if N = 0, the multi-frame confidence can be determined to be zero. In some embodiments, if N = 1, the multi-frame confidence can be determined to be zero.

[0224] In some embodiments, if N ≥ 2, v can be calculated as follows:

[0225]

[0226] where h is the median height, and h i is the height of each sample in step 806. Note that squaring (N - 1) 2 penalizes small N.

[0227] In some embodiments, the confidence C is given by:

[0228] C = 5 * (1 - S * v) (17)

[0229] which yields a confidence value between zero and 5. The ratio S can be determined empirically. In some embodiments, S can be set equal to 100. In some embodiments, the confidence values of multiple points in the image can be indicated by the saturation of the color of the respective points in the image.

[0230] Ignore shadows from moving vehicles and objects

[0231] In some embodiments, a camera-based DAS and / or an autonomous driving system can detect shadows in the captured images. For example, on a bright day, typically, a moving vehicle or other objects such as bicyclists and pedestrians in an adjacent lane or on a sidewalk cast moving shadows onto the road near the host vehicle. When such a shadow moves on the road, it produces residual motion and may be inaccurately interpreted as a fixed ridge or depression in the road surface and cause an undesired controller response.

[0232] It should be noted that the shadows from moving objects typically have a significant amount of residual motion and may thus be detected as very high ridges in the road. For example, the shadow of an oncoming vehicle moving at the same absolute speed as the host vehicle can appear as a ridge half the height of the camera without correction. In some embodiments, these shadows are not well tracked and will have a low confidence or low multi-frame confidence value. In these cases, the system simply ignores the shadows. However, based on the confidence value alone, moving shadows may not always be successfully ignored. Thus, there is a need for systems and methods that can distinguish moving shadows from fixed ridges and depressions in the road, including by actively filtering out data determined to correspond to moving shadows.

[0233] In some embodiments, if a moving object is detected by the DAS and / or other components of the self-driving system (e.g., camera-based vehicle detection, radar-based object detection), this information can be used to filter out the shadow by reducing the confidence value associated with the residual motion corresponding to the motion in the road plane at the speed of the detected object.

[0234] For example, consider a host vehicle traveling at 20 m / s and a target vehicle detected in an adjacent lane with a relative speed of -38 m / s. Simple subtraction will determine that the target vehicle is moving at -18 m / s relative to the stationary road. For example, the homography matrix in Equation (6) (Equation number 6) as explained in step 618 of reference method 600 above can be used to estimate the expected motion of points on the road moving at that speed. In some embodiments, the homography matrix can be decomposed, and the motion vector T 远 can be adjusted to account for the target vehicle motion during the time period between the two frames used to create the decomposed homography matrix, and a new homography matrix can be reconstructed. Then, the expected residual motion of the points can be calculated as the difference between the warping according to the initial H 远 and the reconstructed matrix based on the adjusted T 远 The term "warping" as used herein can refer to a transformation from image space to image space. In some embodiments, the confidence value assigned to the points along the path having a measured residual motion close to the suspect residual motion can subsequently be reduced. In some embodiments, if many points along the path have a residual motion close to the suspect residual motion, the entire frame can be invalidated.

[0235] Techniques for detecting shadows of moving objects are further discussed in U.S. Patent Application 14 / 737,522, filed on June 12, 2015, entitled "HAZARD DETECTION FROM A CAMERA IN A SCENE WITH MOVING SHADOWS". In some embodiments, the techniques disclosed in that application can be incorporated into the techniques disclosed herein to more accurately calculate and output a road profile, including detecting small vertical deviations detected in the road surface in a similar manner so that larger vertical deviations due to road hazards can be detected.

[0236] Ignore moving shadows from trees

[0237] Just as camera-based DAS and autonomous driving systems can detect shadows of moving objects such as vehicles and pedestrians (as just described above), camera-based DAS and autonomous driving systems can also detect shadows cast by tree branches. Typically, the wind causes these shadows to move slowly across the road surface. In some embodiments, these small movements are not detected in the same way as the movements attributed to large objects; rather, these small movements may appear as residual movements of small deviations in the road profile. Therefore, systems and methods are needed that can distinguish slowly moving tree branch shadows (and the like) from small vertical deviations in the road surface, including by actively filtering out data determined to correspond to shadows of trees.

[0238] In some embodiments, the movement of a shadow can be determined to be attributable to a tree branch and can be ignored based at least in part on the following three criteria:

[0239] First, in some embodiments, a swaying tree branch can be detected as jumping back and forth to produce shadow movement that is inconsistent over time. Thus, the resulting height profile of points along a path affected by the shadow of a swaying tree branch can be detected as changing over time; in some embodiments, such points can change sign and thus can be detected as a ridge at one moment and then as a hole at the next moment. In some embodiments, data for a given point can be collected over a period of time, and any difference in the detected height that exceeds a threshold difference can be determined to correspond to a moving shadow. For example, data can be detected over a period of time such as half a second, and any difference in a single point that exceeds a predetermined threshold within that half-second period can be used to indicate a moving shadow attributable to a swaying tree branch. Thus, the confidence value for that point can be reduced.

[0240] Second, in some embodiments, the motion from a swaying branch may not be towards the expansion focus. In some embodiments, this gives low confidence to the tracking that enforces the constraint. In some embodiments, to verify that what appears to be a ridge is not due to a moving shadow, 2D tracking can be performed, and then the motion not towards the expansion focus can reduce the confidence. Alternatively or additionally, in some embodiments, a score can be calculated (e.g., as shown in Figure 3B ), but with a strip of lateral shift plus or minus a few pixels; if the score is high, it indicates that the motion is not towards the expansion focus.

[0241] Third, in some embodiments, the shadow edges from the branches may always be flexible. This may be due to the finite size of the sun. The shadow from the branch may fall at the edge at a distance of about 1 / 100 of the distance from the object to the road surface along the path of the sunlight. Thus, learning techniques such as neural networks can be used to detect the atypical texture of the shadow, or the atypical texture can be explicitly detected by measuring the shadow gradient. In some embodiments, the detection of a shadow with a flexible edge can reduce the confidence value of the corresponding point.

[0242] In some embodiments, compared with examples of patches that are not tree shadows, machine learning such as a deep neural network (DNN) can be used to train the system with examples of patches of tree shadows. In some embodiments, the DNN can be fed in two strips I2 and warped I 远 so that it can not only use the texture and motion of the image, but also implicitly use the shape of the surface. Improved plane detection using image-based free space analysis

[0243] Various techniques can analyze the road scene to determine which parts of the image correspond to the road and which parts of the image do not correspond to the road (e.g., which parts correspond to other things such as cars, pedestrians, curbs, hedges, obstacles, etc.). In some embodiments, such techniques specifically detect the presence of obstacles near the vehicle. For example, if another vehicle is detected, it can be determined that the area of the image corresponding to the other vehicle and some areas around it do not correspond to the road.

[0244] In some embodiments, the system can attempt to determine whether a pixel corresponds to the road; this can be done based on the texture, color, and position of the pixel in the image, and can be performed by a neural network trained with examples of which parts of the image represent and do not represent the road.

[0245] In some embodiments, the regions of an image determined to correspond to a road surface with no obstacles may be referred to as "free space". In some embodiments of algorithms for determining vertical deviations of the road surface and / or for determining a road profile, information about which portions of the image correspond to free space can be used to improve the algorithms.

[0246] In some embodiments, when determining inlier points of a road model, points located outside of the free space region may be ignored. For example, such points can simply be considered as outliers, or they can also be eliminated from a number of potential inlier points (so, if 13 points are outside and the remaining 20 inlier values are all inliers, the confidence may be quite high). In some embodiments, grid points can be reallocated in the image such that all points fall within the free space. In some embodiments, when calculating a road profile, points along the path outside of the free space can be given low confidence.

[0247] Although certain embodiments of the present disclosure are presented in the context of DAS applications, some embodiments may equally apply to other real-time signal processing applications and / or digital processing applications, by way of example, such as communications, machine vision, audio, and / or speech processing.

[0248] As used herein, the indefinite articles "a" and "an" (such as "an image") have the meaning of "one or more" (such as "one or more images").

[0249] Although the selected features have been shown and described, it should be understood that the invention is not limited to the described features. On the contrary, it should be recognized that changes can be made to these features without departing from the principles and spirit of the invention, and the scope of the invention is defined by the claims and their equivalents.

Claims

1. A method for detecting road surface deviation, which is performed by a driver assistance system installed in a vehicle, Among them, The driver assistance system includes: - A processor, - A camera operably connected to the processor, and - A memory storing instructions executable by the processor; and The method includes: - Capturing a plurality of images through the camera, the plurality of images including a first image of the road surface captured at a first time and a second image of the road surface captured at a second time, the second time being after the first time, - Determining, by the processor, a current estimate of the planar normal of the road surface based at least on the first image and the second image, - Creating, by the processor, a model of the road surface based on a combination of: (i) A previous estimate of the planar normal of the road surface updated based on the self-motion of the vehicle; and (ii) The current estimate of the planar normal of the road surface based on the first image and the second image; - Determining, by the processor, a residual motion along the projection path of the vehicle using the model of the road surface, - Calculating, by the processor, the vertical deviation of the road surface based on the residual motion, and - Transmitting, by the processor, the vertical deviation data to a vehicle control system for controlling the movement of the vehicle.

2. The method according to claim 1, wherein, The model of the road surface is a homography matrix.

3. The method according to claim 1, wherein, Determining the current estimate of the planar normal of the road surface includes: Calculating an initial homography matrix based at least on the first image of the road and the second image of the road; and Decomposing the initial homography matrix to determine the self-motion of the vehicle and the current estimate of the planar normal of the road surface.

4. The method according to claim 3, wherein, Calculating the initial homography matrix includes tracking points between the first image and the second image, where the points are distributed in the images to correspond to positions evenly spaced apart from each other on the road surface.

5. The method according to claim 3, wherein, Calculating the initial homography matrix includes tracking points between the first image and the second image, where variable weights are assigned to the points according to the positions of the points in the images.

6. The method according to claim 1, wherein The vehicle control system is configured to adjust the suspension system of the vehicle in response to the calculated vertical deviation in the road surface.

7. The method according to claim 1, wherein The model of the road surface is at least based on a historical average of the planar normals.

8. The method according to claim 7, wherein, The updated previous estimate of the planar normal is weighted more heavily in the model of the road surface than the historical average of the planar normals.

9. The method according to claim 1, wherein, The updated previous estimate of the planar normal is weighted more heavily in the model of the road surface than the current estimate of the planar normal.

10. The method according to claim 1, further comprising: Calculating the vehicle path in world coordinates; And Projecting the vehicle path onto the second image using stable plane parameters determined according to the model of the road surface.

11. The method according to claim 1, further comprising: Correcting the coordinates of points in the plurality of images for at least one of radial distortion and rolling shutter.

12. The method according to claim 11, wherein, After computing the model of the road surface, perform the correction for at least one of radial distortion and rolling shutter.

13. A driver assistance system installed in a vehicle, the driver assistance system comprising: a processor; a camera operably connected to the processor; and a memory storing instructions which, when executed by the processor, cause the driver assistance system to perform the following operations: - capture a plurality of images via the camera, the plurality of images including a first image of a road surface captured at a first time and a second image of the road surface captured at a second time, the second time being after the first time, - determine, by the processor, a current estimate of the planar normal of the road based on at least the first image and the second image, - create, by the processor, a model of the road surface based on a combination of: (i) a previous estimate of the planar normal of the road surface updated based on the ego-motion of the vehicle; and (ii) the current estimate of the planar normal of the road surface based on the first image and the second image; - determine, by the processor, a residual motion along a projected path of the vehicle using the model of the road surface, - calculate, by the processor, a vertical deviation of the road surface based on the residual motion, and - transmit, by the processor, vertical deviation data to a vehicle control system for controlling the motion of the vehicle.

14. A non-transitory computer-readable storage medium storing instructions which, when executed by a processor operably coupled to a driver assistance system installed in a host vehicle, cause the driver assistance system to perform the following operations: capture a plurality of images via a camera operably connected to the processor, the plurality of images including a first image of a road surface captured at a first time and a second image of the road surface captured at a second time, the second time being after the first time; determine, by the processor, a current estimate of the planar normal of the road based on at least the first image and the second image; create, by the processor, a model of the road surface based on a combination of: (i) a previous estimate of the planar normal of the road surface updated based on the ego-motion of the vehicle; and (ii) the current estimate of the planar normal of the road surface based on the first image and the second image; determine, by the processor, a residual motion along a projected path of the vehicle using the model of the road surface; calculate, by the processor, a vertical deviation of the road surface based on the residual motion; and transmit, by the processor, vertical deviation data to a vehicle control system for controlling the motion of the vehicle.

15. A method for detecting a road surface deviation, the method being performed by a driver assistance system installed in a vehicle, Among them, the driver assistance system comprising: a processor, a camera operably connected to the processor, and a memory storing instructions executable by the processor; and the method comprising: - The processor selects at least two road profiles, where each of the at least two road profiles includes a common portion of the road, and where each of the at least two road profiles includes one or more height values associated with the road; The processor optimizes a function including a data term, a smoothness term, and a regularization term to align a first profile from the at least two road profiles; and The processor combines the at least two road profiles into a multi-frame profile.

16. The method according to claim 15, wherein, Aligning the first profile from the at least two road profiles includes: Based on the optimization, determining an optimized slope value and an optimized offset value; and Using the optimized slope value and the optimized offset value to modify the first profile from the at least two road profiles.

17. The method according to claim 15, wherein, The at least two road profiles are generated using a stable world coordinate system.

18. The method according to claim 15, wherein, Combining the at least two road profiles includes calculating the median of the road profiles.

19. A driver assistance system installed in a vehicle, the driver assistance system including: A processor; A camera operably connected to the processor; And A memory storing instructions that, when executed by the processor, cause the driver assistance system to perform the following operations: - The processor selects at least two road profiles, where each of the at least two road profiles includes a common portion of the road, and where each of the at least two road profiles includes one or more height values associated with the road, - The processor optimizes a function including a data term, a smoothness term, and a regularization term to align a first profile from the at least two road profiles, and - The processor combines the at least two road profiles into a multi-frame profile.

20. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor operably coupled to a driver assistance system installed in a host vehicle, cause the driver assistance system to perform the following operations: At least two road vertical profiles are selected by the processor, wherein, Each of the at least two road profiles includes a common portion of the road, and where each of the at least two road profiles includes one or more height values associated with the road; The processor optimizes a function including a data term, a smoothness term, and a regularization term to align a first profile from the at least two road profiles; and The processor combines the at least two road profiles into a multi-frame profile.

Images