Road vertical profile detection using a stable coordinate system

Abstract

The present application relates to road vertical profile detection using a stable coordinate system. In some embodiments, a first homography matrix created from two images of a road is decomposed to determine ego motion, and the ego motion is used to adjust a previous estimate of a road plane. The adjusted previous estimate of the road plane is combined with the current plane estimate to create a second homography matrix, and the second homography matrix is ​​used to determine residual motion and vertical deviation in the road surface. In some embodiments, multiple road profiles, each corresponding to a common portion of a road, are adjusted in slope and offset by optimizing a function having a data term, a smoothness term, and a regularization term; and the adjusted road profiles are combined into a multi-frame road profile. In some embodiments, road profile information of a predetermined number of data points is transmitted in periodic data bursts, wherein each data burst has more than one data point.

Description

[0001] This application is a divisional application of an application with an application date of February 28, 2016, application number 201680017324.3, and invention name “Road vertical profile detection using a stable coordinate system”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to the following applications: U.S. Provisional Patent Application 62 / 120,929, entitled “Road Plane Profile Output in a Stabilized World Coordinate Frame,” filed on February 26, 2015; U.S. Provisional Patent Application 62 / 131,374, entitled “Road Plane Output in aStabilized World Coordinate Frame,” filed on March 11, 2015; U.S. Provisional Patent Application 62 / 149,699, entitled “Road Plane Profile Output in a Stabilized World Coordinate Frame,” filed on April 20, 2015; and U.S. Provisional Patent Application 62 / 238,980, entitled “Road Plane Output in a StabilizedWorld Coordinate Frame,” filed on October 8, 2015; the disclosures of all applications are incorporated herein by reference.

[0004] Technically related applications

[0005] This application is related to U.S. application No. 14 / 554,500 (now U.S. Patent No. 9,256,791), entitled “Road Vertical Contour Detection,” filed on November 26, 2014; and U.S. application No. 14 / 798,575, entitled “Road Contour Vertical Detection,” filed on July 14, 2015, both of which are incorporated herein by reference in their entirety. background Technical Field

[0007] The present application relates to a driver assistance system and method for detecting a vertical deviation of a road profile using a camera, and more particularly to a driver assistance system and method for detecting a vertical deviation of a road profile using a stable coordinate system.

[0008] Description of Related Technology

[0009] In recent years, camera-based driver assistance systems (DAS) have entered the market, including lane departure warning (LDW), automatic high beam control (AHC), traffic sign recognition (TSR), forward collision warning (FCW) and pedestrian detection.

[0010] Reference now Figure 1 and Figure 2 , which shows a system 16 according to some embodiments, which includes a camera or image sensor 12 installed in a vehicle 18. The image sensor 12, which images a field of view in a forward direction, provides image frames 15 in real time, and the image frames 15 are captured by an image processor 30. The processor 30 can be used to process the image frames 15 simultaneously and / or in parallel for use by multiple DAS / applications. The processor 30 can be used to process the image frames 15 to detect and identify images or portions of images in the forward field of view of the camera 12. The DAS can be implemented using specific hardware circuits (not shown) with on-board software and / or software control algorithms in memory 13. The image sensor 12 can be monochrome or black and white, that is, without color separation, or the image sensor 12 can be color sensitive. By Figure 2 , according to some embodiments, the image frame 15 is used for pedestrian detection 20, TSR 21, FCW 22, and real-time detection 23 of the vertical profile of the road or deviation from the road plane.

[0011] In some embodiments, more than one camera may be mounted in a vehicle. For example, a system may have multiple cameras pointing in different directions. A system may also have multiple cameras pointing in the same direction or similar directions relative to the vehicle, but mounted at different locations. In some embodiments, a system may have multiple cameras with partially or completely overlapping fields of view. In some embodiments, two side-by-side cameras may operate in stereo. The non-limiting examples discussed herein consider single camera systems, but they may be similarly implemented in multi-camera systems, where some or all of the related images and frames may be captured by different cameras, or may be created by a synthesis of images captured from multiple cameras.

[0012] In some cases, the image frames 15 are divided between different driver assistance applications, and in other cases, the image frames 15 may be shared between different driver assistance applications.

[0013] Some prior art methods are known that use on-board cameras to detect vertical deviations from road profiles.

[0014] Some previously known algorithms can be summarized as follows:

[0015] 1. Align the first pair of two consecutive frames captured by the onboard camera using the homography matrix of the road. This gives the ego-motion (rotation R and translation T) between the frames. Figure 3A Shown are the 33 grid points in the image that are tracked and used to compute the homography matrix.

[0016] 2. A second pair of frames is then selected, the current frame and the nearest previous frame representing the time point from which the vehicle began its current movement over a minimum threshold distance. The concatenation of the first pair of frames (consecutive frames) is used to create an initial guess of the homography matrix, and then a more accurate homography matrix is ​​calculated for the second pair of frames and the reference plane is determined.

[0017] 3. Then project the vehicle's path onto the image plane (e.g. Figure 3C The strips along each path are used to calculate the residual motion, which gives a distribution map relative to a determined reference plane. Figure 3B The normalized correlation scores for a 31 pixel wide strip along the path of the left wheel for vertical motion ±6 pixels are shown. A small bend can be seen indicated by the arrow. This bend represents a small residual motion indicating a speed bump.

[0018] 4. Finally, the residual motion is converted to metric distance and height and combined into a multi-frame model. The results are shown in Figure 3C In the above picture.

[0019] One previously known algorithm according to the brief description above is described in more detail in US Pat. No. 9,256,791.

[0020] exist Figure 4A and Figure 4B An example of data generated according to the previous method is shown in . Figure 4A and Figure 4B A road profile from a series of frames aligned using the restored rotation (R), translation (T) and plane normal (N) is shown. A ridge of slowing speed can be seen near 40 meters, but also a strong divergence of the signal due to incorrect motion and plane normals; sampling shows a divergence of about 0.1 meters in height between profiles.

[0021] Brief Overview

[0022] One disadvantage of known methods such as the one described above is that a different reference plane is used at each time step. The reference plane used at each time step is relative to the true ground plane of the road calculated based on the frames at each corresponding step. By recalculating this ground plane at each step, there may be slight deviations between the reference ground planes in each step, making it cumbersome to combine information from multiple frames at different time steps. For example, when there are slow-speed ridges on the road, particularly when slow-speed ridges or ridges occupy a large portion of one or more frames, then the main plane may combine portions of the ridge and portions of the road, resulting in inaccurate calculations of vertical deviations from the road. In addition, small deviations in the plane of the road may cause new measurements to be poorly aligned. In theory, the ego-motion and plane information derived from the homography matrix can align new measurements; however, noise in the plane normals and ego-motion (particularly translation) does not facilitate accurate alignment. These problems can be seen in Figure 2. Fig. 7A and Figure 7B The example results shown in FIG. 1 and described in more detail below.

[0023] Another disadvantage of some prior methods is that they fail to account for lens distortion (e.g., radial distortion) and rolling shutter effects. If the effects of lens distortion and rolling shutter are ignored, prior known methods may produce a qualitatively correct single result with a vertical deviation from the identified road plane, but the distance may be inaccurate, the ego-motion may be inaccurate, and the resulting profile may have a small upward curvature.

[0024] Therefore, there is a need for systems and methods for calculating vertical deviations of road surfaces using a consistent ground plane. As explained herein, an improved method can create a homography matrix based on a consistent ground plane at each time step, and then use the homography matrix to align frames and calculate residual motion and road profiles.

[0025] As used herein, the term "homography" refers to a reversible transformation from a projected space to itself that maps straight lines to straight lines. In the field of computer vision, two images of spatially identical planar surfaces are related by a homography matrix that has the characteristics of a pinhole camera model. In some embodiments of the present disclosure that reference homography matrices, alternative models of road surfaces may also be used in a similar manner; for example, in some embodiments, a road may be modeled as a quadratic surface with a certain curvature.

[0026] In particular, by way of example, for a given camera 12 of height (1.25 m), focal length (950 pixels) and vehicle motion between frames (1.58 m), the motion of points on the road plane between two image frames 15 a and 15 b can be predicted, respectively. Using a model of an almost flat surface for the motion of road points, the second image 15 b can be warped towards the first image 15 a. The following Matlab TM The code will perform the initial warping step 501:

[0027]

[0028] Where dZ is the forward motion of the vehicle 18, H is the height of the camera 12, and f is the focal length of the camera 12. P0=(x0;y0) is the vanishing point of the road structure. Alternatively, initial calibration values ​​may be used during installation of the system 16 in the vehicle 18, where x0 is the forward direction of the vehicle 18 and y0 is the horizontal line when the vehicle 18 is on a horizontal surface. The variable S is an overall scaling factor associated with the image coordinates between two image frames 15a and 15b captured at different vehicle distances Z from the camera 12. The term "relative scaling change" as used herein refers to an overall scaling change of the image coordinates that depends on the distance Z to the camera 12.

[0029] Additionally, systems and methods are needed that account for the effects of radial distortion and rolling shutter in order to accurately detect distances, accurately measure ego-motion, and accurately generate road profiles without artificial upward curvature. In some embodiments, such systems and methods may account for radial distortion and rolling shutter, but may use the original image for point tracking (without adjustments for rolling shutter and radial distortion) in order to avoid computationally expensive image pre-warping.

[0030] Various driver assistance systems and computerized methods that can be installed in a host vehicle are provided herein for detecting vertical deviations of a road surface. The method can be performed by a DAS installed in the host vehicle while the host vehicle is moving. The DAS can include a camera operably connected to a processor.

[0031] 1) In some embodiments, a computerized road surface deviation detection method is performed by a driver assistance system installed in a vehicle, wherein the driver assistance system includes a camera operably connected to a processor. In some embodiments, the method includes: capturing a plurality of images by the camera, the plurality of images including a first image of the road surface captured at a first time and a second image of the road surface captured at a second time, the second time being after the first time; determining a first estimate of a plane normal of the road and a second estimate of the plane normal of the road based on at least the first image and the second image; creating a model of the road surface based on at least the first estimate of the plane normal and the second estimate of the plane normal; using the model of the road surface to determine a residual motion along a projected path of the vehicle; calculating a vertical deviation of the road surface based on the residual motion; and transmitting vertical deviation data to a vehicle control system.

[0032] 2) The method as described in 1), wherein the model of the road surface is a homography matrix.

[0033] 3) The method of 1), wherein determining the first estimate of the plane normal of the road and the second estimate of the plane normal of the road comprises:

[0034] calculating an initial homography matrix based at least on the first image of the road and the second image of the road,

[0035] decomposing the initial homography matrix to determine the second estimate of ego-motion and a plane normal of the road, and

[0036] A previous estimate of the plane normal is adjusted to generate the first estimate of the plane normal, the adjustment being based at least on the determined ego-motion.

[0037] 4) A method as described in 3), wherein calculating the initial homography matrix includes tracking points between the first image and the second image, wherein the points are distributed in the first image or the second image to correspond to positions evenly spaced from each other on the road.

[0038] 5) A method as described in 3), wherein calculating the initial homography matrix includes tracking points between the first image and the second image, wherein variable weights are assigned to the points according to their positions in the first image or the second image.

[0039] 6) The method of 1) wherein the vehicle control system is configured to adjust a suspension system of the vehicle in response to the calculated vertical deviation in the road surface.

[0040] 7) The method of 1), wherein the model of the road surface is based at least on a historical average of the plane normals.

[0041] 8) The method of 7) wherein the first estimate of the plane normal is weighted more heavily in the model of the road surface than a historical average of the plane normal.

[0042] 9) The method of 1), wherein the first estimate of the plane normal is weighted more heavily in the model of the road surface than the second estimate of the plane normal.

[0043] 10) The method as described in 1), comprising:

[0044] Compute the vehicle path in world coordinates, and

[0045] The vehicle path is projected onto the second image using stable plane parameters determined from a model of the road surface.

[0046] 11) The method as described in 1), comprising:

[0047] Coordinates of points in the plurality of images are corrected for at least one of radial distortion and rolling shutter.

[0048] 12) The method of 11), wherein the correction is performed for at least one of radial distortion and rolling shutter after calculating the model of the road surface.

[0049] 13) In some embodiments, a driver assistance system is installed in a vehicle, the system including a processor, a camera operably connected to the processor, and a memory storing instructions, which when executed by the processor causes the system to perform the following operations: capture multiple images through the camera, the multiple images including a first image of a road surface captured at a first time and a second image of the road surface captured at a second time, the second time being after the first time; determine a first estimate of a plane normal of the road and a second estimate of the plane normal of the road based on at least the first image and the second image; create a model of the road surface based on at least the first estimate of the plane normal and the second estimate of the plane normal; use the model of the road surface to determine a residual motion along a projected path of the vehicle; calculate a vertical deviation of the road surface based on the residual motion; and transmit the vertical deviation data to a vehicle control system.

[0050] 14) In some embodiments, a computer-readable storage medium stores instructions that, when executed by a processor operably coupled to a driver assistance system installed in a host vehicle, cause the system to perform the following operations: capture a plurality of images with a camera operably connected to the processor, the plurality of images comprising a first image of a road surface captured at a first time and a second image of the road surface captured at a second time, the second time being subsequent to the first time; determine a first estimate of a plane normal of the road and a second estimate of the plane normal of the road based on at least the first image and the second image; create a model of the road surface based on at least the first estimate of the plane normal and the second estimate of the plane normal; determine a residual motion along a projected path of the vehicle using the model of the road surface; calculate a vertical deviation of the road surface based on the residual motion; and transmit the vertical deviation data to a vehicle control system. In some embodiments, the computer-readable storage medium is transient. In some embodiments, the computer-readable storage medium is non-transitory.

[0051] 15) In some embodiments, a computerized road surface deviation detection method is performed by a driver assistance system installed in a vehicle, wherein the driver assistance system includes a camera operably connected to a processor. In some embodiments, the method includes: selecting at least two road profiles, wherein each of the at least two road profiles includes a common portion of the road; optimizing a function including a data term, a smoothness term, and a regularization term to align a first profile from the at least two profiles; and combining the at least two road profiles into a multi-frame profile.

[0052] 16) The method of 15), wherein aligning the first longitudinal section from the at least two longitudinal sections comprises:

[0053] Based on the optimization, determining an optimized slope value and an optimized offset value,

[0054] The first profile from the at least two road profiles is modified using the optimized slope value and the optimized offset value.

[0055] 17) The method as described in 15), wherein a stable world coordinate system is used to generate the at least two road longitudinal sections.

[0056] 18) The method of 15), wherein combining the at least two road longitudinal sections comprises calculating a median of the longitudinal sections.

[0057] 19) In some embodiments, a driver assistance system installed in a vehicle includes a processor, a camera operably connected to the processor, and a memory storing instructions that, when executed by the processor, cause the system to perform the following operations: select at least two road longitudinal sections, wherein each of the at least two road longitudinal sections includes a common portion of the road; optimize a function including a data term, a smoothness term, and a regularization term to align a first longitudinal section from the at least two longitudinal sections; and combine the at least two road longitudinal sections into a multi-frame longitudinal section.

[0058] 20) In some embodiments, a computer-readable storage medium stores instructions that, when executed by a processor operably coupled to a driver assistance system installed in a host vehicle, cause the system to perform the following operations: select at least two road profiles, wherein each of the at least two road profiles includes a common portion of a road; optimize a function including a data term, a smoothness term, and a regularization term to align a first profile from the at least two profiles; and combine the at least two road profiles into a multi-frame profile. In some embodiments, the computer-readable storage medium is transient. In some embodiments, the computer-readable storage medium is non-transitory.

[0059] 21) In some embodiments, a method for transmitting road profile information in a driver assistance system installed in a vehicle is performed, wherein the driver assistance system includes a camera and a data interface operably connected to a processor. In some embodiments, the method includes: selecting a first distance in front of a wheel of the vehicle; sampling a predetermined number of data points from the road profile along a projected path of the vehicle, wherein the road profile is based at least in part on an image captured by the camera, and wherein the data points are sampled along a segment of the path having an endpoint corresponding to the selected first distance, and wherein one or more data points are generated based on a stable reference plane; and transmitting the road profile information corresponding to the predetermined number of data points in a predetermined number of data transmissions through the data interface, wherein the number of data transmissions is less than the number of data points.

[0060] 22) The method as described in 21), comprising:

[0061] A frequency for transmitting data via the data interface is selected, wherein the selected frequency defines a frequency at which the predetermined number of data transmissions are transmitted.

[0062] 23) The method as described in 21), comprising:

[0063] A second distance traveled by the vehicle within a time period equal to the inverse of the selected frequency is estimated, wherein the data points are sampled along a segment of the path corresponding to the estimated second distance that exceeds the selected first distance.

[0064] 24) The method of 23), wherein estimating the second distance comprises determining that the second distance will be travelled within the time period at a known current speed of the vehicle.

[0065] 25) A method as described in 21), wherein the road longitudinal section information transmitted through the data interface is configured so that the road longitudinal section information corresponding to the corresponding data point can be read individually by the receiving system and associated with the corresponding part of the road represented by the road longitudinal section.

[0066] 26) The method of 21), wherein selecting the first distance in front of the wheels of the vehicle includes taking into account system delays.

[0067] 27) A method as described in 21), wherein one or more of the data points are associated with a corresponding confidence value.

[0068] 28) A method as described in 21), wherein the interval of the data points is determined so that it is estimated that the vehicle will travel from one data point to the next data point within a predetermined time period.

[0069] 29) A method as described in 21), wherein one or more of the data points are generated at least in part by combining longitudinal profile information from multiple frames.

[0070] 30) In some embodiments, a driver assistance system is installed in a vehicle, the system including a processor, a camera operably connected to the processor, and a memory storing instructions that, when executed by the processor, cause the system to perform the following operations: select a first distance in front of a wheel of the vehicle; sample a predetermined number of data points from a road profile along a projected path of the vehicle, wherein the road profile is based at least in part on an image captured by the camera, and wherein the data points are sampled along a segment of the path having an endpoint corresponding to the selected first distance, and wherein one or more data points are generated based on a stable reference plane; and transmit road profile information corresponding to the predetermined number of data points in a predetermined number of data transmissions through a data interface, wherein the number of data transmissions is less than the number of data points.

[0071] 31) In some embodiments, a computer readable storage medium stores instructions that, when executed by a processor operably coupled to a driver assistance system installed in a host vehicle, cause the system to perform the following operations: select a first distance in front of a wheel of the vehicle; sample a predetermined number of data points from a road profile along a projected path of the vehicle, wherein the road profile is based at least in part on an image captured by a camera operably connected to the processor, and wherein the data points are sampled along a segment of the path having an endpoint corresponding to the selected first distance, and wherein one or more data points are generated based on a stable reference plane; and transmit road profile information corresponding to the predetermined number of data points in a predetermined number of data transmissions through a data interface, wherein the number of data transmissions is less than the number of data points. In some embodiments, the computer readable storage medium is transient. In some embodiments, the computer readable storage medium is non-transitory.

[0072] 32) In some embodiments, a computerized road surface deviation detection method is performed by a driver assistance system installed in a vehicle, wherein the driver assistance system includes a camera operably connected to a processor. In some embodiments, the method includes: capturing a plurality of images depicting a portion of a road by the camera; determining whether a feature of the image is attributable to a shadow cast by a tree branch; based on the determination that the feature of the image is not attributable to a shadow cast by a tree branch, processing the feature to determine a vertical deviation of the portion of the road; and based on the determination that the feature of the image is attributable to a shadow cast by a tree branch, reducing consideration of the feature in determining the vertical deviation of the portion of the road.

[0073] 33) A method as described in 32), wherein reducing consideration of the feature when determining the vertical deviation of the portion of the road includes ignoring the feature when determining the vertical deviation of the portion of the road.

[0074] 34) The method of 32) wherein determining whether a feature of the image is attributable to shadows cast by tree branches comprises determining whether a point on the road is detected as changing in height over time.

[0075] 35) A method as described in 32), wherein determining whether a feature of the image is attributable to shadows cast by tree branches includes determining whether the height of a point on the road is detected as changing in sign over time.

[0076] 36) A method as described in 32), wherein determining whether a feature of the image is attributable to a shadow cast by a tree branch includes determining whether the height of a point on the road is detected as varying by more than a threshold height amount over a predetermined time period.

[0077] 37) A method as described in 32), wherein determining whether the characteristics of the image are attributable to shadows cast by tree branches includes determining whether the detected motion is not towards a focal point of the expansion.

[0078] 38) A method as described in 32), wherein determining whether a feature of the image is attributable to a shadow cast by a tree branch includes measuring a gradient to determine the softness of an edge.

[0079] 39) In some embodiments, a driver assistance system is installed in a vehicle, the system comprising a processor, a camera operably connected to the processor, and a memory storing instructions that, when executed by the processor, cause the system to perform the following operations: capture, via the camera, a plurality of images depicting a portion of a road; determine whether a feature of the image is attributable to a shadow cast by a tree branch; based on the determination that the feature of the image is not attributable to a shadow cast by a tree branch, process the feature to determine a vertical deviation of the portion of the road; and based on the determination that the feature of the image is attributable to a shadow cast by a tree branch, reduce consideration of the feature in determining the vertical deviation of the portion of the road. In some embodiments, the computer readable storage medium is transitory. In some embodiments, the computer readable storage medium is non-transitory.

[0080] 40) In some embodiments, a computer-readable storage medium stores instructions that, when executed by a processor operably coupled to a driver assistance system installed in a host vehicle, cause the system to perform the following operations: capture, via a camera operably connected to the processor, a plurality of images depicting a portion of a road; determine whether a feature of the image is attributable to a shadow cast by a tree branch; based on a determination that the feature of the image is not attributable to a shadow cast by a tree branch, process the feature to determine a vertical deviation for the portion of the road; and based on a determination that the feature of the image is attributable to a shadow cast by a tree branch, reduce consideration of the feature in determining the vertical deviation for the portion of the road.

[0081] 41) In some embodiments, a computerized road surface deviation detection method is performed by a driver assistance system installed in a vehicle, wherein the driver assistance system includes a single camera operably connected to a processor. In some embodiments, the method includes: capturing multiple images depicting a portion of a road by the camera; generating a road profile based at least in part on one or more of the multiple images of the road; associating confidence values ​​with points in the road profile; and transmitting data corresponding to the points in the road profile and the confidence values ​​associated with the points in the road profile to a vehicle control system.

[0082] 42) A method as described in 41), wherein the data corresponding to the points in the longitudinal section of the road includes the height of the road.

[0083] 43) A method as described in 41), wherein the data corresponding to a point in the road longitudinal section includes a distance in front of the vehicle.

[0084] 44) A method as described in 41), wherein associating a confidence value with a point in the road longitudinal section includes determining the confidence value based on a forward / backward verification performed between two images of the multiple images.

[0085] 45) A method as described in 41), wherein associating a confidence value with a point in the road longitudinal section includes determining the confidence value based on the curvature of a score matrix associated with a comparison of two images spaced apart in time.

[0086] 46) The method as described in 41), comprising:

[0087] A multi-frame confidence value is transmitted with the data corresponding to the point in the road profile and the confidence value associated with the point in the road profile, wherein the multi-frame confidence value is determined based on multiple height measurement results from different images regarding the point in the road profile and the difference of the height measurement results.

[0088] 47) In some embodiments, a driver assistance system is installed in a vehicle, the system including a processor, a single camera operably connected to the processor, and a memory storing instructions that, when executed by the processor, cause the system to perform the following operations: capture, via the camera, multiple images depicting a portion of a road; generate a road profile based at least in part on one or more of the multiple images of the road; associate confidence values ​​with points in the road profile; and transmit data corresponding to the points in the road profile and the confidence values ​​associated with the points in the road profile to a vehicle control system.

[0089] 48) In some embodiments, a computer-readable storage medium stores instructions that, when executed by a processor operably coupled to a driver assistance system installed in a host vehicle, cause the system to perform the following operations: capture multiple images depicting a portion of a road via a single camera operably connected to the processor; generate a road profile based at least in part on one or more of the multiple images of the road; associate confidence values ​​with points in the road profile; and transmit data corresponding to the points in the road profile and the confidence values ​​associated with the points in the road profile to a vehicle control system. In some embodiments, the computer-readable storage medium is transient. In some embodiments, the computer-readable storage medium is non-transitory.

[0090] 49) In some embodiments, a computerized road surface deviation detection method is performed by a driver assistance system installed in a vehicle, wherein the driver assistance system includes a camera having a rolling shutter and radial distortion, operably connected to a processor. In some embodiments, the method includes: capturing a plurality of images depicting a portion of a road by the camera; compensating for the effects of the rolling shutter and radial distortion; generating a longitudinal profile of the portion of the road based on at least the plurality of images depicting the portion of the road; transmitting information about the longitudinal profile of the portion of the road to a vehicle control system.

[0091] 50) The method as described in 49), wherein compensating for the influence of the rolling shutter comprises:

[0092] Project image points to 3D points on the road,

[0093] adjust the forward coordinates according to the vehicle's speed, and

[0094] The adjusted 3D points are projected back into the image.

[0095] 51) In some embodiments, a driver assistance system is installed in a vehicle, the system including a processor, a camera with a rolling shutter and radial distortion, and a memory storing instructions, the camera being operably connected to the processor, the instructions, when executed by the processor, causing the system to perform the following operations: capture multiple images depicting a portion of a road through the camera; compensate for the effects of the rolling shutter and radial distortion; generate a longitudinal profile of the portion of the road based at least on the multiple images depicting the portion of the road; and transmit information about the longitudinal profile of the portion of the road to a vehicle control system.

[0096] 52) In some embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by a processor operably coupled to a driver assistance system installed in a host vehicle, cause the system to perform the following operations: capture multiple images depicting a portion of a road via a camera operably connected to the processor, the camera having a rolling shutter and radial distortion; compensate for effects of the rolling shutter and radial distortion; generate a longitudinal profile of the portion of the road based at least on the multiple images depicting the portion of the road; and transmit information about the longitudinal profile of the portion of the road to a vehicle control system.

[0097] The above and / or other aspects will become apparent from the following detailed description when considered in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] The following disclosure is made with reference to the accompanying drawings, in which:

[0099] Figure 1 and Figure 2A system including a camera or image sensor installed in a vehicle is illustrated, according to some embodiments.

[0100] Figure 3A Illustrated are points overlaid on a view of a road, where the points may be tracked to compute a homography matrix, in accordance with some embodiments.

[0101] Figure 3B Illustrated are normalized relevance scores along a vehicle's path, according to some embodiments.

[0102] Figure 3C The path of a vehicle is illustrated projected onto an image of a road in accordance with some embodiments.

[0103] Figure 4A-4B An unstable road profile is illustrated in accordance with some embodiments.

[0104] Figure 5 A flow chart illustrating a method of correcting radial distortion and rolling shutter is shown, in accordance with some embodiments.

[0105] Figure 6A-6D A flow chart illustrating a method of detecting deviations in a road surface is shown, according to some embodiments.

[0106] Figure 7A-7B A stabilized road profile is illustrated in accordance with some embodiments.

[0107] Figure 8 A flow chart illustrating a method of creating a multi-frame road profile is shown, according to some embodiments.

[0108] Figure 9A-9B A stabilized road profile with fine tuning according to some embodiments is illustrated.

[0109] Figure 10A-10B A stabilized road profile is illustrated with fine tuning and adjustments for ridge detection, according to some embodiments.

[0110] Figures 11A-11C A multi-frame road profile is illustrated in accordance with some embodiments.

[0111] Fig.12 A flow chart illustrating a method of generating data from ID sampling according to some embodiments is shown.

[0112] Figure 13A-13C The results of 1D road profile sampling according to some embodiments are illustrated.

[0113] Fig.14A Illustrated is a predicted path of a wheel overlaid on a scene of a road in accordance with some embodiments.

[0114] Fig. 14B The predicted paths of the wheels are illustrated, showing the predicted vertical deviation from the road on the y-axis of each graph. Detailed Description

[0115] Reference will now be made in detail to features of the present disclosure, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like elements throughout. These features are described below to explain the technology disclosed herein by referring to the drawings.

[0116] Before explaining the features of the present technology in detail, it should be understood that the technology taught herein is not limited in its application to the details of the design and arrangement of the components stated in the following description or illustrated in the drawings. The technology taught herein can incorporate other features or be practiced or implemented in various ways. Moreover, it should be understood that the words and terms used herein are for descriptive purposes and should not be regarded as limiting.

[0117] In this document, lowercase coordinates (x, y) are used to represent image coordinates; and uppercase coordinates (X, Y, Z) are used to represent world coordinates in a right-handed coordinate system, where X is forward, Y is left, and Z is up, relative to the onboard camera.

[0118] By way of introduction, various embodiments of the technology described herein are for use with the above reference Figure 1 The described camera 12 mounted in the host vehicle 18 is useful for accurately detecting road shape, such as vertical sections of the road. Using the systems and methods provided herein, ridges and / or holes, such as speed bumps, curbs, and manhole covers, can be accurately detected with vertical deviations from the road plane as small as one centimeter. The systems and methods disclosed herein can be similarly applied to forward-looking, side-looking, and rear-looking cameras 12. The various methods described herein can accurately estimate a planar (or bi-quadratic) model of the road surface and then calculate small deviations from the planar (or bi-quadratic) model to detect ridges and holes.

[0119] Radial distortion and rolling shutter correction

[0120] Reference now Figure 5 , which shows that in references such as above Figure 2 Flowchart of method 500 for correcting radial distortion and rolling shutter in a road image of image frame 15 is described. In some embodiments, correction for both radial distortion and rolling shutter may be beneficial in order to accurately detect distance, accurately measure ego motion, and accurately generate road profiles without artificial upward curvature.

[0121] In some embodiments, at step 502 , the coordinates of inlier points in each captured image may be first corrected for radial distortion using a calibration model and default lens parameters.

[0122] In some embodiments, after the coordinates of the inbounds points are corrected, at step 504, the rolling shutter compensation can be performed on the points in each image according to the following steps. First, at step 506, a reference line y is selected in the image. t0 As t0. Then, in step 508, the image y coordinate gives the time offset of the reference line according to the following equation:

[0123] δ t =(y–y t0 )*t 线

[0124] Next, at step 510, assuming a default road model ([0,0,1] is good for normal vehicle speeds and calibrated camera heights), the point is projected to a 3D point on the road. Then, at step 512, the X (forward) coordinate is scaled by v*8 t to adjust, where v is the vehicle speed. Finally, at step 514, the 3D points are projected back into the image.

[0125] Single frame longitudinal section generation in a stable coordinate system

[0126] Reference now Figures 6A-6D , Figures 6A-6D is a flow chart of method 600, which creates a homography matrix based on a consistent ground plane at each time step, and which can then be used to align frames and compute residual motion and road profiles. In some embodiments, techniques according to method 600 can be superior to previously known techniques because method 600 can allow multiple frames to be analyzed relative to the same reference frame, rather than allowing each frame to be analyzed relative to a unique reference frame uniquely computed from the corresponding frame itself. As explained in further detail below, in some embodiments, method 600 enables a stable world coordinate system to be computed based on at least a combination of: (a) a reference plane computed from a current frame, (b) a reference plane computed from one or more previous frames, and (c) a default reference plane based at least in part on an assumption of a fixed position of the camera relative to the vehicle. In some embodiments, the combination of these factors can result in a stable world coordinate system that can be used to compare frames corresponding to different time steps in order to compute residual motion and identify vertical deviations from the road surface. In some embodiments, generating the longitudinal profile in a stable road world coordinate system may be more desirable or advantageous for a receiving system (such as an OEM vehicle control system that controls the vehicle's suspension); by generating the longitudinal profile in a stable coordinate system, the information can be more easily applied by the control system without having to be converted to a road afterward to be coordinated by the control system.

[0127] It should be noted that in some embodiments, another type of model of the road surface can be used in a similar manner without using a homography matrix. For example, in some embodiments, the road surface can be modeled as a quadratic surface with a certain curvature. For the sake of brevity, in the description of the exemplary method 600 herein, the present disclosure will refer to the use of a homography matrix.

[0128] As explained in further detail below, one aspect of method 600 can provide the creation of a homography matrix, which is used to warp one image toward another image in order to calculate residual motion and detect vertical deviations of the road surface. In some embodiments, the creation of the homography matrix used in the warping process can include a combination of information related to the previous estimate of the plane normal, the current estimate of the plane normal, and the average estimate of the distance and normal based on the historical time period (e.g., the previous 20 seconds) that the car has driven through. In some embodiments, the reconstructed homography matrix explained below can be understood as not only based on the information collected in the image used by the system when it determines whether there is any vertical deviation in the road surface, but also based on predetermined speculations about the road plane and / or historical calculations about the road plane. In some embodiments, such a homography can be calculated and used to analyze the frame to detect the vertical deviation of the road surface according to the following steps.

[0129] At step 602, in some embodiments, an initial guess H0 of a homography matrix for the road is used to track points between two consecutive frames. Figure 5 The method is explained, but in some embodiments this initial step 602 may be performed without correcting for radial distortion and rolling shutter. These are point correspondences between the pre-warped image I1w and the image I2.

[0130] In some embodiments, the points may be placed uniformly in each image, giving more points closer to the vehicle, or the points may be spaced uniformly in distance, requiring a focus on more points in the image that are farther from the vehicle. In some embodiments, the points may be spaced uniformly vertically in the image, but more weight may be given to points farther away in the least squares calculation of the homography matrix. For example, when performing least squares (e.g., solving Ax=b), each row of A may be weighted equally, or some rows may be given more weight (e.g., solving wAx=wb, where w is a diagonal matrix of weights).

[0131] At step 604 , in some embodiments, the inverse matrix of H0 is used to obtain point correspondences between images I1 and I2 .

[0132] At step 606, in some embodiments, the coordinates of the points in each image are corrected for radial distortion and compensated for rolling shutter. Figure 5 The explained techniques perform these corrections and compensations.

[0133] At step 608, in some embodiments, the homography matrix H of the road is found using random sample consensus (RANSAC) on the corrected points. RANSAC is an example of a robust algorithm that rejects outliers. Iteratively reweighted least squares (IRLS) and other robust algorithms may also be used.

[0134] At step 610, in some embodiments, the homography matrix H that aligns the road in the distorted image may be calculated using the same inliers / outliers classification on the uncorrected points. d .

[0135] At step 612, in some embodiments, the homography matrix H may be decomposed into frames and N 当前 and D 当前 The ego-motion (rotation R and translation T) between (plane normal and distance to the plane, respectively). Note that R and T represent the vehicle motion and are thus unique, while planes N and D are somewhat arbitrary and will therefore have significantly more noise.

[0136] At step 614, in some embodiments, based on the ego motion (rotation R and translation T) and N 当前 and D 当前 The new plane estimate is used to adjust the previous estimate of the plane (N 先前 , D 先前 The adjustment value N can be calculated according to the following equation: 新 and D 新 ,

[0137] (N 新 ,D 新 ) = 0.8 * update R,T (N 先前 ,D 先前 )+0.1*(N 当前 ,D 当前 )+0.1*History(N 当前 ,D 当前 ) (1)

[0139] The function history() is (N 当前 and D 当前 ), the function updates R,T Given by:

[0140] N 更新 =R*N先前 (2)

[0141] D 更新 =T*N 更新 +D 先前 (3)

[0142] Note that the values ​​of 0.8, 0.1, and 0.1 in equation (1) are merely exemplary, and any other set of three values ​​that sum to 1.0 may be used.

[0143] In step 616, in some embodiments, R is calculated by chaining according to the following sub-steps: 远 and T 远 First, for each close frame pair between the current frame and the previous far one, a matrix can be created according to:

[0144]

[0145] Next, the matrix C i Multiply to get C 远 :

[0146]

[0147] Then finally, the product C 远 The upper 3x3 matrix is ​​taken as R 远 , and the top three elements of the right column are taken as T 远 .

[0148] In step 618, in some embodiments, based on the 远 and T 远 The calculated values ​​create a new homography matrix H 远 ,

[0149]

[0150] where T represents transpose, and where the prime symbol represents the second image (eg, 'T is in the coordinate system of the second image).

[0151] In step 620, in some embodiments, the linking process may be repeated using the homography matrix of the distorted image. 近 Links Rd 远 and Td 远 (e.g., according to a similar equation as described above with reference to step 618) and N from the undistorted image 更新 and D 更新 To construct the homography matrix Hd 远 .

[0152] In step 622, in some embodiments, the vehicle path is calculated in world coordinates (e.g., X, Y, Z) based on information from the vehicle, such as information indicating the vehicle's speed, velocity, direction, acceleration, etc.; this information may be obtained from sources other than a camera-based navigation system, such as other sensors equipped with the vehicle that can determine its speed, velocity, direction, acceleration, and other state information.

[0153] At step 624, in some embodiments, the vehicle path is projected onto the undistorted image using stable planar parameters, and then the coordinates are distorted according to radial distortion and rolling shutter to obtain the path coordinates (x, y) in the image. In some embodiments, due to radial distortion and rolling shutter, the path is projected onto the current image in two steps; first, the path is projected onto virtual undistorted image coordinates; then, the path's coordinates are distorted to give distorted coordinates. Alternatively, in some embodiments, reasonably accurate results may be obtained simply by projecting the vehicle path into the distorted image (particularly if the lens has low radial distortion).

[0154] At step 626, in some embodiments, the homography matrix Hd 远 Make the strip from image I 远 Warp towards I2, where I2 is the current image and I 远 is the previous image. Note that in some embodiments, I 远 It may not be the image immediately preceding I2. In some embodiments, it may be necessary for the vehicle to travel a minimum distance (e.g., about 0.7 m) between I 远 For example, in a system running at 18 frames per second, if the vehicle is traveling at 45 km / h or above, I 远 It can be a previous image; if the vehicle is traveling between 22.5KMH and 45KMH, then I 远 It can be the image before the previous image. The sub-pixel alignment of the rows in the two strips gives the residual motion (d y ). In some embodiments, sub-pixel alignment may be performed according to the techniques discussed in U.S. Patent No. 9,256,791, filed on November 26, 2014, and entitled “Road Vertical Contour Detection,” and U.S. Application No. 14 / 798,575, filed on July 14, 2015, and entitled “Road Contour Vertical Detection.”

[0155] In step 628, in some embodiments, Hd 远 The inverse matrix of can be used to make the point (x,y+d y) is distorted back from the distorted distorted image to obtain the distorted image ID 远 The matching points in .

[0156] At step 630, the point (x, y) and the matching point are undistorted relative to the undistorted image coordinates. As a result of step 628, at all points (x, y) and Id along the path of the distorted I2 远 There is a sub-pixel match between corresponding points in . To obtain accurate numbers, these coordinates can be undistorted in both images (for radial distortion and rolling shutter) and distorted using H 远 Utilize I 远 The image is re-warped using the coordinates of the image (rather than the image itself). In some embodiments, the process may include manipulating only two vectors of image coordinates with approximately 200 points per path, rather than the actual image; therefore, the process may be computationally inexpensive.

[0157] In step 632, in some embodiments, H 远 to distort the matching undistorted points and subtract the undistorted point (x, y) to get the true residual motion. For example, the y coordinate can be subtracted to get the undistorted d y .

[0158] In step 634, in some embodiments, the residual motion and the undistorted point (x, y) are used to calculate the distance and height of the longitudinal section of the single frame. In some embodiments, the calculation can be performed according to the technology discussed in U.S. Patent No. 9,256,791, entitled "RoadVertical Contour Detection" filed on November 26, 2014 and U.S. Application No. 14 / 798,575, entitled "Road Contour Vertical Detection" filed on July 14, 2015. Fig. 7A and Figure 7B An exemplary result of a single-frame longitudinal section generated using a stable plane according to the above method is shown. Fig. 7A The wider part of the longitudinal section is shown, while Figure 7B An image zoom-in on the area before a detected ridge in the road is shown. Fig. 7A and Figure 7B Example profile of , using only forward translation along the plane to perform the alignment. The profile is stable during and after the ridge. Note that the vertical axis is -0.15m to 0.15m over the 90m forward run. Note that the range of sampling in height is reduced to 0.05 meters, which is significantly less than what would be calculated without using a stable coordinate system. Figure 4A and Figure 4B The range of profile heights shown in .

[0159] Multi-frame longitudinal section generation in a stable coordinate system

[0160] In some embodiments, a plurality of continuous image frames captured near the same point in time may all contain a single common portion of a road as part of an image. In some embodiments, it may be advantageous to associate image data and / or profile data obtained from each corresponding frame and combine it with other images and / or profile data corresponding to the same portion of the road acquired from other frames for analysis. Therefore, a technique for combining road profiles corresponding to overlapping areas of a road is provided. In some embodiments, the techniques described below with respect to method 800 may be used to align the various profiles corresponding to the same portion of the road so that a slight misalignment between multiple profiles is not inaccurately interpreted as a vertical deviation (e.g., a ridge or depression) of the profile of the road. When the same stable world coordinate system is used to calculate and express the profile, it is significantly helpful to ensure that the difference between the profiles does not inaccurately indicate a vertical road profile deviation, and the following corrections with respect to method 800 may further reduce the inaccurate indication of the vertical road profile deviation. In some embodiments, the techniques of method 600 and method 800 may be used alone, although when they are used together, results with minimal inaccurate road profile deviations may be produced.

[0161] Reference now Figure 8 , which shows a flowchart of a method of aligning multiple frames (e.g., frames representing different moments when a vehicle traverses a portion of a road) with a global model to assemble a road longitudinal section corresponding to overlapping portions of the road.

[0162] Let T z is the forward motion of the vehicle along the path. In some embodiments, due to T z is much smaller than the length of the longitudinal section, so for the same point along the road there may be many measurements over time. In other words, there are many consecutive frames captured by the camera where the same part of the road is visible in the respective frames. Therefore, in some embodiments it would be advantageous to derive a multi-frame model that uses all the measurements of the same point combined. One approach is to take all single frame longitudinal sections and translate them forward by shifting T z Draw them. They will be fairly well aligned by this method, but there will be small vertical shifts and variations in slope. Simply averaging the heights at each point along the path can smooth out the ridges, so it is important to account for these shifts in height and slope. Therefore, in some embodiments, it is desirable to use the slope and offset values ​​(a, b) to align the current single frame profile P 当前 Aligning with the global model. In some embodiments, the alignment can be done according to the following steps.

[0163] In step 802, in some embodiments, k may be selected as the number of the latest longitudinal sections to be used for analysis. Then, the current longitudinal section data P from 5m to 12m is selected. 当前 and fall in the same region P k k adjusted previous single-frame profiles in the same region where all P k .

[0164] At step 804, stochastic gradient descent may be used to minimize a cost function E on the slope and offset values ​​(a, b), where E includes a data term, a smoothness term, and a regularization term. In some embodiments, the data term D may be any norm, not just L_1. The data term D may also be a saturation norm, such as Sum(min(abs(P_ 当前 ), threshold)). In some embodiments, the smoothness term S may also be a norm or a saturation norm. In some embodiments, the regularization term R may be attributed to a penalty designed to avoid large slope and offset terms (such as terms including the squared or absolute values ​​of a and b, respectively). In some embodiments, E may be given by the following equation:

[0165] E a，b =λ1D+λ2S+λ3R (7)

[0166] where the data term D taking into account the difference from zero is given by:

[0167] D=∑(abs(P 当前 )) (8)

[0168] Considering the adjusted current profile and the previously adjusted single-frame profile P k The smoothness term of the difference between :

[0169]

[0170] And the regularization term is designed to avoid large slope and bias terms:

[0171] R = a 2 +b 2 (10)

[0172] Note that in the above equation, P 当前 The offset and slope of P have been shifted (a, b). k is displaced by appropriate forward motion and by the slope and pitch (a k ,b k) is adjusted by the previous single frame measurement results. In some embodiments, stochastic gradient descent uses the gradient calculated for a random point along the longitudinal section and the value of E(a,b) to adjust the value of (a,b). In some embodiments, a fixed sequence of points can be used instead of truly random points, which can avoid unpredictable random results with an unfavorable large number of outliers and may not be reproducible.

[0173] In some embodiments, each of the K most recent profiles may be individually adjusted toward the current profile.

[0174] At step 806, in some embodiments, the optimal values ​​of (a, b) may be applied to the longitudinal section P 当前 and can be adjusted in the P 当前 and all adjusted profiles P k For example, at each distance along the longitudinal section, the system may have a median value from P 当前 The measurement results may also have some number (e.g., 5) of previous measurement results P k The system can calculate the median of those 6 values ​​(excluding invalid values ​​in some embodiments). In the described embodiment, this gives a multi-frame profile of the current frame.

[0175] exist Fig.9A , Fig. 9B , Figures 10A-10B and Figures 11A-11C In the example shown in FIG, the adjusted P 当前 and all adjusted profiles P at all 140 values ​​from 5m to 12m with an interval of 0.05m k Perform median calculation between . Figures 9A-9B A single frame profile is shown after fine tuning of the A and B values ​​(eg, step 804). Fig.9A shows the road profile formed using the stabilization plane after fine alignment, while Fig. 9B shows a zoom-in of the same data in an area of ​​the road preceding a slow-speed ridge. Note that the samples are more tightly distributed in height, but the height of the ridge is reduced. Fig. 7A and Figure 7B and Fig.9A and Fig. 9B For comparison: Fig. 7A and Figure 7B In the flat section, the longitudinal section range is about 0.05m; Fig.9A and Fig. 9B , the range is about half that. However, note that Fig. 7A and Figure 7B In the longitudinal section, the ridge is higher than about 0.08m, while in Fig.9A and Fig. 9B Some longitudinal sections are lower than 0.08m. Fig. 10A and 10B A single frame profile is shown (at two different zoom levels) where A and B are calculated taking into account slow ridges as explained below with reference to ignoring data items regarding slow ridges. Fig. 10A and Fig. 10B In keeping the ratio Fig. 7A and Figure 7B While in a tighter range, push down as 9A and Fig. 9B This problem is alleviated by the slow-speed ridge. Figures 11A-11C A multi-frame profile generated according to method 800 as described above is shown at various zoom levels.

[0176] 1D Sampling

[0177] In some embodiments, transmitting data about a road profile of a portion of the road in front of a vehicle may require the transmission of a large number of values. For example, the output of a road profile of 5 to 12 meters in front of the vehicle sampled every 0.05 meters requires 140 values; for longer profiles, the number of required values ​​is obviously greater. The large number of values ​​that may be required may be problematic for a controller area network (CAN) and may require considerable computing resources on the receiving end. For example, in some embodiments, 200 data points may be transmitted per wheel per frame, which may require the transmission of more than 1KB of data per frame. In some embodiments, transmitting this amount of data per frame may be infeasible and / or computationally too expensive for the receiving system.

[0178] Additionally, some DAS and self-driving systems require data to be received at a predetermined frame rate (such as 100 Hz), such that the system requires one data point corresponding to the height of the road (at a predetermined distance in front of the vehicle) every 10 milliseconds. In some embodiments, sending data to the system every 10 milliseconds may not be feasible because it may monopolize a data channel in the vehicle, preventing other information from being sent over the same data channel.

[0179] Therefore, there is a need for systems and methods for transmitting computationally significant amounts of data about road profiles in a manner such that: (a) the total amount of data sent is computationally manageable, and (b) the data transmission does not monopolize the data channel at all points in time.

[0180] In some embodiments, a data format for sending information about a road profile is provided, wherein the road height is output at a specific distance in front of the wheel. In some embodiments, the distance is a fixed distance (e.g., 7 meters), and in some other embodiments, the distance can be determined dynamically. For example, the distance in front of the wheel can be dynamically determined based on the speed of the vehicle, with a reference distance increased at higher speeds. In some embodiments, the distance can be set based on the distance covered by the vehicle in a given amount of time (e.g., the distance covered by the vehicle in 0.5 seconds at its current speed). Sending only data corresponding to a certain fixed distance in front of the wheel, rather than transmitting the entire profile or profile data related to the road height along the entire known wheel path at each time step, allows less total data to be transmitted, which can save computing resources and bandwidth.

[0181] In some embodiments, a data format is also provided in which multiple data points are transmitted substantially simultaneously, rather than just one data point, to effectively double the frame rate. For example, if the frame rate is 10 Hz, in some embodiments, the system can simulate a 100 Hz data output by sending 10 data points at a time in a burst at a true frequency of 10 Hz. After the transmission, the receiving component can split the 10 data points and query them one at a time as needed. In some embodiments, all data points corresponding to a single frame can be transmitted in a single data transmission, while in some embodiments, they can be transmitted in multiple transmissions with a number less than the number of data points per frame (for example, a frame of 7 data points per frame is sent in only two CAN messages). In some embodiments, less than 10 data points can be sent per burst. In some embodiments, 16 data points can be sent per burst.

[0182] Now pay attention Fig.12 , Fig.12 is a flow chart illustrating an exemplary method 1200 of generating data according to the 1D sampling technique discussed above.

[0183] In some embodiments, at step 1202, the system may select a distance in front of the wheels of the vehicle to which the transmitted data should correspond. The distance in front of the wheels may be predetermined by the vehicle control system to which the data is to be transmitted. In some embodiments, selecting the distance in front of the wheels may include taking into account system latency so that the system will receive the data when the corresponding data corresponds to a target distance in front of the wheels. For example, if the target distance is 7 meters, the system may select 71 centimeters as the distance in front of the wheels so that the system latency causes the system to receive the data when the corresponding data point is 7 meters in front of the wheels (rather than 71 centimeters).

[0184] In some embodiments, the system may select the number of data points that should be sent per burst at step 1204. For example, as discussed above, the system may send 10 data points per burst. In some embodiments, the system may send multiple data transmissions per burst, such as two CAN messages per frame. In some embodiments where multiple transmissions are sent per burst, the system may select the number of data points (which may be the same or different) to be sent per transmission in the data burst.

[0185] In some embodiments, the system may select a frequency at which to send data bursts at step 1206. For example, the system may send a data burst (e.g., 10 data points) every 100 milliseconds. The frequency at which data bursts are sent may be determined by the bandwidth of the data channel in the vehicle, or may otherwise be predetermined by the receiving vehicle control system.

[0186] In some embodiments, at step 1208, the system may determine the distance the host vehicle is expected to travel in a time corresponding to the frequency at which data is to be sent. For example, the system may determine the distance the host vehicle is expected to travel in 100 milliseconds. In some embodiments, the distance the vehicle is expected to travel may be determined by sampling the vehicle's current speed and / or acceleration (from existing image data and / or from other sensors and systems in the vehicle).

[0187] In some embodiments, at step 1210, the system may sample a road profile at n points along the projected path of the wheel, where n is the number of data points per burst. In some embodiments, the points at which the road profile is sampled may be spaced between a selected distance in front of the wheel and that distance plus an estimated distance the vehicle will travel before sending the next data burst (e.g., may span, or may be evenly spaced therebetween). In some embodiments, the road profile may be sampled at points along a path segment that extends beyond a selected distance in front of the wheel plus an estimated distance the vehicle will travel before sending the next data burst; in this way, there may be some overlap between portions of the road corresponding to data bursts from different frames, creating redundancy that, in some embodiments, may improve the accuracy of vertical deviation calculations. In some embodiments, one or more data points may be associated with a corresponding confidence value; other techniques regarding confidence values ​​are discussed below.

[0188] In some embodiments, the data points may be generated based on a stable reference plane. In some embodiments, the data points may be generated at least in part by combining longitudinal profile information from multiple frames.

[0189] In some embodiments, the vehicle control system can receive all data points substantially simultaneously in a single data burst, and can split the data points as needed to read them individually in the time period before the next data burst arrives. For example, the system can receive a data burst containing 10 data samples, and can split the burst and read one sample every 10 milliseconds until a second data burst (with 10 additional data points) is received 100 milliseconds later. In this way, the data burst can simulate sending a data point every 10 milliseconds, even though the data is only transmitted once every 100 milliseconds.

[0190] For example, if a data point corresponding to 7 meters in front of the wheel is needed, the system can estimate how far the vehicle will move every 10 milliseconds and can sample the known road profile with 10 data points, starting at the point 7 meters in front of the wheel and continuing sampling. TM Notation:

[0191] Z=7+[0:0.1:0.9]*dZ / 10 (11)

[0192] Where dZ is the distance traveled between frames (100ms). Thus, in the case where the vehicle travels 5cm in 10ms, the transmitted data points may correspond to 700cm, 705cm, 710cm, 715cm, and so on, up to 745cm. After 100ms, a second pulse of another 10 data points may be sent, at which point the vehicle will have moved forward approximately 0.5m. As the vehicle moves forward approximately 0.5m, the next 10 data points will be roughly evenly spaced from the previous 10 data points in real-world coordinates, creating a continuous path of data points every 5cm that the vehicle's driver assistance software can query every 10ms for a point 7m ahead of the current vehicle. In this way, by sending a burst of data points at a time, the system can simulate sending one data point at a higher frame rate without continuously monopolizing the data channel in the vehicle.

[0193] In some embodiments, if 10 data points are transmitted according to the above method, but each set of 10 data points is sampled from a different road profile, there may be a small discontinuity each time the data points switch from one profile to the next. To avoid this problem, a smooth transition from one profile to the next can be generated by sampling 20 data points between, for example, 7m and 2*dZ:

[0194] Z=7+[0:0.1:1.9]*dZ / 10 (12)

[0195] Then, the current first 10 measurements are averaged with the previous second 10 measurements, gradually increasing the weight of the current measurement and decreasing the weight of the previous frame measurement. For example:

[0196] w=[0;0.1;0.9]*10 / 9; (13)

[0197] P 输出 =w*P 当前 (1:10)+(1-w)*P 先前 (11:20) (14)

[0198] The results of the 1D distribution diagram according to the method and examples explained above are shown in Figure 13A-13C middle. Fig.13A A 1D sample distribution map is shown, where 10 samples from each frame are displayed as segments. Fig.13A The gap at 50 meters is due to inefficiency in the frame. Fig. 13B shows a zoom-in of the area preceding the detected slowing ridge, and Fig. 13C A magnification of the area centered on the slowing ridge itself is shown. Fig.14A and Fig. 14B shows the above reference 1D sampling and Figure 13A-13C Examples of system results for explained examples. Fig.14A The predicted path and residual motion are shown overlaid on a road image, where the residual motion d y Represented graphically as the deflection of a curve in the x-direction, where a deflection to the right indicates a ridge; it can be seen that the calculated deflection starts at the beginning of the ridge, increases towards the middle of the ridge, and then decreases on the far side of the ridge. In the example shown, the deflection has been scaled 30 times to make the small deflection visible on the graph. Fig. 14B Representation and Fig.14A The same drawn lines as shown in , but shown in metric coordinates in a cross-sectional view. Fig. 14B The top two curves in show the current left and right profiles from 5 to 20 meters sampled every 5 cm from 5 to 10 meters and every 10 cm from 10 to 20 meters; the y-axis is -0.2m to 0.2m, and the detected ridge is approximately 0.08m high. Fig. 14B The bottom two curves in show the left and right 1D samples, which show the distribution map accumulated from 7 meters in front of the vehicle to the vehicle itself, where the vertical line represents the point 7 meters in front of the wheel. The segment to the right of the vertical line is the part calculated for the current frame: from 7m to (7+dZ)m. The part to the left of the vertical line is calculated in the previous frame. Over time, the ridge will move to the left, and when it reaches zero, the wheel actually hits the ridge and the car can react.

[0199] In some embodiments, the intervals of the data points for 1D sampling can be determined based on alternative techniques. For example, in some embodiments, the data points can be spaced apart in a manner that separates the distance that the vehicle is expected to travel in a fixed time. For example, the vehicle control system can be configured to process a data point every 10 milliseconds regardless of the vehicle speed. In this case, the system can estimate the position where the vehicle is expected to be located at intervals of 10 milliseconds (e.g., based on the current speed of the vehicle) (e.g., estimating the corresponding position where the vehicle will be at various time points, with the time points spaced 10 milliseconds apart), and can place data points corresponding to each of these estimated positions.

[0200] In one exemplary system, bursts are output at 18 bursts per second (every 55 milliseconds), and data points are spaced to sample every 10 milliseconds at the distance covered by the vehicle. Thus, a burst may include enough samples to cover the time between bursts (55 milliseconds), for example by outputting 7 samples covering 60 milliseconds, so that there is some overlap in the portion of the road covered by each burst.

[0201] In another exemplary system, the bursts each include 10 data points, and the data points are spaced apart at a distance corresponding to 5.5 milliseconds. In this system, 10 samples cover 49.5 milliseconds, leaving 5.5 milliseconds until the second burst arrives 55 milliseconds after the first burst, so that no data points overlap.

[0202] Calculating system latency

[0203] In some embodiments, a DAS or autonomous driving system may require data corresponding to a point a certain distance (e.g., 7 meters) in front of the vehicle. However, in some embodiments, due to system latency, by the time the image data is processed and received by the controller, the vehicle has moved and the data reflects a point that is less than the desired distance in front of the vehicle. Therefore, systems and methods are needed that account for system latency when processing image data and transmitting road profiles and other data to a vehicle controller.

[0204] In some embodiments, if the delay T L is known, the sampling distance of the data points can be adjusted as follows:

[0205] Z=7+[0:0.1:1.9]*dZ / 10+v*T L (15)

[0206] where v is the vehicle speed.

[0207] Based on the above adjustment for the delay, the profile can be sampled at a distance adjusted for the delay to give a 1D profile at the desired distance in front of the wheel, correcting for the time when the signal is received by the controller. In some embodiments, the value TL Delays in controller calculations or delays in any other components of the system may also be taken into account.

[0208] In some embodiments, the system delay may not be fixed and may therefore be unknown to the road profile estimation system. In some embodiments, if the data output includes a timestamp for the image, the vehicle controller can use the timestamp to estimate the delay from image capture to the data being received by the controller. The controller can then correct for changes in system delays accordingly. In some embodiments, it may be advantageous if the road profile system oversamples the profile and outputs data with denser intervals than required by the receiving controller or system. The controller or system can then determine the actual delay and select the data to use. If a 2 millisecond interval is used instead of a 10 millisecond interval, the controller can be corrected to better than 1 millisecond (e.g., an error of less than half a millisecond). Since all values ​​selected by the controller are calculated by the road profile system, it is easy to reproduce the results offline, such as in testing. In some embodiments, since the controller does not interpolate values ​​and the numbers output by the controller are values ​​calculated by the road profile system, it is possible to understand where the value comes from.

[0209] Ignore data items about slow ridges

[0210] In some embodiments, the use of multi-frame alignment techniques such as those described above with reference to method 600 may tend to "push down" the height of the detected slow-down ridge because the algorithm tends to seek to push the entire longitudinal section (including the road in front of the slow-down ridge and the slow-down ridge itself) down to zero. In order to offset this trend, in some embodiments, systems and methods are provided that can ignore data items about values ​​exceeding a certain threshold height. In some embodiments, the threshold height can be predetermined or dynamically determined. For example, in (a, b) optimization, the system can ignore data items of all values ​​exceeding 3.5 centimeters.

[0211] In some embodiments, the system can determine the possible detected slow ridge by fitting the profile to a straight line. If the profile does not fit a straight line (eg, if the mean square error is large), then the profile is suspected to contain a slow ridge.

[0212] Confidence value

[0213] In some embodiments, it is advantageous to calculate one or more confidence values ​​associated with a road profile and / or a road image. In some embodiments, confidence values ​​associated with a single frame or associated with multiple frames may be calculated. In some embodiments, confidence values ​​associated with corresponding points in an image or corresponding points in a road profile may be calculated.

[0214] Single frame confidence

[0215] In some embodiments, the confidence value of a single frame may be calculated according to the following.

[0216] In some embodiments, the first step in calculating the confidence is to use the relevance score (e.g. Figure 3B ) to perform forward / backward validation. Figure 3B There are 13 columns, each representing the score for a given offset (-6 to 6) for that row. Figure 3B , darker shading represents higher correlation scores, while lighter shading represents lower correlation scores. In some embodiments, the correlation scores can be represented by different colors, such as red represents high correlation scores, yellow represents low correlation scores, and green represents even lower correlation scores. The residual motion from image 2 to image 1 for a point should be equal in magnitude, and the residual motion from image 1 to image 2 should be opposite in sign. The latter can be determined by looking at the diagonal in the score matrix from image 2 to image 1. If the forward / backward motion does not match within 1 pixel, the confidence level can be determined to be zero.

[0217] In some embodiments, if the forward / backward motion matches, the confidence can be determined as the curvature of the score matrix. When the image texture is significant, a higher confidence score can be determined accordingly. For example, in some embodiments, good texture can give a curvature score of the order of 0.25.

[0218] In some embodiments, the area near the end of the stripe may be assigned a confidence value of zero.

[0219] In some embodiments, the determined confidence values ​​can be used in a multi-frame analysis process, such as the process explained above with reference to method 800. Two thresholds on confidence can be used: first, a high confidence for selecting points in the AB optimization; and second, a much lower threshold for determining which points to use for the median. For example, a curvature score above 0.1 can be used to determine which points can be used for the random descent optimization (e.g., step 804), while a curvature score of 0.01 can be used to determine whether to consider a point in the median (e.g., step 806).

[0220] Multi-frame confidence

[0221] A multi-frame confidence can be output from the system based on the difference in the multiple height measurements and the height used in the median calculation. In some embodiments, for each point along the path, between zero and five samples are used to calculate the median output based on the single-frame confidence of the point in the current frame and the previous four frames. Let N be the number of samples above the threshold, the multi-frame confidence can be calculated as follows:

[0222] In some embodiments, if N=0, the multi-frame confidence may be determined to be zero. In some embodiments, if N=1, the multi-frame confidence may be determined to be zero.

[0223] In some embodiments, if N ≥ 2, v may be calculated as follows:

[0224]

[0225] where h is the median height and h i is the height of each sample in step 806. Note that the square (N-1) 2 Penalize small N.

[0226] In some embodiments, the confidence C is given by:

[0227] C=5*(1–S*v) (17)

[0228] This produces a confidence value between zero and 5. The ratio S can be determined empirically. In some embodiments, S can be set equal to 100. In some embodiments, the confidence values ​​of multiple points in the image can be indicated by the saturation of the color of each point in the image.

[0229] Ignore shadows from moving vehicles and objects

[0230] In some embodiments, a camera-based DAS and / or autonomous driving system can detect shadows in captured images. For example, on a bright day, it is common for moving vehicles or other objects such as bicyclists and pedestrians in adjacent lanes or on sidewalks to cast moving shadows onto the road near the host vehicle. When such a shadow moves across the road, it creates residual motion and may be inaccurately interpreted as a fixed ridge or depression in the road surface and cause an undesirable controller response.

[0231] It should be noted that shadows from moving objects often have considerable residual motion and therefore may be detected as very high ridges in the road. For example, the shadow of an oncoming vehicle moving at the same absolute speed as the host may appear as a ridge half the height of the camera without correction. In some embodiments, these shadows are not tracked well and will obtain low confidence or low multi-frame confidence values. In these cases, the system simply ignores the shadow. However, moving shadows may not always be successfully ignored based on confidence values ​​alone. Therefore, there is a need for systems and methods that can distinguish moving shadows from fixed ridges and depressions in the road, including by actively filtering out data determined to correspond to moving shadows.

[0232] In some embodiments, if a moving object is detected by the DAS and / or other components of the self-driving system (e.g., camera-based vehicle detection, radar-based target detection), this information can be used to filter out shadows by reducing the confidence value associated with the residual motion corresponding to motion on the road plane at the speed of the detected object.

[0233] For example, consider a host vehicle traveling at 20 m / s and a target vehicle detected in an adjacent lane having a relative speed of -38 m / s. Simple subtraction will determine that the target vehicle is moving at -18 m / s relative to the stationary road. For example, the homography matrix in equation (6) (equation number 6) explained above with reference to step 618 of method 600 can be used to estimate the expected motion of a point on the road moving at this speed. In some embodiments, the homography matrix can be decomposed into the motion vector T 远 can be adjusted to account for the target vehicle motion during the time between the two frames used to create the decomposed homography matrix, and a new homography matrix can be reconstructed. The expected residual motion of the points can then be calculated as 远 The distortion and the adjusted T 远 The difference between the reconstruction matrices of the image space and the image space. The term "warp" as used herein may refer to a transformation from image space to image space. In some embodiments, the confidence values ​​assigned to points along the path having measured residual motion that is close to suspicious residual motion may then be reduced. In some embodiments, if many points along the path have residual motion that is close to suspicious residual motion, the entire frame may be invalidated.

[0234] Techniques for detecting shadows of moving objects are further discussed in U.S. Patent Application 14 / 737,522, entitled “HAZARD DETECTION FROM A CAMERA IN ASCENE WITH MOVING SHADOWS,” filed on June 12, 2015. In some embodiments, the techniques disclosed in that application may be incorporated into the techniques disclosed herein to more accurately calculate and output road profiles, including detecting small vertical deviations detected in the road surface in a similar manner so that larger vertical deviations due to road hazards can be detected.

[0235] Ignore moving shadows from trees

[0236] Just as camera-based DAS and autonomous driving systems can detect shadows of moving objects such as vehicles and pedestrians (as described immediately above), camera-based DAS and autonomous driving systems can also detect shadows cast by tree branches. Typically, wind blowing causes these shadows to move slowly across the road surface. In some embodiments, these small motions are not detected in the same manner as the motion attributed to large objects; instead, these small motions may appear as residual motion of small deviations in the road profile. Therefore, there is a need for systems and methods that can distinguish slowly moving tree branch shadows (and the like) from small vertical deviations in the road surface, including by actively filtering out data determined to correspond to tree shadows.

[0237] In some embodiments, the motion of the shadows may be determined to be due to tree branches and may be ignored based at least in part on the following three criteria:

[0238] First, in some embodiments, a swaying branch may be detected as jumping back and forth to produce a shadow motion that is inconsistent over time. Thus, the resulting height profile of points along a path affected by the swaying branch's shadow may be detected as changing over time; in some embodiments, such points may change sign and may therefore be detected as a ridge at one moment and then as a hole at the next moment. In some embodiments, data for a given point may be collected over a period of time, and any difference in detected height that exceeds a threshold difference may be determined to correspond to a moving shadow. For example, data may be detected over a period of time such as half a second, and any difference for a single point that exceeds a predetermined threshold within that half-second period may be taken to indicate a moving shadow attributable to a swaying branch. Thus, the confidence value for that point may be reduced.

[0239] Second, in some embodiments, motion from a swaying branch may not be toward the focal point of the expansion. In some embodiments, this gives low confidence to the track that enforces this constraint. In some embodiments, to verify that what appears to be a ridge is not due to a moving shadow, a 2D track may be performed, and then motion that is not toward the focal point of the expansion may reduce confidence. Alternatively or additionally, in some embodiments, a score may be calculated (e.g., as Figure 3B ), but with a strip of lateral displacement plus or minus a few pixels; a higher score indicates that the motion is not toward the focus of the dilation.

[0240] Third, in some embodiments, the edges of shadows from tree branches may always be soft. This may be due to the finite size of the sun. Shadows from tree branches may fall off along the path of sunlight at edges that are more than about 1 / 100 of the distance from the object to the road surface. Therefore, learning techniques such as neural networks can be used to detect atypical textures of shadows, or atypical textures can be explicitly detected by measuring shadow gradients. In some embodiments, detection of shadows with flexible edges can cause the confidence value of the corresponding point to be reduced.

[0241] In some embodiments, machine learning such as a deep neural network (DNN) can be used to train the system on examples of patches of tree shadows compared to examples of patches that are not tree shadows. In some embodiments, the DNN can be used to train the system on examples of patches of tree shadows in both strips I2 and the distorted I 远 Therefore, it can use not only the texture and motion of the image, but also implicitly the shape of the surface.

[0242] Improved plane detection using image-based free space analysis

[0243] Various techniques can analyze the road scene to determine which parts of the image correspond to the road and which parts of the image do not correspond to the road (e.g., which parts correspond to other things such as cars, pedestrians, curbs, hedges, obstacles, etc.). In some embodiments, such techniques specifically detect the presence of obstacles near the vehicle. For example, if another vehicle is detected, it can be determined that the area of ​​the image corresponding to the other vehicle and some areas around it do not correspond to the road.

[0244] In some embodiments, the system may attempt to determine whether a pixel corresponds to a road; this may be done based on the texture, color, and position in the image of the pixel, and may be performed by a neural network trained on examples of which parts of the image do and do not represent roads.

[0245] In some embodiments, the area of ​​the image determined to correspond to the road surface without any obstacles may be referred to as "free space". In some embodiments of the algorithm for determining the vertical deviation of the road surface and / or for determining the road profile, information about which parts of the image correspond to free space may be used to improve the algorithm.

[0246] In some embodiments, points that are outside the free space region may be ignored when determining inliers for a road model. For example, such points may simply be considered outliers, or they may be eliminated from a certain number of potential inliers (so if 13 points are outside and the remaining 20 inliers are all inliers, the confidence may be quite high). In some embodiments, the grid points may be redistributed in the image so that all points fall in free space. In some embodiments, points along the path that are outside the free space may be given a low confidence when calculating a road profile.

[0247] Although certain embodiments of the present disclosure are presented in the context of DAS applications, some embodiments may be equally applicable to other real-time signal processing applications and / or digital processing applications, such as communications, machine vision, audio and / or speech processing, as examples.

[0248] As used herein, the indefinite articles "a" and "an" (such as "an image") have the meaning of "one or more" (such as "one or more images").

[0249] Although selected features have been shown and described, it should be understood that the invention is not limited to the described features. Instead, it should be recognized that changes can be made to these features without departing from the principles and spirit of the invention, the scope of the invention being defined by the claims and their equivalents.

Claims

1. A method for transmitting road profile information in a driver assistance system installed in a vehicle, wherein: The driver assistance system includes a data interface and a camera operably connected to a processor, the method comprising: Select the first distance in front of the vehicle's wheels, sampling a predetermined number of data points from a road profile along a projected path of the vehicle, wherein the road profile is based at least in part on an image captured by the camera, and wherein the data points are sampled along segments of the path having endpoints corresponding to the selected first distance, and wherein one or more of the data points are generated from a stable reference plane, and The road longitudinal section information corresponding to the predetermined number of data points is transmitted via the data interface in a predetermined number of data transmissions, wherein the number of the data transmissions is less than the number of the data points.

2. The method of claim 1, comprising: A frequency for transmitting data via the data interface is selected, wherein the selected frequency defines a frequency at which the predetermined number of data transmissions are transmitted.

3. The method of claim 2, comprising: A second distance traveled by the vehicle within a time period equal to the inverse of the selected frequency is estimated, wherein the data points are sampled along a segment of the path corresponding to the estimated second distance that exceeds the selected first distance.

4. The method of claim 3, wherein: Estimating the second distance includes determining that the second distance will be traveled within the time period at a known current speed of the vehicle.

5. The method of claim 1, wherein: The road profile information transmitted through the data interface is configured so that the road profile information corresponding to the respective data points can be individually read by a receiving system and associated with the respective portion of the road represented by the road profile.

6. The method of claim 1, wherein: Selecting said first distance in front of the wheels of the vehicle includes accounting for system delays.

7. The method of claim 1, wherein: One or more of the data points are associated with a corresponding confidence value.

8. The method of claim 1, wherein: The spacing of the data points is determined such that it is estimated that the vehicle will travel from one data point to the next within a predetermined time period.

9. The method of claim 1, wherein: One or more of the data points are generated at least in part by combining longitudinal profile information from a plurality of frames.

10. The method of claim 1, further comprising: associating confidence values ​​with points in the road profile, and Data corresponding to points in the road profile and the confidence values ​​associated with the points in the road profile are transmitted to a vehicle control system.

11. The method of claim 10, wherein: The data corresponding to a point in the road profile includes the height of the road.

12. The method of claim 10, wherein: The data corresponding to a point in the road profile includes a distance in front of the vehicle.

13. The method of claim 10, wherein: Associating a confidence value with a point in the road profile includes determining the confidence value based on a forward / backward validation performed between two of the images captured by the camera.

14. The method of claim 10, wherein: Associating a confidence value with a point in the road profile includes determining the confidence value based on a curvature of a score matrix associated with a comparison of two images spaced apart in time.

15. The method of claim 10, comprising: A multi-frame confidence value is transmitted by means of the data corresponding to the point in the road profile and the confidence value associated with the point in the road profile, wherein the multi-frame confidence value is determined based on multiple height measurement results from different images regarding the point in the road profile and the difference of the height measurement results.

16. A driver assistance system installed in a vehicle, the system comprising: processor, a data interface and a camera operably connected to the processor, and A memory storing instructions that, when executed by the processor, cause the system to: Select the first distance in front of the vehicle's wheels, sampling a predetermined number of data points from a road profile along a projected path of the vehicle, wherein the road profile is based at least in part on an image captured by the camera, and wherein the data points are sampled along segments of the path having endpoints corresponding to the selected first distance, and wherein one or more of the data points are generated from a stable reference plane, and The road longitudinal section information corresponding to the predetermined number of data points is transmitted via the data interface in a predetermined number of data transmissions, wherein the number of the data transmissions is less than the number of the data points.

17. The system of claim 16, further comprising instructions for causing the system to: associating confidence values ​​with points in the road profile, and Data corresponding to points in the road profile and the confidence values ​​associated with the points in the road profile are transmitted to a vehicle control system.

18. The system of claim 16, further comprising instructions for causing the system to: A frequency for transmitting data via the data interface is selected, wherein: The selected frequency defines the frequency at which the predetermined number of data transmissions are transmitted.

19. The system of claim 16, wherein: One or more of the data points are associated with a corresponding confidence value.

20. The system of claim 16, wherein: One or more of the data points are generated at least in part by combining road profile information from a plurality of frames.

21. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor operably coupled to a driver assistance system installed in a host vehicle, cause the system to: Select the first distance in front of the vehicle's wheels, A predetermined number of data points are sampled from a road profile along the projected path of the vehicle, and wherein, The road profile is based at least in part on an image captured by a camera operatively connected to the processor, and wherein the data points are sampled along segments of the path having endpoints corresponding to the selected first distance, and wherein one or more of the data points are generated from a stable reference plane, and The road longitudinal section information corresponding to the predetermined number of data points is transmitted in a predetermined number of data transmissions via a data interface, wherein the number of the data transmissions is less than the number of the data points.

22. The non-transitory computer-readable storage medium of claim 21, further comprising instructions for causing a system to: associating confidence values ​​with points in the road profile, and Data corresponding to points in the road profile and the confidence values ​​associated with the points in the road profile are transmitted to a vehicle control system.

23. A method for managing the transmission of road profile information to a suspension controller of a vehicle via a data interface, the method comprising: Obtaining a first distance in front of a wheel of the vehicle; obtaining a frequency of transmitting data through the data interface; estimating a second distance traveled by the vehicle within a time period equal to the inverse of the frequency; sampling a predetermined number of data points from a road profile along a projected path of the vehicle based at least on the frequency and the second distance, wherein the data points are sampled along a segment of the path defined by the second distance that exceeds the first distance; as well as The road longitudinal section information corresponding to the predetermined number of data points is transmitted through the data interface.

24. The method of claim 23, wherein: The first distance is adjusted to account for computation time or transmission delay, along with vehicle speed.

25. The method of claim 24, wherein: The first distance is set to be a set distance from a vehicle when the data is received by the suspension controller.

26. The method of claim 24, wherein: The first distance is derived from a set time that the vehicle will pass through a first point from when the data is received by the suspension controller.

27. The method of claim 24, further comprising estimating a delay from a capture time in the road profile calculation model module to a reception time at the transmission module.

28. The method of claim 27, wherein: The road profile computational model module sends an oversampled set of data points from the road profile, and the transmission module estimates the delay and transmits a subset or a resampled set of points from the road profile.

29. The method of claim 23, wherein: The driver assistance system includes the data interface and a camera operably connected to the processor.

30. The method of claim 23, further comprising controlling an adaptive suspension system using the road profile information.

31. A driver assistance system installed in a vehicle, the system comprising: processor, a data interface and a camera operably connected to the processor, and A memory storing instructions that, when executed by the processor, cause the system to: obtaining a first distance in front of a wheel of a vehicle; Obtaining a frequency for transmitting data via a data interface; estimating a second distance traveled by the vehicle within a time period equal to the inverse of the frequency; sampling a predetermined number of data points from a road profile along a projected path of the vehicle based at least on the frequency and the estimated second distance, wherein the data points are sampled along a segment of the path defined by the estimated second distance that exceeds the first distance; as well as The road longitudinal section information corresponding to the predetermined number of data points is transmitted through the data interface.

32. The system of claim 31, wherein: The first distance is adjusted to account for computation time or transmission delay, along with vehicle speed.

33. The system of claim 32, wherein: The first distance is set to be a set distance from the vehicle when the data is received by the suspension controller.

34. The system of claim 32, wherein: The first distance is derived from a set time at which the vehicle will pass through a first point from when the data is received by the suspension controller.

35. The system of claim 32, further comprising a transmission module to estimate a delay from a capture time in the road profile calculation model module to a reception time at the transmission module.

36. The system of claim 35, wherein: The road profile computational model module sends an oversampled set of data points from the road profile, and the transmission module estimates the delay and transmits a subset or a resampled set of points from the road profile.

37. The system of claim 31 further comprising an adaptive suspension system, and wherein: The memory is configured to transmit the road profile information for controlling the adaptive suspension system.

38. A non-transitory computer-readable storage medium storing instructions, which, when executed by a processor having a data interface installed in a vehicle, causes the processor to perform the following operations: Obtaining a first distance in front of a wheel of the vehicle; obtaining a frequency of transmitting data through the data interface; estimating a second distance traveled by the vehicle within a time period equal to the inverse of the frequency; sampling a predetermined number of data points from a road profile along a projected path of the vehicle based at least on the frequency and the estimated second distance, wherein the data points are sampled along a segment of the path defined by the estimated second distance that exceeds the first distance; as well as The road longitudinal section information corresponding to the predetermined number of data points is transmitted to the suspension controller via the data interface.

39. The non-transitory computer-readable storage medium of claim 38, wherein: The first distance is adjusted to account for computation time or transmission delay, along with vehicle speed.

40. The non-transitory computer-readable storage medium of claim 39, wherein: The first distance is set to a set distance from a vehicle when data is received by the suspension controller.

41. The non-transitory computer-readable storage medium of claim 39, wherein: The first distance is derived from a set time at which the vehicle will pass through a first point from the time the data is received from the suspension controller.

42. The non-transitory computer readable storage medium of claim 39, further comprising instructions for estimating a delay from a capture time in the road profile computational model module to a reception time at the transmission module.

43. The non-transitory computer readable storage medium of claim 42, wherein: The road profile computational model module sends an oversampled set of data points from the road profile, and the transmission module estimates the delay and transmits a subset or a resampled set of points from the road profile.

44. The non-transitory computer-readable storage medium of claim 38, further comprising instructions for transmitting the road profile information for use in controlling an adaptive suspension system.

Images