Method, system, and storage medium for processing road surface visual data

Abstract

The present application relates to methods, systems, and storage media for processing road surface vision data. The present disclosure generally relates to processing vision data of a road surface that includes a vertical deviation with a lateral slope. In some embodiments, a system determines a path that is projected to be traversed by at least one wheel of a vehicle on a road surface. In some embodiments, the system determines a height of the road surface at at least one point along the path that is to be traversed by the wheel using at least two images captured by one or more cameras. In some embodiments, the system calculates an indication of a lateral slope of the road surface at the at least one point along the path. In some embodiments, the system outputs the indication of the height of the point and the indication of the lateral slope at the at least one point along the path on a vehicle interface bus.

Description

[0001] This application is a divisional application of the application filed on March 15, 2017, with application number 201780029603.6, entitled "Method, System and Storage Medium for Processing Visual Data of Road Surface" (originally entitled "Road Plane Output with Lateral Slope").

[0002] Cross-references to related applications

[0003] This application claims the benefit of U.S. Provisional Application No. 62 / 308,631, filed March 15, 2016, entitled “Road Plane Output with LateralSlope,” the entire contents of which are incorporated herein by reference for all purposes.

[0004] This application relates to U.S. Application No. 14 / 554,500 (now U.S. Patent No. 9,256,791), filed November 26, 2014, entitled "Road Vertical Contour Detection"; U.S. Application No. 14 / 798,575, filed July 14, 2015, entitled "Road Contour Vertical Detection"; U.S. Provisional Application No. 62 / 120,929, filed February 26, 2015, entitled "Road Plane Profile Output in a Stabilized World Coordinate Frame"; U.S. Provisional Application No. 62 / 131,374, filed March 11, 2015, entitled "Road Plane Profile Output in a Stabilized World Coordinate Frame"; and U.S. Provisional Application No. 62 / 131,374, filed April 20, 2015, entitled "Road Plane Profile Output in a Stabilized World Coordinate Frame". The entire contents of U.S. Provisional Application No. 62 / 149,699 entitled “Road Plane Output in a Stabilized World Coordinate Frame”, filed October 8, 2015; and U.S. Provisional Application No. 62 / 238,980 entitled “Road Plane Output in a Stabilized World Coordinate Frame”, filed February 26, 2016, entitled “Road VerticalContour Detection Using a Stabilized Coordinate Frame”, are incorporated herein by reference for all purposes. Technical Field

[0005] This disclosure generally relates to driver assistance systems, and more specifically, to the detection of features of the road surface. Background Technology

[0006] In recent years, the use of Driver Assistance Systems (DAS) has increased significantly and will only continue to do so. A DAS can be a hardware and / or software component that assists in driving or maneuvering a vehicle. In some cases, DAS can achieve fully autonomous control of the vehicle (e.g., no driver intervention required during operation) or semi-autonomous control (e.g., some driver intervention required during operation). In some cases, DAS can be coupled with driver control (e.g., making minor corrections or providing useful information about road conditions). In some cases, the control level of DAS can be changed (e.g., by the driver) to fully autonomous, semi-autonomous, coupled with driver control, or disabled. Some examples of DAS functions include Lane Departure Warning (LDW), Automatic High Beam Control (AHC), Traffic Sign Recognition (TSR), and Forward Collision Warning (FCW).

[0007] Many DAS systems rely on one or more cameras to capture images of the vehicle's surroundings, for example, to determine features in the road plane (e.g., an image plane that images the road surface). Some techniques used to determine features in the road plane cannot correctly detect features under certain conditions. As a result, a vehicle traveling under DAS system control may be unable to make appropriate adjustments to address the actual road plane features, thus adversely affecting ride comfort and / or safety. Summary of the Invention

[0008] Therefore, the technique presented in this paper allows for improved determination of features in road planes, particularly features with lateral slopes.

[0009] In some embodiments, a method for processing visual data of a road surface is performed, the method comprising: accessing visual data representing the road surface; determining an initial road profile of the road surface, wherein the road profile is derived from residual motion along a vehicle path associated with the road surface; segmenting a portion of the visual data representing the road surface into a first segment and a second segment, wherein the segmented portion of the visual data includes visual data representing vertical deviations on the road surface; determining a first segment road profile of the first segment of the portion of the visual data; determining a second segment road profile of the second segment of the portion of the visual data; comparing one or more of the first segment road profile, the second segment road profile, and the initial road profile; and outputting an indication of the lateral slope of the vertical deviations on the road surface, at least in part based on the result of the comparison.

[0010] In some embodiments, a system for processing visual data of a road surface includes: one or more processors; a memory; and one or more programs, wherein one or more programs are stored in the memory and configured to be executed by one or more processors, the programs including instructions for: accessing visual data representing the road surface; determining an initial road profile of the road surface, wherein the road profile is derived from residual motion along a vehicle path associated with the road surface; segmenting a portion of the visual data representing the road surface into a first segment and a second segment, wherein the segmented portion of the visual data includes visual data representing vertical deviations on the road surface; determining a first segment road profile of the first segment of the portion of the visual data; determining a second segment road profile of the second segment of the portion of the visual data; comparing one or more of the first segment road profile, the second segment road profile, and the initial road profile; and outputting an indication of the lateral slope of the vertical deviations on the road surface, at least in part based on the result of the comparison.

[0011] In some embodiments, a non-transitory computer-readable storage medium stores one or more programs, which include instructions that, when executed by one or more processors of the vehicle, cause the processors to perform the following operations: access visual data representing a road surface; determine an initial road profile of the road surface, wherein the road profile is derived from residual motion along a vehicle path associated with the road surface; segment a portion of the visual data representing the road surface into a first segment and a second segment, wherein the segmented portion of the visual data includes visual data representing vertical deviations on the road surface; determine a first segment road profile of the first segment of the portion of the visual data; determine a second segment road profile of the second segment of the portion of the visual data; compare one or more of the first segment road profile, the second segment road profile, and the initial road profile; and output an indication of the lateral slope of the vertical deviations on the road surface, at least in part based on the result of the comparison.

[0012] This application provides the following:

[0013] 1) A method for processing visual data of road surfaces, the method comprising:

[0014] Determine the path on the road surface that is expected to be traversed by at least one wheel of a vehicle;

[0015] Using at least two images captured by one or more cameras on the vehicle, determine the height of the road surface at at least one point along the path the wheels will traverse;

[0016] Calculate an indication of the lateral slope of the road surface at at least one point along the path; and

[0017] The vehicle interface bus on the vehicle outputs an indication of the height of the point and an indication of the lateral slope at at least one point along the path.

[0018] 2) The method described in 1) further includes:

[0019] Access visual data representing the road surface;

[0020] An initial road profile is determined for the road surface, wherein the road profile is derived from residual motion along a vehicle path associated with the road surface;

[0021] A portion of the visual data representing the road surface is divided into a first segment and a second segment, wherein the segmented portion of the visual data includes visual data representing vertical deviations on the road surface.

[0022] Determine the first segment of the road outline of the first segment of the portion of the visual data;

[0023] Determine the second segment of the road outline of the second segment of the portion of the visual data;

[0024] Compare one or more of the first road profile, the second road profile, and the initial road profile; and

[0025] Based at least in part on the results of the comparison, an indication of the height of the point and an indication of the lateral slope at the at least one point along the path are output.

[0026] 3) The method according to 2), wherein outputting an indication of the height of the point and an indication of the lateral slope at the at least one point along the path includes outputting one or more of the following:

[0027] The slope measurement associated with the vertical deviation on the road surface;

[0028] The direction of the slope associated with the vertical deviation on the road surface; and

[0029] A confidence value for the residual motion or height value associated with the road profile and the vertical deviation on the road surface.

[0030] 4) The method according to 2), wherein the first road profile, the second road profile, and the initial road profile each include data derived from residual motion along a path associated with the road surface, and wherein comparing the road profiles includes:

[0031] Determine whether the residual motion at points along the path of each road contour is within each other's threshold.

[0032] 5) The method described in 2) further includes:

[0033] The lateral slope of the vertical deviation is determined at least in part based on the residual motion data of the road profile of the first segment and the residual motion data of the road profile of the second segment.

[0034] 6) According to the method described in 5), determining the transverse slope includes:

[0035] The estimated height of the vertical deviation is determined based on one or more of the first road segment profile, the second road segment profile, and the initial road profile; and

[0036] The lateral slope is calculated at least in part based on the estimated height.

[0037] 7) The method according to 2) further includes:

[0038] The confidence score of one or more road profiles is determined based in part on the comparison of the first road profile segment, the second road profile segment, and the initial road profile.

[0039] 8) According to the method described in 2), wherein determining the road outline includes:

[0040] Determine the residual motion of the pixel sequence in the visual data representing the road surface.

[0041] 9) The method according to 8), wherein the pixel sequence is a row of pixels along the center of the visual data representing the road contour.

[0042] 10) The method according to 2), wherein the first segment of the portion of the visual data is different from the second segment of the portion of the visual data.

[0043] 11) The method according to 2) further includes:

[0044] The vehicle component settings are adjusted in part based on the comparison results of the first road profile, the second road profile, and the initial road profile.

[0045] 12) According to the method described in 11), adjusting the vehicle component settings includes:

[0046] Adjust the vehicle steering settings to modify the vehicle path, wherein the modified vehicle path avoids the vertical deviation on the road surface.

[0047] 13) According to the method of 11), adjusting the vehicle component settings includes adjusting the vehicle suspension settings.

[0048] 14) The method according to 2) further includes:

[0049] Determine the vertical deviation of the vehicle path across the road surface, wherein the vertical deviation is represented by the raised residual motion in the initial road profile; and

[0050] Based on the determination of the vertical deviation of the vehicle path along the road surface:

[0051] The portion representing the visual data of the road surface is divided into the first segment and the second segment.

[0052] 15) A system installed on a vehicle, the system comprising:

[0053] One or more cameras; and

[0054] One or more processors, wherein the one or more processors are configured to execute instructions for:

[0055] Determine the path on the road surface that is expected to be traversed by at least one wheel of the vehicle;

[0056] The height of the road surface at at least one point along the path that the wheel will traverse is determined using at least two images captured by the one or more cameras.

[0057] Calculate an indication of the lateral slope of the road surface at at least one point along the path; and

[0058] The vehicle interface bus on the vehicle outputs an indication of the height of the point and an indication of the lateral slope at at least one point along the path.

[0059] 16) According to the system described in 15), the processor is further configured to execute instructions for:

[0060] Access visual data representing the road surface;

[0061] An initial road profile is determined for the road surface, wherein the road profile is derived from residual motion along a vehicle path associated with the road surface;

[0062] A portion of the visual data representing the road surface is divided into a first segment and a second segment, wherein the segmented portion of the visual data includes visual data representing vertical deviations on the road surface;

[0063] Determine the first segment of the road outline of the first segment of the portion of the visual data;

[0064] Determine the second segment of the road outline of the second segment of the portion of the visual data;

[0065] Compare one or more of the first road profile, the second road profile, and the initial road profile; and

[0066] Based at least in part on the results of the comparison, an indication of the height of the point and an indication of the lateral slope at the at least one point along the path are output.

[0067] 17) The system according to 16), wherein outputting an indication of the height of the point and an indication of the lateral slope at the at least one point along the path includes outputting one or more of the following:

[0068] The slope measurement associated with the vertical deviation on the road surface;

[0069] The direction of the slope associated with the vertical deviation on the road surface; and

[0070] A confidence value for the residual motion or height value associated with the road profile and the vertical deviation on the road surface.

[0071] 18) The system according to 16), wherein the second road profile and the initial road profile each include data derived from residual motion along a path associated with the road surface, and wherein comparing the road profiles includes:

[0072] Determine whether the residual motion at points along the path of each road contour is within each other's threshold.

[0073] 19) According to the system of 16), the one or more programs further include instructions for the following operations:

[0074] The lateral slope of the vertical deviation is determined at least in part based on the residual motion data of the road profile of the first segment and the residual motion data of the road profile of the second segment.

[0075] 20) According to the system described in 19), determining the lateral slope includes:

[0076] The estimated height of the vertical deviation is determined based on one or more of the first road segment profile, the second road segment profile, and the initial road profile; and

[0077] The lateral slope is calculated at least in part based on the estimated height.

[0078] 21) According to the system of 16), the one or more programs further include instructions for the following operations:

[0079] The confidence score of one or more road profiles is determined based in part on the comparison of the first road profile segment, the second road profile segment, and the initial road profile.

[0080] 22) According to the system described in 16), determining the road contour includes:

[0081] Determine the residual motion of the pixel sequence in the visual data representing the road surface.

[0082] 23) The system according to 22), wherein the pixel sequence is a row of pixels along the center of the visual data representing the road contour.

[0083] 24) The system according to 16), wherein the first segment of the portion of the visual data is different from the second segment of the portion of the visual data.

[0084] 25) According to the system of 16), the one or more programs further include instructions for the following operations:

[0085] The vehicle component settings are adjusted in part based on the comparison results of the first road profile, the second road profile, and the initial road profile.

[0086] 26) According to the system described in 25), adjusting the vehicle component settings includes:

[0087] Adjust the vehicle steering settings to modify the vehicle path, wherein the modified vehicle path avoids the vertical deviation on the road surface.

[0088] 27) According to the system described in 25), adjusting the vehicle component settings includes adjusting the vehicle suspension settings.

[0089] 28) According to the system of 16), the one or more programs further include instructions for the following operations:

[0090] Determine the vertical deviation of the vehicle path across the road surface, wherein the vertical deviation is represented by the raised residual motion in the initial road profile; and

[0091] Based on the determination of the vertical deviation of the vehicle path along the road surface:

[0092] The portion representing the visual data of the road surface is divided into the first segment and the second segment.

[0093] 29) A non-transitory computer-readable storage medium storing one or more programs, said one or more programs comprising instructions that, when executed by one or more processors of a vehicle, cause the processor to perform the following operations:

[0094] Determine the path on the road surface that is expected to be traversed by at least one wheel of the vehicle;

[0095] Using at least two images captured by one or more cameras on the vehicle, determine the height of the road surface at at least one point along the path the wheels will traverse;

[0096] Calculate an indication of the lateral slope of the road surface at at least one point along the path; and

[0097] The vehicle interface bus on the vehicle outputs an indication of the height of the point and an indication of the lateral slope at at least one point along the path.

[0098] 30) According to the storage medium of 29), the one or more programs further include instructions for the following operations:

[0099] Access visual data representing the road surface;

[0100] An initial road profile is determined for the road surface, wherein the road profile is derived from residual motion along a vehicle path associated with the road surface;

[0101] A portion of the visual data representing the road surface is divided into a first segment and a second segment, wherein the segmented portion of the visual data includes visual data representing vertical deviations on the road surface.

[0102] Determine the first segment of the road outline of the first segment of the portion of the visual data;

[0103] Determine the second segment of the road outline of the second segment of the portion of the visual data;

[0104] Compare one or more of the first road profile, the second road profile, and the initial road profile; and

[0105] Based at least in part on the results of the comparison, an indication of the height of the point and an indication of the lateral slope at the at least one point along the path are output.

[0106] 31) The storage medium according to 30), wherein outputting an indication of the height of the point and an indication of the lateral slope at the at least one point along the path includes outputting one or more of the following:

[0107] The slope measurement associated with the vertical deviation on the road surface;

[0108] The direction of the slope associated with the vertical deviation on the road surface; and

[0109] A confidence value for the residual motion or height value associated with the road profile and the vertical deviation on the road surface.

[0110] 32) The storage medium according to 30), wherein the second road profile and the initial road profile each include data derived from residual motion along a path associated with the road surface, and wherein comparing the road profiles includes:

[0111] Determine whether the residual motion at points along the path of each road contour is within each other's threshold.

[0112] 33) According to the storage medium of 30), the one or more programs further include instructions for the following operations:

[0113] The lateral slope of the vertical deviation is determined at least in part based on the residual motion data of the road profile of the first segment and the residual motion data of the road profile of the second segment.

[0114] 34) According to the storage medium described in 33), determining the lateral slope includes:

[0115] The estimated height of the vertical deviation is determined based on one or more of the first road segment profile, the second road segment profile, and the initial road profile; and

[0116] The lateral slope is calculated at least in part based on the estimated height.

[0117] 35) According to the storage medium of 30), the one or more programs further include instructions for the following operations:

[0118] The confidence score of one or more road profiles is determined based in part on the comparison of the first road profile segment, the second road profile segment, and the initial road profile.

[0119] 36) According to the storage medium of 30), wherein determining the road outline includes:

[0120] Determine the residual motion of the pixel sequence in the visual data representing the road surface.

[0121] 37) The storage medium according to 36), wherein the pixel sequence is a row of pixels along the center of the visual data representing the road contour.

[0122] 38) The storage medium according to 30), wherein the first segment of the portion of the visual data is different from the second segment of the portion of the visual data.

[0123] 39) According to the storage medium of 30), the one or more programs further include instructions for the following operations:

[0124] The vehicle component settings are adjusted in part based on the comparison results of the first road profile, the second road profile, and the initial road profile.

[0125] 40) According to the storage medium described in 39), adjusting the vehicle component settings includes:

[0126] Adjust the vehicle steering settings to modify the vehicle path, wherein the modified vehicle path avoids the vertical deviation on the road surface.

[0127] 41) According to the storage medium described in 39), adjusting the vehicle component settings includes adjusting the vehicle suspension settings.

[0128] 42) According to the storage medium of 30), the one or more programs further include instructions for the following operations:

[0129] Determine the vertical deviation of the vehicle path across the road surface, wherein the vertical deviation is represented by the raised residual motion in the initial road profile; and

[0130] Based on the determination of the vertical deviation of the vehicle path along the road surface:

[0131] The portion representing the visual data of the road surface is divided into the first segment and the second segment. Attached Figure Description

[0132] Figure 1 An exemplary point grid for calculating homography is depicted.

[0133] Figure 2 A camera view is depicted, which includes the residual motion of points along the predicted wheel path, as well as a graph of metric height and distance.

[0134] Figure 3 The normalized correlation scores of points along the wheel path are plotted.

[0135] Figure 4A A camera view is depicted, in which the overlapping right wheel path passes through the center of the speed bump.

[0136] Figure 4B A camera view is depicted, in which the overlapping right wheel path passes the edge of the speed bump.

[0137] Figure 5A The image strip depicts the road profile used to calculate the road profile when the path is above the center of the speed bump.

[0138] Figure 5B An image strip depicting the road profile was used to calculate the road contour when the path was above the edge of the speed bump.

[0139] Figure 6 A graph depicting the residual motion of the vehicle path calculated for the image band when the path is above the center of the speed bump.

[0140] Figure 7 A camera view is depicted, in which the overlapping right wheel path passes the edge of the speed bump.

[0141] Figure 8 An image strip depicting the road profile was used to calculate the road contour when the path was above the edge of the speed bump.

[0142] Figure 9 The image strip depicts the segmentation used to calculate the road profile when the path is above the edge of the speed bump.

[0143] Figure 10A The graph depicts the residual motion of the vehicle path calculated for the entire image band and for the left and right portions of the band, respectively, when the path is above the center of the speed bump.

[0144] Figure 10B The graph depicts the residual motion of the vehicle path calculated for the entire image band and for the left and right portions of the band, respectively, when the path is above the edge of the speed bump.

[0145] Figure 11 A flowchart illustrating an exemplary process for processing visual data of a road surface is depicted.

[0146] Figure 12 A flowchart illustrating an exemplary process for processing visual data of a road surface is depicted.

[0147] Figure 13 An exemplary system for processing visual data of road surfaces is described. Detailed Implementation

[0148] The following description is presented to enable those skilled in the art to make and use various embodiments. The descriptions of specific devices, techniques, and applications are provided by way of example only. Various modifications to the examples described herein will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other examples and applications without departing from the spirit and scope of the various embodiments. Therefore, the various embodiments are not intended to be limited to the examples described and illustrated herein, but are consistent with the scope of the claims.

[0149] 1. Overview

[0150] As briefly described above, driver assistance systems can rely on one or more imaging devices to collect visual data of the road plane around the vehicle. This visual data can then be used to determine the structural characteristics of the vehicle's environment, including features of the road surface, objects or obstacles on the road surface, or other vertical deviations of the road surface. The term "plane" as used herein is not limited to a purely geometric plane (i.e., flat, two-dimensional), but is used in a general sense. Thus, for example, a "road plane" (also referred to as a "road surface") can be substantially flat, but includes three-dimensional components.

[0151] Some existing methods for detecting the vertical deviation of a road profile using onboard cameras are known. Some previously known algorithms can be summarized as follows:

[0152] 1. Use the homography of the road to align the first pair of consecutive frames captured by the vehicle-mounted camera. This gives the self-motion (rotation R and translation T) between frames. For example, the point grid in the image (e.g., as shown in the image). Figure 1 (As shown) is tracked and used to calculate homography.

[0153] 2. Then, a second pair of frames is selected, the current frame and the most recent previous frame representing the point in time when the vehicle has moved more than the minimum threshold distance. The linking of the first pair of frames (continuous frames) is used to create an initial guess of homography, then the more accurate homography of the second pair of frames is calculated, and the reference plane is determined.

[0154] 3. Then project the vehicle's path onto the image plane (e.g., Figure 2 (As shown by the lines in the diagram). The strips along each path are used to calculate the residual motion, which gives the profile relative to a defined reference plane. Figure 3 The normalized correlation score is shown for a 31-pixel wide band along the path of the left wheel, with a vertical motion of ±6 pixels. A small curvature, indicated by the arrow, can be seen. This curvature represents the small residual motion of the speed bump.

[0155] 4. Finally, the residual motion was converted into metric distance and height, and combined into a multi-frame model. The results are in... Figure 2 As shown in the upper middle figure.

[0156] Figure 1 , Figure 2 , Figure 3 The use of visual data on speed bumps on a road surface is shown. Figure 1 A grid 102 consisting of 33 points overlaid on an image is shown; these points are tracked and used to calculate homography. In some examples, the homography of roads is used to align two consecutive image frames. In other examples, non-consecutive image frames can be used. For example, two image frames captured several frames apart (e.g., with one or more intermediate captured frames between the two image frames) can be used to provide a larger baseline for homography. Aligning image frames is also known as warping the second (earlier) image frame toward the first image frame. Homography can be used to determine structure from motion data using image frames.

[0157] In some examples, the camera's ego-motion (rotation R and translation T) between frames is also determined using homography. In some implementations, one or more sensors on the vehicle are used to determine ego-motion. In some implementations, one or more image processing techniques are used to determine ego-motion. For example, image processing may include calculating ego-motion from two or more image frames and a road model. In some examples, it is assumed that the road model has a flat (0% slope) road surface ([0,0,1] is good for normal vehicle speed and calibrated camera height), but other road models may also be used. Examples of other road models include, but are not limited to, 1% slope roads, 2% slope roads, etc. More complex road surfaces can be modeled and used, including road models with varying road slopes, road models with uneven road surfaces, etc. Given a road model, image points can be projected onto 3D points on the road model.

[0158] As used herein, unless otherwise stated, “on a vehicle” means attachment, placement, positioning, installation, etc., inside or outside the vehicle (e.g., attached to the body) or otherwise in contact with a part of the vehicle. For example, components mounted on the windshield inside the vehicle are considered to be within the scope of the phrase “on a vehicle,” as are components fixed or otherwise mounted outside the vehicle.

[0159] In some examples, the points in the grid can be evenly spaced in the image, which makes more points closer to the vehicle. In some examples, the points can be evenly spaced at a certain distance, which will make the points more concentrated in the higher parts of the image. In other examples, the points are evenly spaced vertically in the image, and, for example, in the least squares calculation of homography, points that are farther away are given greater weight.

[0160] In some implementations, the system accesses two image frames where the vehicle has translated forward beyond a minimum threshold, and uses the linking of consecutive frames to determine initial homography. More precise homography is then calculated, and a reference plane is determined. In some implementations, the system accesses multiple image frames in addition to two. For example, any number of image frames can be used to determine initial homography and / or more precise homography (also known as fine homography).

[0161] Figure 2 Exemplary visual data of the road surface captured by the imaging device of a driver assistance system is shown. Figure 2 This includes vehicle path data projected onto an image plane (also known as the road plane) onto the road surface. Vehicle paths 202 and 204 represent the corresponding paths of the left and right wheels of the vehicle and are depicted as dashed lines overlaid on the image. A strip along each vehicle path is used to calculate residual motion. The residual motion can be used to represent the contour of the road relative to the image plane (e.g., a reference plane) that images the road surface.

[0162] Vehicle paths such as 202 and 204 are virtual representations of the actual path a vehicle has traveled or will travel on the road surface. In some implementations, the “virtual” vehicle path has a width of one or more pixels. In some examples, the width of the vehicle path is a constant number of units (e.g., pixels). In other examples, the vehicle path has a non-constant width. In some implementations, the width of the “virtual” vehicle path is related to the width of the actual path. For example, the width associated with the actual path may be based on the actual vehicle tire width (or other contact point between the vehicle and the road surface), may be based on the average tire width, may be predetermined based on user input, may be the centerline of the tire track (independent of tire width), or may be based on any other suitable value.

[0163] Figure 3 The normalized correlation score is shown for a 31-pixel wide image band along the vehicle path (path 202) of the left wheel, for vertical motion ±6 pixels. (Examples of related data are omitted as they are not relevant to the translation.) Figure 3 The normalized correlation score shown can indicate the deviation or shift (e.g., in pixels) required to align each given row of the second image frame with the first image frame. Figure 3 In the diagram, darker areas represent higher correlation, where the vertical axis (e.g., the y-axis) represents pixel rows (each 31 pixels wide), and the horizontal axis (x-axis) represents vertical motion in pixels. A slight curvature is visible in the darker areas. Line 302 is overlaid on top of the normalized correlation score for reference. Using line 302 as a reference, a small curvature (indicated by arrow 304) can be seen. This curvature indicates residual motion and vertical deviations on the imaged road surface, such as contours or obstacles (e.g., speed bumps).

[0164] Techniques for determining the presence of a road vertical profile are described in more detail in: U.S. Application No. 14 / 554,500 (now U.S. Patent No. 9,256,791), filed November 26, 2014, entitled “Road Vertical Contour Detection”; U.S. Application No. 14 / 798,575, filed July 14, 2015, entitled “Road Contour Vertical Detection”; and U.S. Application No. 15 / 055,322, filed February 26, 2016, entitled “Road Vertical Contour Detection Using a Stabilized Coordinate Frame”, the contents of which are incorporated herein by reference.

[0165] In some implementations, the residual motion along one or more vehicle paths is converted into metric distance and metric height and combined into a multi-frame model (also known as a multi-frame profile). A multi-frame model is a model created by aligning multiple image frames (e.g., frames representing different moments when a vehicle traverses a portion of the road) with a global model to combine road profiles corresponding to overlapping road sections. In this example, respectively in Figure 2 The graphs in the upper part—graphs 206 and 208—show exemplary road profiles for left-wheel and right-wheel vehicle paths. In some implementations, one or more road profiles (individual or combined into multi-frame models) are transmitted via a communication bus, such as a controller area network (CAN) bus.

[0166] Additional descriptions and algorithms for determining multi-frame models are provided in: U.S. Application No. 14 / 554,500 (now U.S. Patent No. 9,256,791), filed November 26, 2014, entitled "Road Vertical Contour Detection"; and U.S. Application No. 14 / 798,575, filed July 14, 2015, entitled "Road Contour Vertical Detection," the contents of which are incorporated herein by reference.

[0167] A stable world reference can be used to provide a consistent road profile represented as a 1D profile with a very compact representation. As described later in this paper, lateral slope calculations can be used with either representation: a full profile or a compact 1D profile.

[0168] The following includes a brief description of the calculation of a stable world reference; however, further description is found in U.S. Application No. 15 / 055,322, filed February 26, 2016, entitled “Road Vertical Contour Detection Using a Stabilized Coordinate Frame,” the contents of which are incorporated herein by reference. In some embodiments, a stable world coordinate system is calculated based at least on a combination of (a) a reference plane calculated according to the current frame, (b) a reference plane calculated according to one or more previous frames, and (c) an assumed default reference plane based at least in part on the fixed position of the camera relative to the vehicle. In some embodiments, the combination of these factors can produce a stable world coordinate system used to compare frames corresponding to different time steps in order to calculate residual motion and identify vertical deviations of the road surface.

[0169] A brief description of the contour generation using 1D sampling techniques is included below; however, further description is found in U.S. Application No. 15 / 055,322, filed February 26, 2016, entitled “Road Vertical Contour Detection Using a Stabilized Coordinate Frame,” the contents of which are incorporated herein by reference.

[0170] In some implementations, transmitting data about the road profile of a portion of the road ahead of the vehicle may require transmitting a large number of values. For example, outputting a road profile of 5 to 12 meters ahead of the vehicle, sampled every 0.05 meters, requires 140 values; for longer profiles, the number of values ​​required is obviously much greater. The potentially large number of values ​​may be problematic for a Controller Area Network (CAN) and may require significant computational resources at the receiving end. For example, in some implementations, 200 data points may be transmitted per frame per round, which may require transmitting more than 1KB of data per frame. In some implementations, transmitting this amount of data per frame may be infeasible and / or computationally too expensive for the receiving system.

[0171] Furthermore, some DAS and self-driving systems require receiving data at a predefined frame rate (e.g., 100Hz), so that the system calls a data point corresponding to the road height (at a predetermined distance in front of the vehicle) every 10 milliseconds. In some implementations, sending data to the system every 10 milliseconds may be impractical because it could monopolize the data channel in the vehicle, preventing other information from being sent through the same data channel.

[0172] Therefore, it is advantageous to transmit the calculated effective amount of data about the road profile in a manner that (a) the total amount of data sent is manageable and (b) the data transmission does not monopolize the data channel at all times.

[0173] In some implementations, a data format is provided for transmitting information about the road profile, wherein the road height is output at a specific distance in front of the wheels. In some implementations, this distance is a fixed distance (e.g., 7 meters), and in some other implementations, this distance can be determined dynamically. For example, the distance in front of the wheels can be dynamically determined based on the vehicle's speed, thereby increasing the reference distance at higher speeds. In some implementations, the distance can be set based on the distance covered by the vehicle in a given amount of time (e.g., the distance covered by the vehicle in 0.5 seconds at its current speed). Instead of transmitting the entire profile or road height-related profile data along the entire known wheel path at every time step, only transmitting data corresponding to a fixed distance in front of the wheels allows for the transmission of less total data, which can save computational resources and bandwidth.

[0174] In some implementations, a data format is also provided in which multiple data points are transmitted substantially simultaneously, rather than just a single data point, to effectively multiply the frame rate. For example, if the frame rate is 10Hz, in some implementations, the system can simulate a 100Hz data output by transmitting 10 data points at a time at the actual 10Hz frequency in a burst. After transmission, the receiving component can unpack the 10 data points and query one of them at a time as needed. In some implementations, all data points corresponding to a single frame can be transmitted in a single data transmission, while in other implementations, they can be transmitted in multiple transmissions with a number fewer than the number of data points per frame (e.g., 7 data points per frame transmitted only in two CAN messages). In some implementations, each burst can transmit fewer than 10 data points. In some implementations, each burst can transmit 16 data points.

[0175] A compact 1D profile can be determined from the burst of data points. In some implementations, the system can determine the distance the main vehicle is expected to travel within a time period corresponding to the frequency at which data will be transmitted. In some implementations, the system can determine the distance along the projected path of the wheels. n The road profile was sampled at several points, among which... nThis refers to the number of data points per burst. In some implementations, the points sampling the road profile can be spaced between a selected distance in front of the wheels and that distance plus an estimated distance the vehicle will travel before sending the next data burst (e.g., they can span the selected distance in front of the wheels, or they can be evenly spaced between them). In some implementations, the road profile can be sampled at points along a path segment that extends beyond the selected distance in front of the wheels plus the estimated distance the vehicle will travel before sending the next data burst; in this way, there may be some overlap between road portions corresponding to data bursts from different frames, resulting in redundancy, which in some implementations can improve the accuracy of vertical deviation calculations.

[0176] Now, let’s turn our attention to some examples illustrating specific real-world situations that driver assistance systems might encounter. In short, the first example presented below illustrates a simple case where the vehicle path (e.g., the path of the wheels) passes through the center of a ridge on the road surface, where the ridge has a generally uniform lateral height (e.g., the entire width of the wheel will traverse the height of the ridge). The second example given below illustrates a more difficult case for driver assistance systems where the vehicle path passes through a portion of a road surface with a lateral slope (e.g., the edge of a ridge or other obstacle). In this second example, a typical vehicle navigation system might simply treat the portion with the lateral slope as a simple ridge (e.g., the same as in the first example), or not treat it as a ridge at all (e.g., the average detection height of the ridge across the wheel width would be lower than in the first example, which might not cause the ridge to be registered as a significant obstacle by the driver assistance system).

[0177] Example 1: Vehicle path that completely traverses obstacles

[0178] Figure 4A , Figure 4B , Figure 5A , Figure 5B , Figure 6 Visual data and measurement results for a first example are shown. As described above, this first example illustrates a situation where the driver assistance system detects that the vehicle path (e.g., the path of the wheels) passes through the center of an obstacle (e.g., a ridge) with approximately uniform lateral height on both sides of the vehicle path.

[0179] Figure 4A The visual data received by the driver assistance system is shown. The visual data includes a representation of the road surface 402 on which the vehicle is traveling. Figure 4A It also includes a vehicle path 404 corresponding to the right wheel of the vehicle, superimposed on the visual data. Included on the vehicle path 404 is a measure of the residual motion of the road surface along the vehicle path. Figure 4ASpeed ​​bumps 406 on road surface 402 are also depicted. As can be seen, vehicle path 404 passes completely over bump 406, which appears to have a roughly uniform lateral height on each side of vehicle path 404. Notably, as vehicle path 404 passes over bump 406, the overlapping residual motion on path 404 deflects upward. This upward deflection in the graphical representation of residual motion indicates the presence of bumps (e.g., other objects, features, obstacles, etc.) on the road surface. Data representing the residual motion of the road surface can be used to calculate the road profile.

[0180] In some implementations, the road profile is the road height along the path the wheels will travel. In some examples, the height is relative to a reference plane. For example, the road profile is calculated from residual motion by converting residual motion, in pixels, into a distance metric (e.g., centimeters) applicable to the road surface. In some examples, the road profile can be represented in pixels.

[0181] Figure 4A It also includes graph 410, which depicts the residual motion of the vehicle's left wheel (top of graph 410) and the residual motion of the vehicle's right wheel (bottom of graph 410). Arrow 412 indicates the portion of the residual motion data representing the bulge 406. Note the upward deflection of the residual motion graph at arrow 412, which occurs at a distance of approximately 8 meters (e.g., from the vehicle).

[0182] Figure 4B Similar to Figure 4A However, it depicts the situation where the vehicle path passes over the edge of the speed bump. Figure 4B Included to illustrate the analysis performed by a previous driver assistance system, as exemplified herein. Graph 430 depicts the road profile determined by the previous driver assistance system. Arrow 432 indicates the presence of speed bump 406. When compared, arrow 412 ( Figure 4A ) and arrow 432 ( Figure 4B The residual motion indicated by the diagrams looks very similar. That is, the contour diagrams show similar ridges and do not indicate that half of the wheel path is above the ridge and half is away from it. Therefore, the diagrams present... Figure 4A and 4B Driver assistance systems in either of the described scenarios can treat them equally. However, in Figure 4B The lateral slope of the edge of the ridge that the vehicle passes through may not be detected, thus preventing the driver assistance system (or other vehicle components) from making appropriate adjustments or taking appropriate actions. It will be understood that... Figure 4A and 4BThe example shown is merely a non-limiting example of a scenario in which the implementation of the present invention and its algorithm react and behave differently from existing technical solutions.

[0183] Figure 5A Describes the calculation Figure 4A The image band depicting the road outline. Specifically, Figure 5A A band for detecting bumps is shown, with a bandwidth of 31 pixels (e.g., 15 pixels on each side of the vehicle path, plus 1 pixel representing the center of the vehicle path). The speed bump 502 is clearly visible across the entire width of the band, starting near row 100. In some embodiments, the length of the band represents the distance in front of the vehicle. For example, the band may represent visual data along the vehicle path from a distance of 5 meters to 20 meters in front of the vehicle. In some embodiments, this distance may be constant. In some embodiments, the distance may be variable. For example, the distance represented by the length of the image band may be variable and depend on one or more conditions, such as vehicle speed. Similarly, the width of the band (e.g., in pixels) may be constant or variable. Generally, bands with more pixels require more computational resources to process.

[0184] Note that for more efficient processing, Figure 4A The diagonal path of vehicle path 404 has been straightened (e.g., perpendicular). In some implementations, the vehicle path is not straightened before processing. The center of the path can be made into a straight line by moving each row of the "straightening" strip.

[0185] Techniques related to the creation and straightening of image bands are described in further detail in: U.S. Application No. 14 / 554,500 (now U.S. Patent No. 9,256,791), filed November 26, 2014, entitled “Road Vertical Contour Detection”; and U.S. Application No. 14 / 798,575, filed July 14, 2015, entitled “Road Contour Vertical Detection”, the contents of which are incorporated herein by reference.

[0186] Figure 5B Describes the calculation Figure 4B The image band depicting the road outline. Specifically, Figure 5B The diagram shows a speed bump detection strip. From column 12 to column 31, the speed bumps cover only slightly more than half the width of the strip, but from column 1 to column 12, the strip is a flat road. Figure 5BUsing the depicted strips for road profile calculations may lead to uncertain or unsatisfactory results. The results may depend on the intensity of the texture on the road and the hump. If the road texture is strong, the hump may appear flat. If the hump texture is dominant, the result may be hump detection, such as... Figure 4B As shown in the example, in some cases, the result may be an average. This uncertainty is undesirable.

[0187] For example, when a vehicle travels along a path with one wheel half on a speed bump (or object on the road) and the other half off, the vehicle's experience and its impact on passengers can depend on a variety of factors, including those related to vehicle weight distribution, vehicle suspension, and vehicle speed. The impact on the vehicle may be sensitive to the exact path it follows (e.g., within a few centimeters). Therefore, it would be advantageous to be able to distinguish between a path entirely on a bump or a flat road and a wheel path close to the edge of an object or obstacle on the road surface. Alternatively, if the profile passes the edge of a speed bump (or any other object), it can be given low confidence. In some examples, suspension control may have a specific response to a partial bump, and the profile estimation system can provide appropriate indications to the suspension control system, allowing the suspension control system to activate a specific partial bump response. Partial bumps may also generate lateral yaw forces on the vehicle, and steering control may also benefit from knowing that this force is about to occur. In this case, the profile estimation system can provide appropriate indications to the steering control. If the vehicle has some form of automatic steering, the contour estimation system can provide indications of some bumps, and the automatic steering control can be configured to subtly adjust the vehicle's path (as needed), for example, so that the wheel path does not pass over the bumps at all.

[0188] Figure 6 Depicting Figure 4A An exemplary graph of the residual motion of a vehicle path, where the vehicle path passes through the center of a ridge. Graph 600 depicts the residual motion (measured in increments of image row pixels) along the road plane of vehicle path 404. Figure 6 As can be seen from this, along Figure 5A The residual motion of the path at the center of the image band depicted includes a sharp increase starting near pixel number 100. The resulting peaks represent bulges.

[0189] Example 2: Vehicle path partially passing through obstacles

[0190] Figure 7 , Figure 8 , Figure 9 , Figure 10A , Figure 10BVisual data and measurement results for a second example are shown. As described above, this second example illustrates a situation where the driver assistance system detects that a portion of the vehicle path (e.g., the path of the wheels) passes over a ridge with a uniform lateral height.

[0191] Figure 7 Exemplary visual data received (or otherwise captured) by a driver assistance system is shown. The visual data includes a representation of road surface 702 corresponding to road surface 402 on which the vehicle travels. Figure 7 It also includes a vehicle path 704 corresponding to the right wheel of the vehicle, superimposed on the visual data. Included on the vehicle path 704 is a measure of the residual motion along the road plane of the vehicle path. Figure 7 Speed ​​bump 706 on road surface 702 is also depicted. Rise 706 corresponds to rise 406, and appears to have a generally uniform lateral height. However, it is noteworthy that in this example, vehicle path 704 instead passes over the edge of rise 706. As vehicle path 704 passes over rise 706, the overlapping residual motion on path 704 deflects upwards.

[0192] Based on calculations using existing technology, Figure 7 The upward deflection shown is Figure 4B The upward deflection shown is the same. However, as mentioned above, in situations such as... Figure 7 The residual motion of road features calculated in the present case may lead to uncertain or unsatisfactory results. Implementation methods for improving the treatment of road surfaces with lateral slopes are given below.

[0193] Figure 8 An example road strip used to determine the road profile is shown. Figure 8 Corresponding to Figure 5B The image band is used to detect bumps on the road surface (e.g., road surface 702). The speed bumps from columns 12 to 31 only cover slightly more than half the width of the band, but from columns 1 to 12, the band is a flat road surface. Recall that in the section about... Figure 5A In the example described, the residual motion of the image strip is determined for a vehicle path passing through the center of the image strip.

[0194] The size (width) of the belt and the size of the segments described below can be set or selected based on certain parameters, including, for example, vehicle speed, the reaction time of the vehicle suspension system and other operating parameters, the frame rate of the camera, and the distance at which the system can react (or react optimally) to detected obstacles.

[0195] The longitudinal extent (length) of the band can also be predefined, or selected based on certain parameters (in the case of predefined settings, such predefined settings can also be selected based on the following parameters). For example, the lateral extent of the band can be selected based on vehicle speed, distance to the candidate (or object), and frame rate. Further, the lateral extent of the band can be selected such that each candidate (or object) is captured in at least two (usually consecutive) frames. For example, for a vehicle traveling at 100 km / h, the lateral extent of the band could be at least 150 pixels, such that the candidate obstacle will be captured in at least two consecutive image frames within the band. It should be understood that the reference to two consecutive image frames as possible parameters that can be used to determine the lateral extent of the image frame band is merely illustrative, and other lengths of image frame bands can be used, including, for example, lengths provided for capturing candidates in three (or four, etc.) consecutive image frames.

[0196] Not to Figure 8 Instead of processing the image band as a single image band, the image band can be segmented. In some implementations, the image band is segmented into two bands. Figure 9 Two exemplary segments are depicted: a left segment (including pixels 0 to 15 of the image band) and a right segment (including pixels 16 to 31 of the image band). Therefore, Figure 8 The image has been roughly divided into two halves vertically (in the image) – the left segment comprises pixels 1-15 (for each row width), and the right segment comprises pixels 16-31 (for each row width). It will be understood that the image band (or patch) can be divided into any number of segments (e.g., two, three, ... n (Segments), where each segment comprises two or more pixels. In some examples, the segmented band can be further subdivided. In yet another example, the original image band can be segmented into multiple segments that partially overlap but do not include other segments. For example, the image band can be segmented into two segments of each half, or into three segments of each third. Any other appropriate image segmentation is foreseeable.

[0197] In some implementations, as previously described, residual motion is calculated over the entire belt, but residual motion is also calculated separately for each partial belt (segment). In some implementations, residual motion is calculated for fewer than all belts. For example, residual motion may be calculated only for segments (partial belts).

[0198] In some implementations, the image band is segmented into multiple segments. In some implementations, the image band is segmented along the latitudinal direction. In some implementations, the image band is segmented into multiple segments of different sizes. In some implementations, the image band is segmented multiple times. For example, the original image can be segmented multiple times, or the segments can be further segmented. Road contours can then be calculated for all or some of the segmented bands.

[0199] Figure 10A and Figure 10B The diagram shows the contour results (in pixels) calculated for the entire band (“both”) and for the left and right portions of the band, which were generated by segmenting the entire image band.

[0200] Figure 10A The second example corresponds to the path passing through the edge of the ridge ( Figure 8 In this case, the left and right segments present distinctly different profiles. The profile of the entire (unsegmented) strip closely approximates that of the right strip, which lies entirely on the ridge. However, the profiles of the right strip and the entire strip differ significantly from those of the left strip (e.g., having much larger residual motion values ​​than the left strip). The difference in residual motion between the left and right strips can indicate the lateral slope of the vehicle's path. In this example, the vehicle's path crosses the edge of the speed bump, so the lateral slope is a result of the transition from the speed bump to a flat road surface (e.g., when viewed from the right strip to the left strip).

[0201] For comparison, Figure 10B Corresponding to the first example ( Figure 5A The path passes through the center of the ridge. It can be seen that... Figure 10A All three contours are similar, and the overall contour of the band actually looks like the average of the left and right bands.

[0202] In some implementations, once the three contours have been calculated, they are compared. The comparison can be made in units of residual motion (pixels), or the residual motion can be converted into height measurements and the heights can be compared. In some implementations, comparing contours includes determining whether the residual motion (or height) of each road contour is within a threshold of each other. For example, using residual motion, the system determines whether all three measurements for a particular row are consistent within 0.25 pixels. If not, in some examples, the confidence associated with the road contour and / or the residual motion of the road contour is reduced.

[0203] In some implementations, a representation of the lateral slope of the obstacle is determined and provided as output. In some implementations, the representation of the lateral slope includes the direction of the slope. In some implementations, a bitwise representation of the slope is calculated. For example, 2 bits (0 and 1) are used: "00" indicates that all measurements are consistent within a threshold, "10" indicates that (left-right) > (threshold), "01" indicates that (right-left) > (threshold), and "11" may indicate that the contours of "both" (the entire band) are significantly larger or smaller than the contours of both "left" and "right". The last case ("11") indicates some kind of error because the residual motion of "both" is usually somewhere between the "left" and "right" bands, and in some examples, the confidence value may decrease as a response.

[0204] In some implementations, the metric difference is calculated after the residual motion is converted into metric height. For example, the difference between the metric height of the left strip and the metric height of the right strip profile is calculated. In some examples, the difference is obtained between the two profiles at the same pixel row along each of the left and right strip profiles (e.g., for two segments, at approximately the same lateral distance from the vehicle). In other examples, the calculated difference may be the maximum difference between any two heights along the left and right strip profiles. Along the profiles of the segmented strips, the difference can be obtained between any two suitable points.

[0205] In some implementations, the lateral slope is represented by an actual slope value. In some examples, the slope value is represented as an angle. Figure 9 In the example image strip depicted, the centers of the left and right path segments are 15 pixels apart. Given the distance per row (pixel), this pixel value can be converted into the actual metric distance between the centers of the image strip segments. For example, at a distance of 10 meters from a vehicle, a 15-pixel spacing might correspond to 10 centimeters. Given a 5-centimeter profile height difference and a 9-centimeter lateral distance between the centers of the two path segments, this would mean a lateral slope of 5 / 9 (≈0.555556). In some examples, the lateral distance used to calculate the slope is derived in a different way (e.g., not at the centers of the two path segments). Any suitable technical intent is within the scope of this disclosure. Techniques for determining metric lengths from visual data using homography are well known. Any suitable method used to determine metric height and width can be used to determine the slope value of a detected obstacle. In some examples, pixel values ​​are used to determine the slope value.

[0206] In some implementations, the road contour system outputs all three contours. In some implementations, if there are significant differences, the road contour system outputs all three heights, and a downstream controller can make a decision. In some implementations, the output contours are fewer than all the calculated contours.

[0207] Effective calculation

[0208] According to the above embodiments, various techniques can be used to improve the computational efficiency of road profile output. Preferably, the calculation of multiple profiles is performed efficiently, such as the three profiles in the example above ("left" band, "right" band, and "both"). Furthermore, heavy computations are preferably minimized (e.g., performed once).

[0209] In some examples, the distortion of the second image frame to the first image frame, along with any preprocessing (such as rank transformation, low-pass or high-pass filtering), is performed once before the bands are separated. In some examples, preprocessing is performed on one or more image frames to reduce noise, increase texture in the image, or for any other suitable reason to aid in image processing techniques performed using the image frames.

[0210] In some implementations, other computations can be shared between processes. In some examples, the sum of squared differences (SSD) is used to align the first and second image frames. For example, the SSD score for a 15x31 patch can be calculated by adding the SSD scores of the same 15 rows summed over the left column to the SSD scores summed over the left column. To reiterate, the SSD scores of partial image bands can be calculated and then combined to arrive at the score for the entire image band, avoiding performing the SSD operation again on the combined image band. Therefore, so far, there is no significant additional computation. The result is three arrays of scores, such as... Figure 3 As shown, it is almost the cost of one.

[0211] In some implementations, parabolic interpolation is used to first determine the peak value in the normalized correlation score of the road profile as the nearest integer, and then as a sub-pixel.

[0212] The final stage of searching for local maxima in subpixels in the score array is repeated three times. In some examples, for partial image band contours (e.g., left and right bands), some parts of the computation, such as calculating the confidence score for each row, can be skipped. In some examples, calculating normalized correlation instead of SSD is similar in efficiency because the vector of the entire band... and It's simply the sum of the left and right sides. Similarly, the absolute sum of differences (SAD) can be used instead of SSD.

[0213] In some examples, if computation cycles are very expensive, some calculations of subpixel local maxima can be used. Integer local maxima across the entire band can be used to select points for subpixel calculations in the left and right bands. For example, subpixel calculations can be performed on portions of the contour that might represent obstacles on a road surface (e.g., local maxima).

[0214] Three single-frame contours can also be used in the multi-frame contour calculations described above. In some examples, all three measurements can be used to calculate the multi-frame median. To save computation, random descent can be performed using only the contour of the entire strip, and the same slope and offset values ​​can be used to align other strips.

[0215] Lateral slope output

[0216] Driver assistance systems can use the determination of lateral slope associated with features on the road surface in various ways. In some implementations, one or more road profiles are provided as output. For example, the output can be provided to one or more driver assistance systems. In some implementations, the output includes the direction and / or value of the lateral slope. In some examples, the driver assistance system partially adjusts vehicle component settings based on the output. Therefore, comparing the road profile of an image strip with segmented image strips is used to generate actionable data for the driver assistance system.

[0217] In some implementations, the vehicle steering settings are adjusted to modify the vehicle path, wherein the modified vehicle path avoids over bumps in the road surface. For example, the vehicle can be controlled so that it avoids having its wheels go over the edge of a speed bump.

[0218] In some implementations, the vehicle suspension settings are adjusted. For example, to ensure safety and comfort, the vehicle suspension components can be adjusted to prepare the wheels for contact with laterally inclined surfaces.

[0219] In some implementations, the driver assistance system outputs road contour information and / or lateral slope information to one or more advanced systems that make appropriate decisions for vehicle control.

[0220] Output via automotive communication bus

[0221] In some implementations, the output of the system performing the techniques described herein can be transmitted to the vehicle suspension controller via a vehicle communication bus (such as a CAN bus or FlexRay). It should be understood that these are merely examples provided as vehicle communication buses, and any other suitable type of communication bus can be used.

[0222] In the CAN bus example, data needs to be packaged into a series of 8-byte messages. Now consider transmitting a road profile extending from 5.00m to 14.95m. In this example, the system outputs profile points every 5cm, resulting in 200 points. Height values ​​between -0.31m and 0.32m with a resolution of 0.0025m can be encoded as 8 bits. We can reserve 4 bits for the confidence value, 1 bit for the presence of bulges, and 3 bits for the lateral slope. This allows for 3 values ​​for left / right slope and 3 values ​​for right / left slope; 000 can indicate a negligible slope, and 111 can indicate errors detected by lateral slope calculations, such as the residual movement of the entire strip outside the residual range of the left and right sections.

[0223] For example, the table below (Table 1) can be used for three slope positions. It should be understood that Table 1 only provides an example of using one or more positions to represent lateral slope values, and any other suitable lateral slope representation can be used.

[0224]

[0225] Table 1

[0226] In the examples shown in Table 1, "large" means greater than 1 / 1 (e.g., slope value), "medium" means between 1 / 1 and 1 / 2, "small" means 1 / 5, and "no slope" means a slope less than 1 / 5. Therefore, for each profile, each 200 points of 2 bytes is 400 bytes or 50 CAN messages.

[0227] Other techniques for calculating transverse slope

[0228] Any other suitable techniques for calculating lateral slope indication are intended to be within the scope of this disclosure. For example, a computationally dense structure of the entire road surface can be used. In this example, the system can: calculate a dense structure of the entire road surface (e.g., using a multi-camera stereo or motion structure); determine the vehicle path along the surface; define the profile as the height along the path (in camera coordinates or road reference plane coordinates); and define the lateral slope as the lateral variation in height of points to the left and right of the path.

[0229] In some implementations, image cropping is used to calculate the lateral slope. An example of this process is described in more detail below.

[0230] Use shear to calculate transverse slope

[0231] Image cropping can be used as an alternative method to calculate the lateral slope at a point along the path. Consider a patch of 17 rows, each 31 pixels wide, centered vertically around that point. The system can then look for two parameters, rather than just a single vertical displacement. vTo best align the patch obtained from the current image with the previous image: a constant for the entire patch. v 0 and multiplied x Coordinate parameters v x :

[0232] (1)

[0233] Using Horn and Schunk luminance constraints (see, for example, BKPHorn and BGSchunck in 1981) Artificial Intelligence (Published in "Determining optical flow", Vol. 17, pp. 185-203)

[0234] (2)

[0235] And set u =0, because we only align in the vertical direction, and set in x Given a 31-pixel wide band from x=-15 to 15, we get:

[0236] (3)

[0237] In order to solve v 0 and v x We can see this in Lukas and Kanade (see, for example, B.D. Lukas and T. Kanade (1981)). An iterative image registration technique with an application to stereo vision (From the Imaging Understanding Workshop proceedings, pp. 121-130) After that, the least squares method is applied to the entire patch, but the motion (u, v) in x and y (respectively) is replaced with the motion in y and shearing. v 0 and v x Exercise. We discovered... v 0 and v x Minimize the SSD score of the entire patch:

[0238] (4)

[0239] The solution is given by the following formula:

[0240] (5)

[0241] v x A large absolute value indicates a significant slope, and the sign indicates a downward slope from left to right or from right to left. v x This is explained as the slope angle depending on the row of the strip.

[0242] Figure 11 A flowchart illustrating an exemplary process 1100 for processing visual data of a road plan according to some embodiments is depicted. In some embodiments, process 1100 is performed by one or more computing devices and / or systems (e.g., system 1200).

[0243] In box 1110, access visual data representing the road surface.

[0244] In box 1120, an initial road profile for the road surface is determined. In some implementations, the initial road profile is derived from the residual motion of vehicles along the path associated with the road surface.

[0245] In box 1130, a portion of the visual data representing the road surface is segmented into a first segment and a second segment. In some embodiments, the segmented portion of the visual data includes visual data representing obstacles on the road surface.

[0246] In box 1140, the first segment of the road outline of the first segment of this part of the visual data is determined.

[0247] In box 1150, the second segment of the road outline of the second segment of this part of the visual data is determined.

[0248] In box 1160, compare one or more of the first road profile, the second road profile, and the initial road profile.

[0249] In box 1170, based at least in part on the comparison results, an indication of the lateral slope of obstacles on the road surface is output.

[0250] Figure 12 A flowchart illustrating an exemplary process 1200 for processing visual data of a road plan according to some embodiments is depicted. In some embodiments, process 1200 is performed by one or more computing devices and / or systems (e.g., system 1200).

[0251] In box 1210, a path is determined on the road surface that is expected to be traversed by at least one wheel of a vehicle.

[0252] At box 1220, at least two images captured by one or more cameras are used to determine the height of the road surface at at least one point along the path the wheel will traverse.

[0253] In box 1230, an indication of the lateral slope of the road surface at at least one point along the path is calculated.

[0254] In box 1240, an indication of the height of the point and an indication of the lateral slope at at least one point along the path are output on the vehicle interface bus.

[0255] In some implementations, at least two images are used to determine the elevation of the road at two points along the path the wheel will traverse, these two points being laterally displaced from each other and at substantially the same distance along the path. In some implementations, an indication of the lateral slope of the road surface is calculated in part based on a comparison of the elevations of the road surface at the two points.

[0256] In some implementations, at least one of the two points is a point along a calculated road profile. In some implementations, the two points are along two calculated road profiles. For example, each of the two points may lie on a separate road profile.

[0257] In some implementations, at least one point is two or more points. In some implementations, at least one point is five or more points.

[0258] In some implementations, at least two images are captured from a single camera at different times. In some implementations, a second image is captured after the vehicle has moved at least a minimum distance from the vehicle position where the first image was captured.

[0259] In some implementations, the minimum distance is adjustable. For example, the minimum distance from the vehicle position where the first image was captured can be varied based on one or more factors (e.g., vehicle speed). In some implementations, the minimum distance is at least 0.3 meters. It should be understood that other minimum distances may be used.

[0260] In some implementations, at least two images are captured using two cameras. For example, one image may be captured by a first camera, and the second image may be captured by a second camera. In some implementations, the two cameras are laterally shifted within the vehicle. In some implementations, at least one of the two cameras is mounted on the vehicle's windshield.

[0261] In some implementations, the lateral slope indicator indicates whether the height of the point is substantially the same to the left and to the right. For example, the lateral slope indicator may indicate that adjacent points in two directions are at approximately the same height as the point.

[0262] In some implementations, the lateral slope indicator indicates whether the elevation to the left of the point is significantly greater than the elevation to the right of the point. For example, the lateral slope indicator may indicate a downward slope from the left side of the point to the right side of the point.

[0263] In some implementations, the lateral slope indicator indicates whether the elevation to the left of the point is significantly less than the elevation to the right of the point. For example, the lateral slope indicator may indicate a downward slope from the right side of the point to the left side of the point.

[0264] In some implementations, the lateral slope is indicated by the angle of the slope. In some implementations, the angle of the slope is represented by five or more possible values. In some implementations, the angle of the slope is represented by an integer value. For example, the angle of the slope can be indicated by a combination of bits (e.g., 000, 001, 010, etc.).

[0265] Figure 13 Components of an exemplary computing system 1300 configured to perform any of the processes described above are depicted. In some embodiments, the computing system 1300 is a vehicle computer, module, component, etc. The computing system 1300 may include, for example, a processing unit including one or more processors, memory, storage, and input / output devices (e.g., monitor, touchscreen, keyboard, camera, pen, drawing device, disk drive, USB, internet connection, near-field wireless communication, Bluetooth, etc.). However, the computing system 1300 may include circuitry or other dedicated hardware for performing some or all aspects of the processes (e.g., process 1100 and / or process 1200). In some operating setups, the computing system 1300 may be configured as a system comprising one or more units, each unit configured to perform certain aspects of the process in software, hardware, firmware, or some combination thereof.

[0266] In computing system 1300, main system 1302 may include an input / output (“I / O”) section 1304, one or more central processing units (“CPUs”) 1306, and a memory section 1308. Memory section 1308 may contain computer-executable instructions and / or data for executing at least a portion of process 1100. I / O section 1304 may optionally be connected to one or more cameras 1310, one or more sensors 1312, non-volatile memory units 1314, or one or more external systems 1320. For example, external system 1320 may be another vehicle control component or system. I / O section 1304 may also be connected to other components (not shown) capable of reading / writing non-transitory computer-readable storage media, which may contain programs and / or data.

[0267] At least some values ​​based on the results of the above process can be saved for later use. Additionally, a non-transitory computer-readable storage medium can be used to store (e.g., tangibly embody) one or more computer programs for executing any of the above processes by means of a computer. For example, the computer programs can be written in general-purpose programming languages ​​(e.g., Pascal, C, C++, Java, etc.) or some specialized application-specific languages.

[0268] For illustrative and descriptive purposes, the foregoing description of specific embodiments has been presented. These descriptions are not intended to be exhaustive or to limit the scope of the claims to the precise forms disclosed, and it should be understood that many modifications and variations are possible in accordance with the foregoing teachings.

Claims

1. A system installed on a host vehicle, the system comprising: camera; and One or more processors, wherein the one or more processors are configured to execute instructions to: Determine the path on the road surface that is expected to be traversed by at least the first wheel of the main vehicle; Using at least two images captured at different times by a single camera of the master vehicle, the road surface height of at least a first point located along the path expected to be traversed by the first wheel is determined, wherein the at least two images are captured during the master vehicle's forward translation of a minimum threshold distance, and wherein the road surface height is determined based on: (i) calculating the homography of the at least two images, (ii) determining a reference plane from the homography, and (iii) measuring the residual motion of the road surface relative to the reference plane, wherein the residual motion represents the profile of the road surface relative to the reference plane, the profile being the height of the road surface along the path to be traveled; Calculate an indication of the lateral slope of the road surface at the first point; and The vehicle interface bus of the main vehicle outputs an indication of the height of the first point and an indication of the lateral slope of the road surface at the first point. The road surface includes the road surface itself and any stationary objects on the road surface. The lateral slope of the road surface is an indicator of the vertical deviation between two points on the road surface, where the two points are laterally displaced relative to each other. The lateral slope of the road surface is determined by a calculated road profile along a left-wheel vehicle path corresponding to the left side of the main vehicle or a right-wheel vehicle path corresponding to the right side of the main vehicle, and wherein separate road profiles for the left and right sides of the main vehicle are calculated, and wherein... The road surface includes speed bumps.

2. The system according to claim 1, wherein, The one or more processors are configured to execute instructions to cause a system response based on the lateral slope of the road surface at the first point and the height of the road surface at the first point.

3. The system according to claim 2, wherein, The system response includes one of the following: steering the main vehicle or adjusting the suspension of the main vehicle.

4. The system according to claim 1, wherein, To determine the height of the road surface at the first point, the one or more processors are configured to execute instructions to: The initial road profile is derived from the residual motion of the road surface along the path; and Calculate the residual motion at the first point.

5. The system according to claim 1, wherein, The one or more processors are configured to execute instructions to determine the height of each of a plurality of points located along the path expected to be traversed by the first wheel, using at least two images captured at different times by the single camera on the master vehicle.

6. The system according to claim 5, wherein, At least one of the plurality of points is on a stationary object on the road surface, and at least one of the plurality of points is not on the stationary object.

7. The system according to claim 1, wherein, The one or more processors are configured to execute instructions to determine the height of the road surface at a second point located along the path expected to be traversed by the first wheel, wherein the first point and the second point are laterally displaced from each other.

8. The system according to claim 7, wherein, The one or more processors are configured to execute instructions to calculate the height difference between the first point and the second point.

9. The system according to claim 1, wherein, The one or more processors are configured to execute instructions to determine the direction of the lateral slope.

10. The system according to claim 1, wherein, The lateral slope of the road surface at the first point indicates the expected height difference between two or more points above which the first wheel is expected to be in front of the current position of the master vehicle.

11. A method for processing visual data of a road surface in a primary vehicle environment, the method comprising: Determine the path on the road surface that is expected to be traversed by at least the first wheel of the main vehicle; Using at least two images captured at different times by a single camera on the master vehicle, the road surface height of at least a first point located along the path expected to be traversed by the first wheel is determined, wherein the at least two images are captured during the master vehicle's forward translation of a minimum threshold distance, and wherein the road surface height is determined based on: (i) calculating the homography of the at least two images, (ii) determining a reference plane from the homography, and (iii) measuring the residual motion of the road surface relative to the reference plane, wherein the residual motion represents the profile of the road surface relative to the reference plane, the profile being the height of the road surface along the path to be traveled; Calculate an indication of the lateral slope of the road surface at the first point; and The vehicle interface bus of the main vehicle outputs an indication of the height of the first point and an indication of the lateral slope of the road surface at the first point. The road surface includes the road surface itself and any stationary objects on the road surface. The lateral slope of the road surface is an indicator of the vertical deviation between two points on the road surface, where the two points are laterally displaced relative to each other. The lateral slope of the road surface is determined by a calculated road profile along a left-wheel vehicle path corresponding to the left side of the main vehicle or a right-wheel vehicle path corresponding to the right side of the main vehicle, and wherein separate road profiles for the left and right sides of the main vehicle are calculated. And among them, The road surface includes speed bumps.

12. The method of claim 11, further comprising causing a system response based on the lateral slope of the road surface at the first point and based on the height of the road surface at the first point.

13. The method according to claim 12, wherein, The system response includes one of the following: steering the main vehicle or adjusting the suspension of the main vehicle.

14. The method according to claim 11, wherein, Determining the height of the road surface at the first point includes: The initial road profile is derived from the residual motion of the road surface along the path; and Calculate the residual motion at the first point.

15. The method of claim 11, further comprising: The height of each of a plurality of points is determined using at least two images captured at different times by the single camera on the master vehicle, along the path that is expected to be traversed by the first wheel.

16. The method according to claim 15, wherein, At least one of the plurality of points is on a stationary object on the road surface, and at least one of the plurality of points is not on the stationary object.

17. The method of claim 11, further comprising determining the height of the road surface at a second point located along the path expected to be traversed by the first wheel, wherein the first point and the second point are laterally displaced from each other.

18. The method of claim 17, further comprising calculating the height difference between the first point and the second point.

19. The method of claim 11, further comprising determining the direction of the lateral slope.

20. The method according to claim 11, wherein, The lateral slope of the road surface at the first point indicates the expected height difference between two or more points above which the first wheel is expected to be in front of the current position of the master vehicle.

21. A non-transitory computer-readable storage medium storing one or more programs, said one or more programs comprising instructions that, when executed by one or more processors of a vehicle, cause the processor to perform the method of any one of claims 11-20.

22. A system installed on a host vehicle, the system comprising: camera; and One or more processors, wherein the one or more processors are configured to execute instructions to: Determine the path that is expected to be traversed by the first wheel of the main vehicle; Using at least two images captured at different times by a single camera on the master vehicle, the height of a first point located along the path expected to be traversed by the first wheel and the height of a second point along the path expected to be traversed by the first wheel are determined, wherein the at least two images are captured during a forward translation of the master vehicle by a minimum threshold distance, and wherein the heights of the first and second points are determined based on: (i) calculating the homography of the at least two images, (ii) determining a reference plane from the homography, and (iii) measuring residual motion relative to the reference plane, wherein the residual motion represents a profile relative to the reference plane, the profile being the height of the road surface along the path to be traveled; Calculate the vertical deviation between the height of the first point located along the path and the height of the second point located along the path. The second point is the point that the first wheel is expected to pass when it is above the first point. The vertical deviation is determined by a calculated road profile along a left-wheel vehicle path corresponding to the left side of the main vehicle or a right-wheel vehicle path corresponding to the right side of the main vehicle, and wherein separate road profiles for the left and right sides of the main vehicle are calculated; and This results in a system response based on the vertical deviation between the height at the first point and the height at the second point.

23. The system according to claim 22, wherein, The first point and the second point are laterally shifted relative to each other.

24. The system according to claim 22, wherein, The lateral slope of the road surface at the first point indicates the expected height difference between two or more points above which the first wheel is expected to be in front of the current position of the master vehicle.

25. The system according to claim 22, wherein, The system response includes one of the following: steering the main vehicle or adjusting the suspension of the main vehicle.

26. The system according to claim 22, wherein, To determine the height at the first point, the one or more processors are configured to execute instructions to: The initial road profile is derived from the residual motion of the road surface along the path; and Calculate the residual motion at the first point.

27. The system according to claim 22, wherein, At least one of the first point or the second point is on a stationary object on the road surface, and at least one of the first point or the second point is not on the stationary object.

28. The system according to claim 27, wherein, The road surface includes speed bumps.

29. A method for processing visual data of a host vehicle's environment, the method comprising: Determine the path that is expected to be traversed by the first wheel of the main vehicle; Using at least two images captured at different times by a single camera on the master vehicle, the height of a first point located along the path expected to be traversed by the first wheel and the height of a second point along the path expected to be traversed by the first wheel are determined, wherein the at least two images are captured during the master vehicle's forward translation of a minimum threshold distance, and wherein the height of the first point and the height of the second point are determined based on: (i) calculating the homography of the at least two images, (ii) determining a reference plane from the homography, and (iii) measuring residual motion relative to the reference plane, wherein the residual motion represents a profile relative to the reference plane, the profile being the height of the road surface along the path to be traveled; Calculate the vertical deviation between the height of the first point located along the path and the height of the second point located along the path. The second point is the point that the first wheel is expected to pass when it is above the first point. The vertical deviation is determined by a calculated road profile along a left-wheel vehicle path corresponding to the left side of the main vehicle or a right-wheel vehicle path corresponding to the right side of the main vehicle, and wherein separate road profiles for the left and right sides of the main vehicle are calculated; and This results in a system response based on the vertical deviation between the height at the first point and the height at the second point.

30. The method according to claim 29, wherein, The first point and the second point are laterally shifted relative to each other.

31. The method according to claim 29, wherein, The lateral slope of the road surface at the first point indicates the expected height difference between two or more points above which the first wheel is expected to be in front of the current position of the master vehicle.

32. The method according to claim 29, wherein, The system response includes one of the following: steering the main vehicle or adjusting the suspension of the main vehicle.

33. The method according to claim 29, wherein, Determining the height of the first point includes: The initial road profile is derived from the residual motion of the road surface along the path; and Calculate the residual motion at the first point.

34. The method according to claim 29, wherein, At least one of the first point or the second point is on a stationary object on the road surface, and at least one of the first point or the second point is not on the stationary object.

35. The method according to claim 34, wherein, The road surface includes speed bumps.

36. A non-transitory computer-readable storage medium storing one or more programs, said one or more programs comprising instructions that, when executed by one or more processors of a vehicle, cause the processor to perform the method of any one of claims 29-35.

Images