METHOD FOR DETERMINING THE DISTANCE BETWEEN A MOTOR VEHICLE AND AN OBJECT

Description

[0001] The invention relates to a method for determining the distance between a motor vehicle and at least one object. The invention also relates to a motor vehicle.

[0002] In vehicles, it is advantageous to determine distances to other objects, such as vehicles ahead or pedestrians, in order to provide relevant information to the driver and / or to use it within driver assistance systems. For example, adaptive cruise control can regulate the distance to a vehicle ahead, or collision avoidance systems can use distance information to intervene in the driving process or issue warnings if minimum distance limits are breached.

[0003] Distances to other objects can be measured using radar or ultrasonic sensors, for example. However, if distances to objects at great distances are also to be measured, radar sensors, for instance, may require complex sensor equipment, or distance information may only be captured within a relatively small solid angle.

[0004] Alternatively or additionally, various approaches to distance measurement using cameras can be employed. For example, it is known from German patent application DE 10 2009 009 047 A1 to use stereo image processing to determine a depth map of the environment in order to ascertain the distance and position of an object's base. However, methods for distance determination based on stereo image processing typically only achieve sufficient accuracy for relatively short distances.

[0005] Another approach is to use known dimensions of other motor vehicles to determine a distance from two-dimensional image data. German patent application DE 10 2011 055 441 A1 proposes capturing a vehicle's license plate. Since license plates can have a prescribed size, the distance to the vehicle can be estimated based on the captured dimensions. A similar procedure is also known from German patent application DE 10 2006 012 914 A1. There, horizontal and vertical edges are detected in an image of the rear of a vehicle driving ahead. A distance is determined depending on the position of a horizontal edge and the distance between the vertical edges. A disadvantage of these approaches for determining distances is that information about the actual width of a captured object or other defined dimensions is required to enable a robust distance determination.Therefore, the described methods are typically not usable for all relevant objects, and a potentially extensive object database is required.

[0006] Document US 2012 / 262580 A1 discloses a surround-view system that can provide a 360° view from inside a vehicle via cameras mounted at various points on the vehicle.

[0007] The article Egmont-Petersen, M. et al., "Image processing with neural networks-a review", Pattern Recognition, Elsevier, GB, vol. 35, no. 10, October 2002 (2002-10-01), pages 2279 - 2301, XP004366785, ISSN: 0031-3203, DOI: 10.1016 / S0031-3203(01)00178-9, discusses various methods for image processing using neural networks.

[0008] The article Sherrah, J., "Learning to Adapt: ​​A Method for Automatic Tuning of Algorithm Parameters", Advanced Concepts for Intelligent Vision Systems. ACIVS 2010, Lecture Notes in Computer Science, vol 6474. Springer, Berlin, Heidelberg. https: / / doi.org / 10.1007 / 978-3-642-17688-3 39, discloses a method for automatically adjusting parameters of image processing algorithms.

[0009] Document WO 2017 / 090410 A1 discloses an approach for determining object distance using a stereo camera to ascertain distances between points within an overlapping area. The distance to an object outside the cameras' overlapping area can be determined by selecting a pixel at the same height in the image that lies within the overlapping area and evaluating the distance to this pixel based on a parallax shift in the image data from both cameras. To also capture changes in height outside the cameras' overlapping area, previously acquired image data can be evaluated.

[0010] The invention is therefore based on the objective of providing an improved method for determining the distance between a motor vehicle and at least one object, which in particular makes it possible to determine distances to objects for which no or little additional information is available, even with simple sensor equipment of the motor vehicle.

[0011] The object of the invention is achieved by a method for determining a distance between a motor vehicle and at least one object, wherein exactly one camera image from exactly one camera is used, which comprises the following steps: Capturing a two-dimensional camera image with a vehicle camera, segmenting the camera image, whereby those image points that depict the object are determined as object segments, determining a contact point of the object on a surface as a function of the object segment, determining the distance between the object and the vehicle as a function of the contact point and a capture geometry of the camera, wherein a processing algorithm generates a subsurface model from the segmented image data of the camera image, which relates to a height profile at least within a subsurface segment, comprising image points that represent the subsurface of the object, wherein the point of contact and / or the distance is determined as a function of the subsurface model, wherein the subsurface model describes several subsurface levels, wherein the processing algorithm is parameterized by several processing parameters which are determined by a machine learning method.

[0012] According to the invention, it is proposed to determine the pixels that depict the object, in particular to perform pixel-accurate segmentation. As will be explained in detail later, this can be done, in particular, using a segmentation algorithm trained by machine learning. Pixel-accurate segmentation of the image makes it possible, in particular, to determine the object's point of contact with a surface with high accuracy. In the simplest case, it can then be assumed that the vehicle is traveling on a plane on which the object is also located. In this case, a specific pixel, and thus a solid angle segment assigned to the pixel via the camera's detection geometry, can be used to assign the point of contact at a defined distance.Preferably, the inventive method takes into account the position of the camera and thus, in particular, of the motor vehicle, for example, a roll and / or pitch angle. As will be explained in detail later, it is also possible to deviate from the assumption of a flat surface, since it may be possible to determine the topography or the elevation profile of the surface, at least approximately, with a high degree of robustness during the evaluation of the camera image.

[0013] In the method according to the invention, exactly one camera image from exactly one camera is used. The method according to the invention can therefore be carried out with minimal sensor requirements.

[0014] The camera image can be segmented by classifying the individual pixels, whereby it can be determined for each pixel whether it is assigned to a specific object or to a background, in particular a surface. In the simplest case, the lowest point of a given object can be chosen as the point of contact. However, in the method according to the invention, partial occlusions of objects by other objects can also be detected, thus avoiding or reducing misjudgments of distances.

[0015] When using the method according to the invention, it is possible that the distance resolution is essentially limited by the pixel resolution of the camera. Thus, high accuracy can also be achieved for distant objects.

[0016] The measured distance between the object and the vehicle can be used, for example, to control vehicle equipment based on this distance, to intervene in driving operations, and / or to provide driver information. For instance, automatic braking interventions and / or warning messages can be issued depending on the measured distance.

[0017] The camera image can be segmented by a further processing algorithm, which is parameterized by several processing parameters determined by a machine learning method. For example, a neural network can be used as the primary processing algorithm and / or another neural network as the secondary processing algorithm. The processing parameters thus act as the cognitive memory of the trained processing algorithm. A major advantage of using machine learning methods is that it eliminates the need to manually define specific processing rules to perform segmentation. Machine learning methods, particularly neural networks, can achieve similar results to the human brain in image processing. It is well known that human observers can typically recognize object boundaries with pixel-level accuracy, even in two-dimensional images.In addition to local features such as edges, relative sizes, occlusions, shadows, and similar phenomena can also be taken into account. This capability can also be trained on the processing algorithm.

[0018] The processing parameters can be determined, in particular, using supervised learning. For this purpose, training datasets can be provided in which image segmentation has already been performed. The segmentation of the images in the training datasets can be done manually, for example. It is also possible to perform automatic segmentation, using additional information such as distance information obtained via a radar or lidar sensor, or similar devices. The segmentation by the processing algorithm can be carried out, in particular, by determining classification information for individual pixels, indicating whether the respective pixel belongs to an object or to the image background. Regarding the image background, the classification can further distinguish between the foreground and other parts of the image background.If multiple objects are depicted, it is also possible to determine for each individual pixel which of the objects it belongs to.

[0019] Approaches to supervised learning, particularly through machine learning algorithms, are generally known in the art and will therefore not be explained in detail. In particular, error backpropagation can be used. More generally, gradient descent methods can be employed, in which a derivative of a cost function or a measure of deviation during the learning process with respect to the processing parameters is calculated to determine the direction of variation for these parameters.

[0020] If a neural network is used as the processing algorithm, the processing parameters could be, for example, the weights used in the individual neurons. A convolutional neural network or similar could be used as the neural network.

[0021] Depending on the camera image data, a subsurface model is generated that represents an elevation profile, at least within a subsurface segment. This segment comprises pixels that depict the object's ground, and the point of contact and / or distance are determined based on the subsurface model. In particular, an elevation profile can be determined for multiple separate subsurface segments. For example, the portion of the image data representing the ground for the vehicle or object can be divided into several subsurface segments, with a corresponding ground plane determined for each segment. These ground planes can be at different elevations and / or angled relative to each other. Alternatively, other subsurface models can be used. For example, the ground can be approximated, at least partially, by splines or polynomial curves.

[0022] According to the invention, the subsurface model describes several subsurface planes. For each of these subsurface planes, a support point can be specified, for example, a point located at a specific solid angle relative to the vehicle and having a specific height, and a slope at this point, for example, in the longitudinal and transverse directions of the vehicle. The subsurface model can be generated, for example, by processing the camera image using the [processor name] or a further processing algorithm, which is parameterized by several processing parameters determined by the [processor name] or a machine learning method. The processing algorithm can, for example, be a neural network or similar. The processing parameters can be learned, as explained above, by supervised learning, in particular using training datasets.For example, the training datasets can include two-dimensional image data with additional depth information, acquired, for instance, using a time-of-flight sensor. Since the three-dimensional position of each pixel is thus known, the corresponding heights are also known. Because the individual captured points of the subsurface are therefore located in three-dimensional coordinates, a cost function can be minimized during training, which calculates the deviation of the subsurface model generated by the processing algorithm from the actual point heights.

[0023] As previously explained, machine learning-trained processing algorithms can perform similar functions to human intuition. Even when viewing two-dimensional image data, a human observer can often easily recognize whether, for example, an incline or decline is depicted. Similarly, a machine learning-trained processing algorithm can recognize corresponding terrain features, particularly road alignments, and represent them, at least approximately, in a subsurface model. Since the height relative to the vehicle at which the point of contact is determined in the camera image data is thus known, at least approximately, the pixel position of the vehicle can be used to determine the vehicle's position.From the associated solid angle in which the point of contact is located relative to the motor vehicle, a distance of the point of contact from the motor vehicle can be determined with significantly increased accuracy.

[0024] To capture the camera image, one of several cameras on the vehicle can be used, whereby the vehicle's immediate surroundings can be mapped across its entire circumference, particularly in the form of a top-view image. This allows for distance measurements of objects in the entire vicinity of the vehicle, at least in the immediate vicinity. Since close-range cameras often have wide-angle or fisheye lenses that distort the captured surroundings, it is advantageous to take these distortions into account when determining the distance. Alternatively or additionally to the close-range cameras that map the vehicle's surroundings, a front-facing camera can also be used in the inventive method, which can be used, in particular, to determine the distances of relatively distant objects in front of the vehicle.

[0025] The camera image can be segmented by determining a pixel class for each pixel, indicating whether that pixel represents the object. Specifically, each pixel can be classified to show what it depicts, such as the object itself, another object, the background of the object or vehicle, or another background element.

[0026] In addition to the method according to the invention, the invention relates to a motor vehicle with a camera and a processing unit, which is configured to capture a camera image via the camera and to process the camera image according to the method according to the invention to determine the distance to the object. The features described for the method according to the invention, with the advantages mentioned therein, can be transferred to the motor vehicle according to the invention and vice versa.

[0027] The motor vehicle may, in particular, include a driver assistance system that, depending on the distance determined, controls a vehicle device in order to intervene in the driving operation and / or to issue information to a driver of the motor vehicle.

[0028] Further advantages and details will become apparent from the following exemplary embodiments and the accompanying drawings. These schematically illustrate: Fig. 1 shows an embodiment of a motor vehicle according to the invention in a driving situation in which the method according to the invention is used, Fig. 2 shows a vehicle in the driving situation in which the method according to the invention is used. Fig. 1 The camera image shown depicts the driving situation and is evaluated within the framework of the exemplary embodiment of the method according to the invention, and Fig. 3 is a flowchart of an exemplary embodiment of the method according to the invention.

[0029] Fig. 1 Figure 1 shows a motor vehicle 1 moving along a substantially level roadway 9. The motor vehicle 1 has several cameras 2-6, each capable of capturing two-dimensional images of its surroundings. In the described example, the front camera 2, which is designed to capture even distant objects, is used to determine the distances to objects 8, 10, and 12. However, the described procedure can also be applied to camera images captured by cameras 3-6, which serve to map the immediate vicinity of the motor vehicle 1 across its entire perimeter, particularly in the form of a top-view image. Thus, the described procedure can be used both to determine the distances to objects in any direction in the immediate vicinity and to determine the distances to distant objects 8, 10, and 12.

[0030] One in the Fig. 1 The driving situation shown is captured by the processing unit 7 via camera 2, as is the camera image 20. Fig. 2 As shown. Fig. 1 und Fig. 2 As can be seen, object 8, i.e., a motor vehicle, is located in front of the vehicle 1 on the level roadway 9. Object 10, a pedestrian, is located on a sidewalk 11 that is slightly elevated above the roadway 9. Object 12, another motor vehicle, is moving down the sloping street 13 towards the roadway 9.

[0031] A possible procedure for determining distances between motor vehicle 1 and objects 8, 10, 12 is described below with reference to Fig. 3 The procedure, as shown by line 14, is divided into a preparatory procedure section comprising steps S1 - S6 and a procedure section carried out in motor vehicle 1 comprising steps S7 - S12.

[0032] In the preparatory section, two processing algorithms are trained using machine learning. The first algorithm segments the camera image 20, while the second determines a subsurface model that at least approximately describes the elevation profile of the depicted subsurface. For this purpose, several training datasets are initially acquired. In step S1, two-dimensional image data is recorded, and in step S2, distance information assigned to the individual pixels is captured. This can be done, for example, using a time-of-flight camera. Alternatively, a camera could be used to capture the two-dimensional image data and be equipped with a radar sensor or another distance sensor to obtain additional depth information for pixels or image areas.

[0033] In step S3, the two-dimensional image data acquired in step S1 is manually segmented to determine, for each pixel of the camera image 20, whether it contains objects 8, 10, 12, or background elements. With regard to the background, a distinction can be made, in particular, between a surface on which objects 8, 10, 12 are located and other background objects. Alternatively, instead of manual segmentation, segmentation could also be performed in step S3 using additional information, especially the depth information acquired in step S2.

[0034] Using the segmented two-dimensional image data determined in step S3, a machine learning processing algorithm is trained in step S4 to segment the image data. The processing algorithm has a multitude of processing parameters, for example, in the case of a neural network, the weights of the individual inputs for each neuron. These processing parameters can initially be chosen randomly. Using the training datasets provided in step S3, the processing parameters can then be optimized so that the segmentation of the image data achieved by the processing algorithm corresponds as closely as possible to the segmentation specified in step S3. This can be achieved, for example, by minimizing a cost function that describes the deviation of the segmentations from one another, such as through error feedback.A similar approach is generally known in the field of machine learning from other application areas and will therefore not be explained in detail.

[0035] To obtain information about the elevation profile of a subsurface, a further processing algorithm is trained for this purpose in steps S5 and S6. In step S5, an elevation profile is first generated from the distance data determined in step S2, at least for those areas identified as subsurface in step S3. The image data segmented in step S3 and the subsurface profile calculated in step S5 together form a training dataset for training the further processing algorithm in step S6. This processing algorithm, which can also be a neural network, for example, evaluates segmented image data as input and determines a subsurface model as output. This model can describe the subsurface profile, for example, as several subsurface segments, each defined by a plane.As explained above, for training a cost function is minimized, whereby in this case this function sums the deviation between the generated subsurface model and the subsurface profile determined in step S5 across the different training data sets.

[0036] The aforementioned prepared steps can be carried out, for example, by a vehicle manufacturer or another service provider, and corresponding processing parameters or processing algorithms parameterized with corresponding processing parameters can be stored in the processing unit 7 of the motor vehicle.

[0037] To carry out the procedure in the vehicle, a two-dimensional camera image, for example camera 20, is first captured by a camera in the vehicle, for example camera 2. In step S8, the processing algorithm, which was parameterized in step S4, is applied to camera image 20 to assign each pixel of camera image 20 a pixel class that indicates whether the respective pixel shows object 8, 10, 12, a background on which objects 8, 10, 12 are located, or other background elements. This results in pixel-accurate segmentation of camera image 20.

[0038] In step S9, a contact point 17, 18, 19 is then determined for each of the detected objects 8, 10, 12. Specifically, the lowest pixel in the camera image 20 that corresponds to the respective object 8, 10, 12 is selected. To detect occlusions, it can be checked whether the respective contact point lies above a pixel that has been classified as part of the background.

[0039] In a simplified version of the in Fig. 3The method shown could now directly determine the distance to each object 8, 10, 12 from its point of contact. Each pixel of the camera image 20 is assigned a defined solid angle by the known detection geometry of the camera 2. In particular, taking into account the pitch and roll angle of the vehicle 1, and assuming that the ground beneath the vehicle 1 can be approximated as a plane, a respective distance can be assigned to each solid angle and thus to each pixel of the camera image 20. For object 8, which is approximately located on such a plane, a correct distance would already be determined in this case. However, for objects 10, 12, which are located above this plane, or for objects not shown, which are located below the plane, a distance that is too large or too small would be determined, respectively.

[0040] To enable more precise distance determination, the further processing function parameterized in step S6 is used in step S10 to generate a subsurface model from the image data segmented in step S8. This model describes, at least approximately, the elevation profile of the subsurface. The subsurface model can be described, in particular, by several planes lying at different elevations or at angles to each other. For example, the sidewalk 11 can be assigned to a plane that lies slightly above the plane of the roadway 9, and the sloping road 13 can be assigned to a plane that is at an angle to the plane of the roadway 9. Simple geometric calculations can be used to determine the intersection of a solid angle assigned to each pixel with the corresponding plane, and from this, the distance to the respective point of contact 17, 18, 19 can be calculated in step S11.

[0041] Depending on the distances to the individual objects 8, 10, 12 calculated in step S11, a vehicle device 15, 16 can be controlled in step S12. Vehicle device 15 could, for example, be a braking system that enables automatic braking of the vehicle 1 in the event of a collision risk. Vehicle device 16 could be a warning device, such as a display, that can inform the driver about distances to other vehicles or warn them of collision risks.

Claims

1. Method for determining a distance between a motor vehicle (1) and at least one object (8, 10, 12), wherein exactly one camera image of exactly one camera is used, comprising the following steps: - detecting a two-dimensional camera image (20) by a camera (2 - 6) of the motor vehicle (1), - segmenting the camera image (20), wherein those pixels which image the object (8, 10, 12) are determined as an object segment, - determining a contact point (17, 18, 19) of the object (8, 10, 12) on a base as a function of the object segment, - determining the distance between the object (8, 10, 12) and the motor vehicle (1) as a function of the contact point (17, 18, 19) and a detection geometry of the camera (2 - 6), wherein a base model is generated from the segmented image data of the camera image (20) by a processing algorithm, which base model relates to a height profile at least within a base segment which comprises pixels which image the base of the object (8, 10, 12), wherein the contact point (17, 18, 19) and / or the distance is determined as a function of the base model, wherein the base model describes a plurality of base planes, wherein the processing algorithm is or has been parameterised by a plurality of processing parameters which are or have been determined by a machine learning method.

2. Method according to claim 1, characterised in that: the camera image (20) is segmented by a further processing algorithm which is or has been parameterised by a plurality of processing parameters which are or have been determined by the or a machine learning method.

3. Method according to claim 2, characterised in that: a neural network is used as the processing algorithm and / or a neural network is used as the further processing algorithm.

4. Method according to any one of the preceding claims, characterised in that: one of a plurality of cameras (2 - 6) of the motor vehicle (1) is used for detecting the camera image (20), wherein the close range of the motor vehicle (1) can be imaged over the entire circumference of the motor vehicle (1) by means of the plurality of cameras (2 - 6), in particular in the form of a top-view image.

5. Method according to any one of the preceding claims, characterised in that: the segmentation of the camera image (20) takes place by a pixel class being determined for each pixel of the camera image (20), which pixel class indicates whether the respective pixel images the object (8, 10, 12).

6. Motor vehicle having a camera (2 - 6), characterised in that: it comprises a processing device (7) which is designed to detect a camera image (20) via the camera (2 - 6) and to process the camera image (20) according to the method according to any one of the preceding claims in order to determine the distance to the object (8, 10, 12).