Method and apparatus for classifying objects on a road, computer program and storage medium

Abstract

The invention relates to a method for classifying an object on a road in a vehicle's surroundings. The method has a step of reading image data from an interface to at least one vehicle camera of the vehicle. The image data represents a region of the surroundings having at least the road. The method also has a step of analyzing the image data, in which road markings of the road are recognized, a model of the surface of the road is generated using the recognized road markings and an object on the road is identified. The method also has a step of determining a first distance value between the vehicle camera and an object image point of the object represented by the image data and a second distance value between the vehicle camera and a road image point of the surface of the road in the object's surroundings defined by the model. Furthermore, the method has a step of performing a comparison of the distance values by means of at least one continuity criterion to distinguish between a protruding object and a flat object, in order to classify the object as a protruding object or a flat object depending on the comparison result.

Description

Technical Field

[0001] This invention relates to an apparatus and method for classifying objects on a road in the environment surrounding a vehicle. The subject matter of this invention also includes a computer program and a storage medium. Background Technology

[0002] Optical flow-based algorithms, known as "structure from motion," are known for distance estimation and 3D scene construction. Additionally, stereo cameras are known for generating depth information from parallax, or gaps, that produce biases. Summary of the Invention

[0003] Against this backdrop, based on the proposed solution, a method for classifying objects on a road is provided, along with an apparatus for using this method, and finally, a corresponding computer program and storage medium. Advantageous extensions and improvements to the described apparatus can be achieved through the further measures listed.

[0004] According to the implementation, a virtual road model can be generated from road markings, particularly for the identification, detection, and classification of protruding objects on the road. This virtual road model can be used as a reference for distinguishing between protruding objects and flat objects. Here, such a protruding object could be, for example, a lost cargo that a vehicle traveling ahead cannot cross. Algorithms based on optical flow or parallax bias can be used, for example, where the road model can be used as a distance reference for object points in the distinction.

[0005] Advantageously, according to the implementation, applications for reliably identifying or detecting and classifying protruding objects on the road can be implemented particularly for camera applications used in highly automated driving. Protruding objects can be reliably detected and classified as protruding objects, for example, at a minimum height of, say, 20 centimeters above the road surface. If a protruding object is present, emergency braking can therefore be reliably initiated, for example, by means of a corresponding braking cascade. In particular, it can reliably prevent flat or non-protruding objects (e.g., manhole covers) and their images in the vehicle's camera from triggering emergency braking, or in other words, it can prevent false negative identifications, and thus also prevent possible indirect losses related to liability, such as to the driver or subsequent vehicles. Therefore, for example, not only can objects in the lane be detected and distance assigned to the object by means of a defined error channel (Fehlerkorridor), but accurate and reliable classification can also be performed as follows: the object is either protruding or flat and therefore drivable.

[0006] A method for classifying objects on the road in the surrounding environment of a vehicle is proposed, wherein the method comprises the following steps:

[0007] Image data is read from an interface of at least one vehicle camera connected to the vehicle, wherein the image data represents at least a road area of ​​the surrounding environment;

[0008] The image data is analyzed and processed, including identifying road markings, generating a model of the road surface using the identified road markings, and identifying objects on the road.

[0009] Calculate the first distance value between the vehicle camera and the object image points represented by image data, and the second distance value between the vehicle camera and the road image points defined by the model on the surface of the road in the surrounding environment of the object;

[0010] Distance values ​​are compared using at least one continuity criterion to distinguish between protruding and flat objects, so that objects can be classified as protruding or flat objects based on the comparison results.

[0011] This method can be implemented, for example, in software, hardware, or a hybrid of software and hardware, such as in a controller or device. The vehicle can refer to a motor vehicle, such as a passenger car or a commercial vehicle. In particular, the vehicle can refer to a vehicle with highly automated driving or a vehicle used for highly automated driving. At least one vehicle camera can be mounted or fixedly installed in the vehicle. In the analysis and processing step, object recognition or an algorithm for object recognition can be performed on the image data. In the execution step, a classification signal representing the comparison result can be generated. The classification signal can be output via an interface to the controller, auxiliary systems, and additionally or alternatively, another vehicle device.

[0012] According to one implementation, image data can be analyzed and processed in the analysis and processing step to generate a three-dimensional model of the road surface using identified road markings via an interpolation method. This implementation offers the advantage of generating a model in a simple and accurate manner, enabling it to be used as a reference for accurate distance comparisons with objects. This is particularly advantageous because road surfaces generally do not have sufficiently rough textures compared to direct graphical representations that can be used for distance measurement.

[0013] Alternatively, the first distance value can be obtained row by row or line by line in at least one image of the surrounding environment represented by image data during the determination step. This implementation offers the advantage of obtaining the first distance value in a particularly simple and reliable manner.

[0014] Furthermore, in the determination step, a first distance value between the vehicle camera and the object image points can be calculated along the object's edge. For this purpose, edges can be identified in the analysis and processing step. Therefore, the comparison of distance values ​​can be limited to particularly convincing image points, i.e., the object image points, to obtain comparison results in a resource-efficient and reliable manner. Additionally or alternatively, the deviation between the first and second distance values ​​can be considered as a continuity criterion when performing the comparison in the execution step. In this way, a simple and accurate distinction can be achieved between protruding objects and flat or smooth objects.

[0015] In particular, at least one continuity criterion can be used in the execution steps, which specifies that, for classifying an object as a protruding object, the deviation between a first distance value and a second distance value is minimal in the foot region of the object on the model-defined surface of the road, increases along the side edges of the object away from the road, and is maximum in the head region of the object opposite the foot region. The foot region may have a foot edge with at least one foot point of the object. The side edge may extend between the foot region and the head region of the object. Here, the second distance value may relate to road image points arranged adjacent to the foot region, the side edge, and the head region. This implementation provides the advantage of reliably and accurately identifying the presence of protruding objects.

[0016] According to one embodiment, the optical flow of image data can be used in the analysis and processing step, and additionally or alternatively in the determination step. This embodiment offers the advantage that distance or depth information can be obtained from multiple image sequences in a simple and reliable manner using image data. According to one embodiment, the parallax deviation or gap of image data can be used in the analysis and processing step, and additionally or alternatively in the determination step. Here, at least one vehicle camera can be implemented as a stereo camera. This embodiment offers the advantage that distance or depth information can be obtained in a simple and accurate manner.

[0017] Furthermore, in the analysis and processing step, a vehicle motion model representing the vehicle's motion can be analyzed and processed to determine the road characteristics that determine the motion, and to generate a model of the road surface using the identified road markings and road characteristics. The vehicle's motion may involve planned, upcoming motion, or additionally or alternatively, motion that has already occurred. The vehicle motion model can be read from the vehicle's motion planning device in the reading step. This implementation offers the advantage of generating road models in a simpler and more accurate manner.

[0018] The proposed solution also provides an apparatus configured to perform, operate, or implement the steps of a variation of the method proposed herein within a corresponding device. The task on which the invention is based can also be solved quickly and efficiently through the apparatus-based implementation of the variation of the invention.

[0019] Therefore, the device may have at least one computing unit for processing signals or data, at least one storage unit for storing signals or data, at least one interface connected to a sensor or actuator for reading sensor signals from the sensor or for outputting data signals or control signals to the actuator, and / or at least one communication interface for reading or outputting data embedded in a communication protocol. The computing unit may be, for example, a signal processor, a microcontroller, etc., and the storage unit may be flash memory, EEPROM, or magnetic storage. The communication interface may be configured to read or output data wirelessly and / or via wired means, wherein a communication interface capable of reading or outputting wired data may, for example, electrically or optically read such data from or output such data to a corresponding data transmission line.

[0020] In this document, a device can be understood as an electrical device that processes sensor signals and outputs control signals and / or data signals accordingly. The device may have an interface, which can be constructed in hardware and / or software. In a hardware construction, the interface may, for example, be part of a so-called system ASIC, which contains a wide variety of functions of the device. However, it is also possible that the interface is its own integrated circuit or at least partially composed of discrete components. In a software construction, the interface may be, for example, a software module that exists on a microcontroller along with other software modules.

[0021] It is also advantageous to have a computer program product or computer program having program code that can be stored on a machine-readable carrier or storage medium (e.g., semiconductor memory, hard disk memory, or optical memory), and especially when the program product or program is implemented on a computer or device, the code is used to perform, implement, and / or manipulate the steps of the method according to any one of the above embodiments. Attached Figure Description

[0022] Embodiments of the proposed solution are shown in the accompanying drawings and further described below. The drawings show:

[0023] Figure 1 A schematic diagram of a vehicle having a device according to one embodiment is shown;

[0024] Figure 2 A flowchart illustrating a method for classification according to one embodiment is shown;

[0025] Figure 3 An image showing the Michelson contrast of the optical system;

[0026] Figure 4 The graph shows the spatial frequency and modulation transfer function of the optical system.

[0027] Figure 5 The diagram shows the difference in optical system performance.

[0028] Figure 6 The flow curve of the optical system is shown.

[0029] In the following description of advantageous embodiments of the invention, the same or similar reference numerals are used for elements shown in different figures that have similar functions, wherein repeated descriptions of these elements are omitted. Detailed Implementation

[0030] Figure 1 A schematic diagram of a vehicle 110 having a classification device 120, or sorting device 120, according to one embodiment is shown. The vehicle 110 relates to a motor vehicle, such as a passenger car, truck, or other commercial vehicle, particularly for highly automated driving. The vehicle 110 is located on a road 100 (here, for example, a two-lane road). The road 100 has road markings 102, here, for example, a center line. Furthermore, objects 105 are arranged on the road 100.

[0031] Vehicle 110, for example, has only one vehicle camera 112. Vehicle camera 112 is configured to detect the surrounding environment of vehicle 110. Vehicle camera 112 is also configured to provide image data 115 representing an area of ​​the surrounding environment of vehicle 110 that has at least road 100 or a portion of road 100. Furthermore, vehicle 110 has a device 120. Device 120 and vehicle camera 112 are connected to each other in a manner capable of signal transmission or data transmission. Device 120 is configured to classify objects 105 on road 100 in the surrounding environment of vehicle 110. For this purpose, device 120 is configured to generate a classification signal 130 when using image data 115. Device 120 has a reading device 122, an analysis and processing device 124, a obtaining device 126, and an execution device 128.

[0032] The reading device 122 is configured to read image data 115 from an interface connected to at least one vehicle camera 112 (in this case, the input interface 121 of device 120). The reading device 122 is connected to the analysis and processing device 124 in a manner capable of signal transmission or data transmission. The analysis and processing device 124 is configured to analyze and process the image data 115. Here, road markings 102 on road 100 are identified, a model M of the surface of road 100 is generated using the identified road markings 102, and objects 105 on road 100 are identified. Figure 1 In the illustration, for illustrative purposes, model M is symbolically drawn on road 100. The analysis and processing device 124 is connected to the acquisition device 126 in a manner capable of signal transmission or data transmission.

[0033] The determining device 126 is configured to: determine a first distance value between the vehicle camera 112 and object 105 represented by image data 115 object image points P11, P12; and determine a second distance value between the vehicle camera 112 and the surface of road 100 in the surrounding environment of object 105 road image points P21, P22 defined by model M. The determining device 126 is connected to the execution device 128 in a manner capable of signal transmission or data transmission. The execution device 128 is configured to perform a comparison of distance values ​​using at least one continuity criterion to distinguish between protruding object 105 and flat object 105, so as to classify object 105 as a protruding object or a flat object based on the comparison result.

[0034] Device 120 is also configured to generate a classification signal 130 and output or provide a classification signal for output via output interface 129. Classification signal 130 represents the result of a comparison, or classifies object 105 as a protruding object or a flat object. Device 120 is connected to at least one vehicle device 140 via output interface 129 in a manner capable of signal or data transmission. At least one vehicle device includes, for example, a controller for highly automated driving, a driver assistance system, etc., relating to vehicle 110.

[0035] Device 120 is specifically configured to perform classification using an algorithm based on the deviation or gap of optical flow and / or parallax of image data 115. More precisely, according to one embodiment, analysis processing device 124 is configured to use the deviation or gap of optical flow and / or parallax of image data 115. Here, additionally or alternatively, determining device 126 is configured to use the deviation or gap of optical flow and / or parallax of image data 115.

[0036] The analysis processing device 124 is specifically configured to generate a three-dimensional model of the surface of road 100 using an inference method when road markings 102 identified from image data 115 are used. According to one embodiment, the analysis processing device 124 is also configured to analyze and process a vehicle motion model representing the motion of vehicle 110 to determine the road characteristics of road 100 that determine this motion, and to generate a model M of the surface of road 100 using the road markings 102 and road characteristics identified from image data 115. Road characteristics, for example, involve road type, slope, curvature, etc. Road characteristics can be physical parameters that determine motion, such as speed, acceleration, etc. In other words, the analysis processing device 124 is here configured to also consider information from the vehicle motion model (including its speed) for modeling or 3D inference of the road surface. Thus, restrictive model assumptions, such as those for the radius of curvature of highways, are stored.

[0037] According to one embodiment, the determining device 126 is configured to determine a first distance value, either row-by-row or bar-by-bar, in at least one image of the surrounding environment represented by image data 115, namely, the distance between the vehicle camera 112 and object image points P11, P12 of the object 105 represented by image data 115. According to another embodiment, the determining device 126 is configured to determine the first distance value between the vehicle camera 112 and object image points P11, P12 along the edge of the object 105.

[0038] According to one embodiment, the execution device 128 is configured to perform a comparison taking into account the deviation between the first distance value and the second distance value as a continuity standard.

[0039] The actuator 128 is specifically configured to use at least one continuity criterion for comparison, which specifies the following conditions a) to c) for classifying object 105 as a protruding object: a) the deviation between the first distance value and the second distance value is minimized in the foot region of object 105 on the surface of road 100 defined by model M. For this purpose, in Figure 1 The illustration shows a pair of corresponding image points, namely the first object image point P11 and the first road image point P21. b) The deviation between the first distance value and the second distance value increases along the side edge of the object 105 in the direction away from the surface of the road 100. c) The deviation between the first distance value and the second distance value is greatest in the head region of the object 105, opposite the foot region. Therefore, in Figure 1 The illustration shows a pair of corresponding image points, namely the second object image point P12 and the second road image point P22.

[0040] Figure 2A flowchart of a classification method 200 according to one embodiment is shown. The classification method 200 can be implemented to classify objects on a road in the surrounding environment of a vehicle. Here, the classification method 200 can combine data from... Figure 1 Vehicles or similar vehicles and / or combinations from Figure 1 The method 200 for classification is implemented using a device or similar equipment. It includes a reading step 210, an analysis and processing step 220, a calculation step 230, and an execution step 240.

[0041] In reading step 210, image data is read from an interface of at least one vehicle camera connected to the vehicle. The image data represents at least one area of ​​the surrounding environment containing the road. Subsequently, in analysis processing step 220, the image data read in reading step 210 is analyzed. Here, road markings are identified. Furthermore, a model of the road surface is generated using the identified road markings. Furthermore, at least one object on the road is identified. In obtaining step 230, a first distance value between the vehicle camera and the object image points represented by the image data, and a second distance value between the vehicle camera and the road image points defined by the model on the surface of the road in the surrounding environment are obtained. Then, in execution step 240, a comparison of the distance values ​​is performed using at least one continuity criterion to distinguish between protruding objects and flat objects, so that objects are classified as protruding objects or flat objects based on the comparison results.

[0042] Figure 3 The Michelson contrast ratio K of the optical system is shown as a function of distance Z. m The curve is 300. Optical systems, for example, could involve vehicle cameras, such as those from... Figure 1 The vehicle camera. In graph 300, distance Z is plotted in meters [m] on the horizontal axis, while Michelson contrast K is plotted on the vertical axis. m .

[0043] Figure 4 Graph 400 shows the spatial frequency F and modulation transfer function (MTF) of an optical system. The optical system could, for example, involve a vehicle camera, such as one from... Figure 1 The vehicle camera. In graph 400, the spatial frequency F is plotted on the horizontal axis in line pairs per millimeter [lp / mm], while the modulation transfer function MTF is plotted on the vertical axis and the distance Z is plotted in meters [m]. Graph 400 is applicable to d s =1cm, y s =1.5m and X=0 foundation.

[0044] in other words, Figure 3 and Figure 4The spatial frequency F of the roadbed, granulated at 1 cm, and its Michelson contrast K with typical image sharpness are shown in the optical image. m It can be seen that the regularity or texture of a road surface cannot be easily represented. A homogeneous road surface does not possess a sufficiently coarse texture that can be represented in an image using affordable optics and a costly imager scanning device. Here, the focusing of a so-called "fixed-focus" camera is only limitedly good, as a fixed-focus camera must focus on objects at distances ranging from, for example, 10 meters to infinity. This results in aberrations, for example, caused by temperature drift of the focus and manufacturing tolerances. Therefore, the diffraction limit is not a crucial relevant criterion for vehicle cameras. The imager's scanning, and therefore the pixel size, cannot be arbitrarily reduced, as sensitivity or signal-to-noise ratio decreases accordingly. Other limiting details include power loss, transmission bandwidth, and memory size. Therefore, it is more advantageous to detect large irregular road features, such as uneven surfaces, asphalt joints, etc., whose spatial frequencies F are below, for example, 100 lp / mm in their images. Since there is a lack of adjacent front-to-back relationships for these objects with spatial frequencies F clearly above the representable 200 lp / mm, this front-to-back relationship cannot be easily represented. Therefore, it is impossible to determine whether objects are protruding or flatly embedded in the road surface by comparing their surroundings and the objects themselves. In the case of protrusions, gaps or discontinuities in flow vector lengths may occur, for example, at the upper edge of the object relative to the background. Furthermore, it is difficult to classify objects as "missing cargo" based on shape and texture because the classification does not follow known or predictable patterns. Missing cargo is not always, for example, a "Europalette" on which a neural network (CNN) can be trained. Therefore, the challenge is to find a method or analysis algorithm that can determine whether an object is protruding or non-protruding despite the lack of information about its surroundings. This is achieved according to an embodiment, where discontinuities in optical flow or discontinuities in the three-dimensional gaps of the object's edges relative to the foundation or road model are analyzed to classify objects as "protruding."

[0045] Figure 5 Figure 500 illustrates the gap difference in the optical system. The optical system could, for example, involve a vehicle camera, such as one from... Figure 1 The vehicle camera. In the difference graph 500, the distance Z is plotted in meters [m] on the horizontal axis, and the difference is plotted in pixels [px] on the vertical axis. In other words, Figure 5 The graph shows that, for X = 0, ys = 1.5m, and the foundation is 0.2m, the expected gap difference between the upper edge of the object and the expected background is shown above the distance Z, where the expected background is a three-dimensional model of the road surface.

[0046] Figure 6 The diagram 600 shows the flow rate curve of the optical system. The optical system could, for example, involve a vehicle camera, such as one from... Figure 1 The vehicle camera. In the flow difference curve diagram 600, the distance Z is plotted in meters [m] on the horizontal axis and the lateral offset X is plotted in meters [m] on the vertical axis. In other words, Figure 6 The graphs in the diagram show the flow difference at the top edge of the object above distance Z and show the lateral offset relative to the expected background, which is a 3D model of the road surface. The first flow difference graph shows the flow difference, or differential flow, at the top left corner of the object for ys = 1.5m in pixels [px]. The second flow difference graph shows the flow difference, or differential flow, at the middle of the top edge of the object for ys = 1.5m in pixels [px]. The third flow difference graph shows the flow difference, or differential flow, at the top right corner of the object for ys = 1.5m in pixels [px].

[0047] Referring again to the above figures, the embodiments and advantages are briefly described in general terms and in other ways.

[0048] According to an embodiment, road markings 102 are used to calculate an inferred three-dimensional model M of the surface of road 100. This can be done not only by means of stereo gaps but also by means of optical flow. Here, the following knowledge can be fully utilized: a) road markings 102 are structures with known shapes and / or trajectories that can be easily found in an image; b) the smoothness or non-protrusion of road markings 102 on the surface of road 100 can be assumed to be given. (For example, this does not apply to any asphalt joints that are indistinguishable from corresponding curved protrusions in an image.) For example, the gaps or optical flow are calculated row by row or line by line in an image of each representable object 105. Here, the immediate surrounding environment of object 105 does not need to be representable (due to the high spatial frequency of the texture of road 100).

[0049] More precisely, the gap or optical flow of object 105 at its limiting edge is compared with the gap or optical flow of the model of road 100, or rather, the three-dimensional model M. Here, in the case of protruding object 105 on road 100, the following conditions or continuity criteria are derived:

[0050] a) Continuous distance information derived from the gap or optical flow at the foot point of object 105 relative to the modeled road surface or relative to model M.

[0051] b) Compared to the modeled road background or model M, the discontinuity of distance information derived from gaps or optical flow increases along the object's side edges with increasing object height.

[0052] c) The discontinuity of distance information derived from gaps or optical flow is greatest along the upper edge of the object compared to the modeled road background or model M.

[0053] If criteria a) through c) are met, then object 105 involves a protruding object that is clearly located on road 100.

[0054] If an embodiment includes an "and / or" association between the first feature and the second feature, it should be interpreted as meaning that the embodiment, according to one implementation, has both the first feature and the second feature, while according to another implementation, it has either only the first feature or only the second feature.

Claims

1. A method (200) for classifying objects (105) on a road (100) in the surrounding environment of a vehicle (110), wherein, The method (200) comprises the following steps: Image data (115) is read (210) from an interface (121) of at least one vehicle camera (112) connected to the vehicle (110), wherein the image data (115) represents an area of ​​the surrounding environment having at least the road (100); The image data (115) is analyzed and processed (220), wherein road markings (102) of the road (100) are identified, a model (M) of the surface of the road (100) is generated using the identified road markings (102), and objects (105) on the road (100) are identified. Calculate (230) a first distance value between the at least one vehicle camera (112) and the object (105) represented by the image data (115) and a second distance value between the at least one vehicle camera (112) and the surface of the road (100) in the surrounding environment of the object (105) and the road image points (P21, P22) defined by the model (M). A comparison between the first distance value and the second distance value is performed using at least one continuity criterion (240) to distinguish between protruding objects and flat objects, so that the object (105) is classified as a protruding object or a flat object based on the result of the comparison.

2. The method (200) according to claim 1, characterized in that, In the analysis and processing step (220), the image data (115) is analyzed and processed in order to generate a three-dimensional model (M) of the surface of the road (100) using the identified road markings (102) by means of an interpolation method.

3. The method (200) according to claim 1 or 2, characterized in that, In the determination step (230), the first distance value is determined row by row or line by line in at least one image of the surrounding environment represented by the image data (115).

4. The method (200) according to claim 1 or 2, characterized in that, In the calculation step (230), the first distance value between the at least one vehicle camera (112) and the object image points (P11, P12) is calculated along the edge of the object (105), and / or wherein, In the execution step (240), the comparison is performed by taking into account the deviation between the first distance value and the second distance value as a continuity standard.

5. The method (200) according to claim 1 or 2, characterized in that, In the execution step (240), at least one continuity criterion is used, which specifies for classifying the object (105) as a protruding object that the deviation between the first distance value and the second distance value is minimal in the foot region of the object (105) on the surface of the road (100) defined by the model (M), increases along the side edge of the object (105) in the direction away from the surface of the road (100), and is maximum in the head region of the object (105) opposite to the foot region.

6. The method (200) according to claim 1 or 2, characterized in that, The deviation or gap of the optical flow and / or parallax of the image data (115) is used in the analysis processing step (220) and / or in the acquisition step (230).

7. The method (200) according to claim 1 or 2, characterized in that, In the analysis and processing step (220), the vehicle motion model representing the motion of the vehicle (110) is analyzed and processed in order to determine the road characteristics that determine the motion, and to generate the model (M) of the surface of the road (100) using the identified road markings (102) and the road characteristics.

8. A device (120) for classifying objects (105) on a road (100) in the surrounding environment of a vehicle (110), the device being configured to implement and / or manipulate the steps of the method (200) according to any one of claims 1 to 7 in corresponding units (122, 124, 126, 128).

9. A computer program product comprising a computer program configured to perform and / or manipulate the steps of the method (200) according to any one of claims 1 to 7.

10. A machine-readable storage medium on which a computer program product according to claim 9 is stored.

Images