Representation of a ground surface in surroundings of an active optical sensor system

Abstract

For the approximate representation of a ground surface (17) in surroundings of an active optical sensor system (3), a point cloud is obtained which contains a plurality of points which comprise a plurality of ground points which correspond to points on the ground surface (17). A first set of ground points of the plurality of ground points is determined, said ground points lying at least approximately on a first plane (13) which is parallel to a specified longitudinal axis of a specified sensor coordinates system. A first compensation curve (14) in the first plane (13) for the first set of ground points is determined. A second set of ground points of the plurality of ground points is determined, said ground points lying at least approximately on a second plane (15) which intersects the longitudinal axis at a specified point of the longitudinal axis. A second compensation curve (16) in the second plane (15) for the second set of ground points is determined. The approximate representation of the ground surface (17) is created based on the first compensation curve (14) and the second compensation curve (16).

Description

[0001] Representation of a ground surface in an environment of an active optical sensor system

[0002] The present invention is directed to a computer-implemented method for generating an approximate representation of a ground surface in the environment of an active optical sensor system. The invention is further directed to a computer-implemented method for determining a height of points of a point cloud above a ground surface in the environment of an active optical sensor system, a computer-implemented method for object detection, and a computer-implemented method for calibrating an active optical sensor system mounted on a motor vehicle.The invention is also directed to a method for at least partially automatically guiding a motor vehicle, in which such a computer-implemented method is carried out, a data processing device for carrying out such a computer-implemented method, an electronic vehicle guidance system with such a data processing device, and corresponding computer program products.

[0003] Active optical sensor systems, in particular lidar systems, for example as laser scanners or flash lidar systems, can be mounted on motor vehicles and used to implement a wide variety of driver assistance functions and / or other functions for partially automated driving of the motor vehicle. In particular, distances to objects in the vicinity of the motor vehicle can be determined, for example, by measuring the time of flight using active optical sensor systems of this type, and these distances can be used for driver assistance or other functions for at least partially automated driving of the motor vehicle.

[0004] The amount of data generated per frame by modern active optical sensor systems, whose detector arrays may have hundreds of rows and hundreds of columns, is very high, which places correspondingly high demands on the computing and storage resources that must be provided to process this data.

[0005] It is an object of the present invention to reduce the computing and memory resources required for processing data generated by an active optical sensor system. This object is achieved by the subject matter of the independent claim. Advantageous further developments and preferred embodiments are the subject matter of the dependent claims.

[0006] The invention is based on the idea of ​​approximately representing a ground surface in the vicinity of the active optical sensor system using two intersecting fitting lines to the ground points of a point cloud generated by the active optical sensor system.

[0007] According to one aspect of the invention, a computer-implemented method is provided for generating an approximate representation of a ground surface in the environment of an active optical sensor system. In this case, a point cloud is obtained which is generated by means of the active optical sensor system and represents the environment and contains a large number of points. The large number of points comprises a large number of ground points which correspond to points on the ground surface, in particular in addition to a large number of object points which correspond to one or more objects in the environment which are different from the ground surface. A first set of ground points of the large number of ground points which lie at least approximately on a first plane is determined. The first plane is parallel to a predetermined longitudinal axis of a predetermined sensor coordinate system of the active optical sensor system.A first regression curve in the first plane for the first set of ground points is determined. A second set of ground points of the plurality of ground points, which lie at least approximately on a second plane, is determined. The second plane intersects the longitudinal axis at a predetermined point of the longitudinal axis, in particular at a predetermined distance from the active optical sensor system. A second regression curve in the second plane for the second set of ground points is determined. The approximate representation of the ground surface is generated, in particular generated and stored, depending on the first regression curve and the second regression curve.

[0008] The active optical sensor system can, for example, be mounted on the motor vehicle or in an infrastructure device for detecting the surroundings of a motor vehicle. Unless otherwise stated, all steps of the computer-implemented method can be carried out by a data processing device which has at least one computing unit, in particular by a data processing device of the motor vehicle. In particular, the at least one computing unit is configured or adapted to carry out the steps of the computer-implemented method. For this purpose, the at least one computing unit can, for example, store a computer program which contains instructions which, when executed by the at least one computing unit, cause the at least one computing unit to carry out the computer-implemented method.

[0009] All computing units of the at least one computing unit can be part of the motor vehicle. However, it is also possible for all computing units of the at least one computing unit to be part of an external computing system outside the motor vehicle, for example, a backend server or a cloud computing system. It is also possible for the at least one computing unit to comprise both at least one vehicle computing unit of the motor vehicle and at least one external computing unit of the external computing system. The at least one vehicle computing unit can, for example, be comprised of one or more control units (ECUs) (electronic control units), and / or one or more zone control units (ZCLIs), and / or one or more domain control units (DCLIs) of the motor vehicle and / or the active optical sensor system.

[0010] Each embodiment of a computer-implemented method according to the invention is directly followed by a corresponding embodiment of a method according to the invention which is not purely computer-implemented by including the corresponding method steps for generating the depth image by means of the active optical sensor system.

[0011] By definition, an active optical sensor system comprises a light source for emitting light or light pulses. The light source can be configured, in particular, as a laser, for example, an infrared laser. Furthermore, by definition, an active optical sensor system comprises at least one optical detector for detecting reflected portions of the emitted light. The active optical sensor system is configured, in particular, to generate, process, or output one or more sensor signals based on the detected portions of the light. The active optical sensor system comprises, in particular, a two-dimensional detector array with a plurality of detector pixels, wherein the plurality of detector pixels each contains one or more optical detectors. It should be noted that an individual detector pixel of the detector array does not necessarily consist of a single optical detector.Rather, it is also possible for a group of several adjacent optical detectors to form a detector pixel. This is particularly possible when using single-photon avalanche diodes (SPADs). In other embodiments, however, it is also possible for a pixel to consist of exactly one optical detector, for example, a single photodiode or avalanche photodiode (APD).

[0012] Here and in the following, the term "light" can be understood to include electromagnetic waves in the visible, infrared, and / or ultraviolet ranges. Accordingly, the term "optical" can also be understood to refer to light in this sense.

[0013] The active optical sensor system is designed in particular as a lidar sensor system, for example as a laser scanner.

[0014] Laser scanners are a well-known type of lidar sensor system in which a laser beam is deflected by a light deflection device, allowing different deflection angles of the laser beam to be achieved. The light deflection device can, for example, contain a rotatably mounted mirror. Alternatively, the light deflection device can have a mirror element with a tiltable and / or pivotable surface. The mirror element can, for example, be designed as a microelectromechanical system (MEMS). The emitted laser beams can be partially reflected in the environment, and the reflected portions can in turn strike the laser scanner, in particular the light deflection device, which can direct them onto the detector array of the laser scanner. Each detector pixel of the detector array or each optical detector of the detector array generates, in particular, an associated detector signal based on the respectively detected portions.Based on the spatial arrangement of the respective detector pixel, together with the current position of the light deflection device, in particular its rotational position or its tilt and / or pivot position, the direction of incidence of the detected reflected components can be determined. An evaluation unit can also, for example, perform a time-of-flight measurement to determine the radial distance of the reflecting object. Alternatively or additionally, a method can be used to determine the distance by evaluating a phase difference between emitted and detected light.

[0015] Other types of lidar sensor systems are flash lidar sensor systems. These are non-scanning systems that do not require such a light deflection arrangement. The laser light generated by the light source is scattered by an optical element, emitting it in a single flash across a wide angle.

[0016] Typically, the data generated by an active optical sensor system can be represented as a point cloud, which is understood as a set of points, where each point is identified by corresponding coordinates in a two- or three-dimensional coordinate system. In the case of a three-dimensional point cloud, the three-dimensional coordinates can be determined, for example, by the direction of incidence of the reflected light components and the corresponding time of flight or the radial distance measured for that particular point. However, the information can also be preprocessed to obtain three-dimensional Cartesian coordinates for each of the points. In general, the points of a point cloud can be specified in an unordered or unsorted manner, in contrast, for example, to a camera image.In addition to the spatial information, namely the two- or three-dimensional coordinates, the point cloud can also store additional information or measured values ​​for the individual points, such as an echo pulse width (EPW) of the respective sensor signal.

[0017] The data generated by an active optical sensor system during a frame can, if the radial distance is determined accordingly, be understood as a 2.5-dimensional point cloud. This can be understood to mean that although a three-dimensional position is stored for each point in the point cloud, the entire three-dimensional space is not represented in the field of view of the active optical sensor system. This is because the emitted light pulses only reach the side of objects in the environment facing the active optical sensor system. As a result, parts of the objects lying behind them or other objects or parts thereof obscured by the objects are not represented by the point cloud, as is the case with a camera image. The sensor coordinate system can, for example, be understood as a Cartesian coordinate system that is rigidly connected to the active optical sensor system, in particular the detector array.The sensor coordinate system is then defined in particular by the longitudinal axis x, a transverse axis y perpendicular to the longitudinal axis x and a height axis z perpendicular to the longitudinal axis x and the transverse axis y.

[0018] For example, a position and an orientation, i.e., a position or pose, of the sensor coordinate system and thus of the active optical sensor system are defined relative to the motor vehicle on which it is mounted or relative to an infrastructure device on which it is mounted. A reference coordinate system can, for example, be understood as a Cartesian coordinate system that is rigidly connected to the motor vehicle or the infrastructure device. The reference coordinate system is then defined in particular by the longitudinal axis x', a transverse axis y' perpendicular to the longitudinal axis x', and a height axis z' perpendicular to the longitudinal axis x' and the transverse axis y'.A deviation of the position of the sensor coordinate system from the reference coordinate system can be determined, for example, by extrinsic calibration data of the active optical sensor system, for example in the form of a translation vector indicating the position deviation, and rotation angles or Euler angles of the sensor coordinate system with respect to the reference coordinate system. The position of the reference coordinate system in the environment is also known, for example, due to the known shape, size, and position of the motor vehicle or the infrastructure device in the environment.

[0019] The extrinsic calibration data indicates, among other things, the position of the sensor coordinate system relative to the local ground surface at the position of the active optical sensor system. For example, the local ground surface at the position of the active optical sensor system can be defined by a plane parallel to the x'-y' plane. The extrinsic calibration data can also be used to identify the ground points of the point cloud. The ground points of the plurality of ground points are, for example, those points of the plurality of points in the point cloud that lie on the plane of the local ground surface or within a specified tolerance range around this plane.It is therefore possible that points originally considered to be ground points, according to the approximate representation of the ground surface provided by the first regression curve and the second regression curve generated by the computer-implemented method according to the invention, are not actually real ground points. However, by determining the first regression curve and the second regression curve based on the first set and the second set of ground points, respectively, in particular by a known regression method or a fitting method, such outliers can be neglected.

[0020] The fact that the plurality of ground points correspond to points on the ground surface can therefore be understood in such a way that the ground points of the plurality of ground points apparently lie on the ground surface, or lie on the plane of the local ground surface or within a predetermined tolerance range around this plane.

[0021] The fact that the ground points of the first set of ground points lie at least approximately on the first plane can be understood as meaning that they lie on the first plane or within a predetermined tolerance range around the first plane. The fact that the ground points of the second set of ground points lie at least approximately on the second plane can be understood as meaning that they lie on the second plane or within a predetermined tolerance range around the second plane.

[0022] The first plane is parallel to the longitudinal axis x and therefore certainly not perpendicular to the vertical axis z. It can therefore be assumed that the first plane intersects the ground surface. For example, the first plane can be parallel to the vertical axis z and thus parallel to the xz plane. The first plane can, for example, contain a coordinate origin of the sensor coordinate system or another predefined reference point.

[0023] The second plane intersects the longitudinal axis x and is therefore definitely not parallel to the longitudinal axis x. It can therefore be assumed that the second plane intersects the ground surface. For example, the second plane can be parallel to the transverse axis y and thus parallel to the yz-plane. Furthermore, it can be assumed that the first best-fit curve and the second best-fit curve intersect each other.

[0024] In particular, the approximate representation of the ground surface by the first

[0025] best fit curve and the second best fit curve or by a

[0026] Approximate surface containing the first best fit curve and the second best fit curve. The type of the first best fit curve and the second best fit curve can be specified. The type of approximate surface can also be specified. If the first best fit curve and the second best fit curve are, for example, straight lines or straight line segments, the approximate surface can be defined by a plane, for example. If the first best fit curve and the second best fit curve are, for example, circular arcs or circular arc segments, the approximate surface can be defined by a torus surface or part of a torus surface, for example. In particular, if the first best fit curve and the second best fit curve have other shapes, other shapes of the approximate surface can be selected, for example other toroidal surfaces.In general, the approximate surface can be understood as a two-dimensional manifold in three-dimensional space.

[0027] The approximate representation of the ground surface can be used in a variety of ways for applications in the context of at least partially automated driving of motor vehicles. For example, the ground points can be removed from the point cloud and the approximate representation of the ground surface used instead, which can significantly reduce the memory and computational effort required for processing and / or storing the data from the active optical sensor system, while retaining relevant information regarding objects in the environment and the shape of the ground surface. Examples of possible applications include, but are not limited to, object localization, object classification, object tracking, obstacle detection, calibration or recalibration of the active optical sensor system, and so on.

[0028] For applications or application situations that may arise in a method according to the invention and which are not explicitly described herein, it may be provided that, according to the method, an error message and / or a request to enter user feedback is output and / or a standard setting and / or a predetermined initial state is set.

[0029] According to at least one embodiment, the first compensation curve is determined as a straight line or as a circular arc section.

[0030] In particular, the shape of the first best-fit curve is defined as a circular arc segment. As a result of the best-fit calculation to determine the first best-fit curve, a radius of the circular arc segment is then determined. The straight line can then be considered the limiting case of an infinitely large radius.

[0031] Since the ground surface in realistic scenarios has a comparatively small local curvature, a comparatively large radius of the circular arc segment is to be expected. This in turn means that the approximation using the circular arc segment is a very good approximation. However, due to the simplicity of a circular arc segment, it can be determined with little effort and requires very little storage space. For example, the radius can be at least 100 m. In the best-fit calculation to determine the first best-fit curve, the radius can also be restricted to a range [Rmin, °°], where, for example, Rmin is at least 50 m, or at least 100 m, or at least 200 m.

[0032] According to at least one embodiment, the second compensation curve is determined as a straight line or as a circular arc section.

[0033] In particular, the shape of the second best-fit curve is defined as a circular arc segment. As a result of the best-fit calculation to determine the second best-fit curve, a radius of the circular arc segment is then determined. The straight line can then be considered the limiting case of an infinitely large radius.

[0034] In the best fit calculation to determine the second best fit curve, the radius can also be limited to a range [R'm in , 00 [ , where, for example, R'min is at least 50 m or at least 100 m or at least 200 m. In particular, R'min can also be equal to Rmin in corresponding embodiments.

[0035] According to at least one embodiment, the approximate representation of the ground surface is determined as a two-dimensional approximate surface on which the first regression curve and the second regression curve lie.

[0036] The approximate representation of the ground surface is thus determined as a two-dimensional manifold in three-dimensional space. This enables, in particular, the simple use of the approximate representation of the ground surface for the aforementioned applications while still requiring minimal memory. The approximate surface can, for example, be used directly to estimate the distance of an object point from the ground surface or a corresponding height coordinate of the object point.

[0037] According to at least one embodiment, the approximate surface is defined as a torus surface or part of a torus surface. In such embodiments, the first regression curve and the second regression curve are each defined, in particular, as a straight line or as a segment of a circular arc.

[0038] A torus can be understood as the set of points that have a fixed distance r < R from a circle with radius R. This enables, in particular, a simple use of the approximate representation of the ground surface for the aforementioned applications while still requiring little memory.

[0039] According to at least one embodiment, the approximate surface is defined as a plane or part of a plane, or as a toroid or part of a toroid. A torus is a special case of a toroid with a circular cross-section.

[0040] According to at least one embodiment, the first plane is perpendicular to the transverse axis y of the sensor coordinate system, which is perpendicular to the longitudinal axis x.

[0041] In other words, the first plane is parallel to the xz plane. This makes it particularly easy to represent the first plane, and consequently the first regression curve, in the sensor coordinate system.

[0042] According to at least one embodiment, the second plane is perpendicular to the longitudinal axis.

[0043] In other words, the second plane is parallel to the yz-plane. This makes it particularly easy to represent the second plane, and consequently the second regression curve, in the sensor coordinate system.

[0044] According to at least one embodiment, the extrinsic calibration data of the active optical sensor system are obtained. The first set of ground points and the second set of ground points are determined depending on the extrinsic calibration data. In particular, the plurality of ground points is selected from the plurality of points depending on the extrinsic calibration data, and the first set of ground points and the second set of ground points are selected from the plurality of ground points.

[0045] In other words, the point cloud is filtered such that the filtered point cloud contains only ground points. The first set of ground points and the second set of ground points are selected from the filtered point cloud. The plurality of ground points is determined in particular such that they are those points of the plurality of points in the point cloud that lie at the level of the local ground surface or within a specified tolerance range around this level.

[0046] According to at least one embodiment, the extrinsic calibration data of the active optical sensor system are obtained. The first set of ground points and the second set of ground points are selected from the plurality of points depending on the extrinsic calibration data.

[0047] Therefore, the step of filtering the point cloud does not occur explicitly before selecting the first set of ground points and the second set of ground points. Rather, the filtering is performed implicitly when selecting the first set of ground points and the second set of ground points from the multitude of points.

[0048] According to at least one embodiment, a local representation of the ground surface is determined based on the extrinsic calibration data. The local representation of the ground surface corresponds in particular to the above-mentioned local ground surface locally at the position of the active optical sensor system. The local representation of the ground surface is extrapolated into the surrounding area. The plurality of ground points corresponds to a subset of the plurality of points that lie within a predetermined tolerance range around the extrapolated local representation of the ground surface. This includes points that lie on the extrapolated local representation of the ground surface.

[0049] To filter the point cloud, for example, the subset of the plurality of points that lie within the specified tolerance range around the extrapolated local representation of the ground surface is determined, and the remaining points are removed. Alternatively, when selecting the first set of ground points and the second set of ground points from the plurality of points, only the points of the specified subset can be considered. Extrapolating the local representation of the ground surface into the environment can be understood as considering the local representation of the ground surface not only locally at the position of the active optical sensor system, but also in the rest of the represented environment or a part thereof.

[0050] According to at least one embodiment, the extrinsic calibration data indicate the position of the sensor coordinate system relative to a vehicle coordinate system of a motor vehicle on which the active optical sensor system is mounted. The vehicle coordinate system corresponds in particular to the aforementioned reference coordinate system x', y', z'.

[0051] According to at least one embodiment, a preliminary first plane is determined which is parallel to the longitudinal axis of the sensor coordinate system. The point cloud contains a plurality of object points which correspond to points which do not lie on the ground surface. The preliminary first plane is shifted parallel to the longitudinal axis if it is determined that at least a part, in particular a predefined part, of the plurality of object points lies at least approximately on the preliminary first plane. The first plane corresponds to the shifted preliminary first plane. Otherwise, i.e. if it is determined that at least the part of the plurality of object points does not lie at least approximately on the preliminary first plane, or if it is not determined that at least the part of the plurality of object points lies at least approximately on the preliminary first plane, the first plane corresponds to the preliminary first plane.

[0052] The object points can, for example, be a further subset of the multitude of points that do not lie within the specified tolerance range around the extrapolated local representation of the ground surface. The object points can, for example, be all points of the multitude of points in the point cloud that are not ground points according to the explanations above.

[0053] The checking of whether at least some of the plurality of object points lie at least approximately on the preliminary first plane, and if necessary, the shifting of the preliminary first plane, can also be performed iteratively. Thus, the shifted first plane can be shifted until it is no longer determined that at least some of the plurality of object points lie at least approximately on the respective shifted first plane, which is then used as the first plane.

[0054] This will increase the reliability that the initial regression curve accurately represents the actual ground surface. This will increase the accuracy of the approximate representation of the ground surface.

[0055] As an alternative to checking whether at least some of the plurality of object points lie at least approximately on the provisional first plane, it can also be determined whether an error in the adjustment calculation for determining the first adjustment curve is greater than a predetermined maximum error. If this is the case, the provisional first plane can be shifted as described; otherwise, it cannot. This can, if necessary, also be combined with checking whether at least some of the plurality of object points lie at least approximately on the provisional first plane. Alternatively or additionally, it can also be taken into account whether the first adjustment curve has a predetermined minimum curve length within which the error is, in particular, less than or equal to the maximum error.

[0056] According to a further aspect of the invention, a computer-implemented method is provided for determining or estimating a height of points of a point cloud above a ground surface in an environment of an active optical sensor system. In this case, a computer-implemented method according to the invention is carried out for generating an approximate representation of the ground surface in the environment of the active optical sensor system. Each point of the plurality of points of the point cloud has a height position corresponding to the height axis z of the sensor coordinate system, which is perpendicular to the longitudinal axis x. For at least some of the plurality of points, which in particular are not part of the plurality of ground points, a height above the ground surface is calculated depending on the respective height position and the approximate representation of the ground surface.

[0057] Alternatively or additionally, a further point cloud can be obtained, which contains a plurality of additional points and is generated by the active optical sensor system. Each point of the plurality of additional points of the further point cloud has an elevation position corresponding to the elevation axis z of the sensor coordinate system, which is perpendicular to the longitudinal axis x. For at least some of the plurality of additional points, a height above the ground surface is calculated based on the respective elevation position and the approximate representation of the ground surface.

[0058] In particular, the height above the ground surface can be calculated as the difference between the elevation position, i.e., the z-coordinate, of the respective point and the corresponding z-coordinate of the approximate surface. The corresponding z-coordinate of the approximate surface is given, in particular, by the z-coordinate of the approximate surface at the xy position of the respective point.

[0059] Height above ground is a particularly useful parameter for a variety of possible applications in the context of automated and semi-automated driving of a motor vehicle. Height above ground can be used, for example, to determine a drivable area in the environment or to classify and / or track objects in the environment.

[0060] According to a further aspect of the invention, a computer-implemented method for object detection is provided. A computer-implemented method according to the invention is used to determine the height of points in the point cloud above the ground surface in the vicinity of the active optical sensor system. An object in the vicinity of the active optical sensor system is detected based on the calculated heights above the ground surface.

[0061] In various embodiments, the detection of the object may include a localization of the object in the environment and / or a classification of the object according to a predetermined object class.

[0062] Localization includes, for example, determining the position of the object, such as a bounding box or other geometric representation of the object, in the sensor coordinate system or the reference coordinate system. To classify the object, one of several predefined object classes can be selected for the object, at least depending on the calculated heights above the ground surface.

[0063] According to a further aspect of the invention, a computer-implemented method for calibrating an active optical sensor system mounted on a motor vehicle is provided. A computer-implemented method according to the invention is used to generate an approximate representation of a ground surface in the vicinity of the active optical sensor system. The position of the sensor coordinate system with respect to the vehicle coordinate system of the motor vehicle is corrected depending on the approximate representation of the ground surface.

[0064] In other words, the extrinsic calibration data are corrected depending on the approximate representation of the ground surface. The computer-implemented calibration method can therefore also be referred to as a computer-implemented method for extrinsic calibration or recalibration, in particular extrinsic recalibration, of the active optical sensor system.

[0065] This can increase the reliability and / or accuracy of applications for at least partially automatic guidance of the motor vehicle, which use the point cloud or other data of the active optical sensor system.

[0066] For example, a deviation of the approach surface from the local ground surface at the position of the active optical sensor system can be determined and the extrinsic calibration data are corrected depending on the deviation.

[0067] According to a further aspect of the invention, a method for at least partially automatically driving a motor vehicle is specified. The motor vehicle has an active optical sensor system. The method involves performing a computer-implemented method according to the invention for generating an approximate representation of a ground surface and / or a computer-implemented method according to the invention for determining the height of points of a point cloud above the ground surface and / or a computer-implemented method according to the invention for object recognition and / or a computer-implemented method according to the invention for calibrating an active optical sensor system.

[0068] Depending on the approximate representation of the ground surface and / or depending on the calculated heights above the ground surface and / or depending on the result of the object detection and / or the corrected extrinsic calibration data, at least one control signal for at least partially automatically guiding the motor vehicle and / or driver assistance information for assisting a driver of the motor vehicle in guiding the motor vehicle is generated.

[0069] The at least one control signal can, for example, be provided to one or more actuators of the motor vehicle, including, for example, one or more brake actuators and / or one or more steering actuators and / or one or more drive motors of the motor vehicle. The one or more actuators can influence a longitudinal and / or lateral control of the motor vehicle in order to guide the motor vehicle at least partially automatically.

[0070] The driver assistance information can be output via an output device of the motor vehicle, for example a display and / or an audio output system and / or a haptic output system.

[0071] According to a further aspect of the invention, a data processing device with at least one computing unit is provided. The at least one computing unit is configured to perform a computer-implemented method for generating an approximate representation of a ground surface and / or a computer-implemented method according to the invention for determining the height of points of a point cloud above the ground surface and / or a computer-implemented method according to the invention for object recognition and / or a computer-implemented method according to the invention for calibrating an active optical sensor system.

[0072] In the present disclosure, a computing unit can be understood, for example, as a data processing device with processing circuits. A computing unit can therefore perform computing operations to process data. The computing operations can also include indexed accesses to a data structure, for example, a look-up table (LUT).

[0073] A computing unit can in particular comprise one or more computers, one or more microcontrollers and / or one or more integrated circuits, for example one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), and / or one or more single-chip systems (SoCs). The computing unit can also contain one or more processors, for example one or more microprocessors, one or more central processing units (CPUs), one or more graphics processing units (GPUs), and / or one or more signal processors, in particular one or more digital signal processors (DSPs). The computing unit can also comprise a physical or virtual cluster of computers or other of the aforementioned units.

[0074] A computing unit may also include one or more hardware and / or software interfaces and / or one or more memory units. A memory unit may be embodied as a volatile data memory, for example, a dynamic random access memory (DRAM) or a static random access memory (SRAM), or as a non-volatile data memory, for example, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory or flash EEPROM, a ferroelectric random access memory (FRAM),a magnetoresistive random access memory (MRAM) or a phase-change random access memory (PCRAM).

[0075] According to a further aspect of the invention, an electronic vehicle guidance system for a motor vehicle is provided, which comprises a data processing device according to the invention. The at least one computing unit is configured to carry out a method according to the invention for the at least partially automatic guidance of a motor vehicle.

[0076] An electronic vehicle guidance system can be understood as an electronic system designed to guide a vehicle fully automatically or autonomously, in particular without requiring driver intervention. The vehicle automatically performs all required functions, such as steering, braking, and / or acceleration maneuvers, monitoring and detecting road traffic, and corresponding reactions. In particular, the electronic vehicle guidance system can implement a fully automatic or fully autonomous driving mode of the motor vehicle according to level 5 of the SAE J3016 classification. An electronic vehicle guidance system can also be understood as an advanced driver assistance system (ADAS), which supports the driver in partially automated or semi-autonomous driving.In particular, the electronic vehicle guidance system can implement a partially automated or semi-autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification. Here and below, "SAE J3016" refers to the corresponding standard in the April 2021 version.

[0077] The at least partially automated vehicle guidance may therefore include driving the vehicle according to a fully automated or fully autonomous driving mode of Level 5 according to SAE J3016. The at least partially automated vehicle guidance may also include driving the vehicle according to a partially automated or semi-autonomous driving mode according to Levels 1 to 4 according to SAE J3016.

[0078] According to at least one embodiment of the electronic vehicle guidance system, it comprises the active optical sensor system.

[0079] According to at least one embodiment, the active optical sensor system is designed as a lidar sensor system, in particular as a laser scanner.

[0080] According to at least one embodiment, the active optical sensor system comprises a detector array, in particular for generating the point cloud, with a plurality of pixels arranged corresponding to a plurality of columns and rows of the detector array.

[0081] The rows are arranged consecutively in particular parallel to the height axis z of the sensor coordinate system and the columns are arranged consecutively in particular parallel to the transverse axis y of the sensor coordinate system.

[0082] According to at least one embodiment, the total number of rows of the detector array is at least 100, for example, 100 to 5,000, and / or the total number of columns of the detector array is at least 100, for example, 100 to 5,000. Further embodiments of the electronic vehicle guidance system according to the invention follow directly from the various configurations of the computer-implemented methods according to the invention, and vice versa. In particular, individual features and corresponding explanations, as well as advantages relating to the various embodiments of the computer-implemented methods according to the invention, can be transferred analogously to corresponding embodiments of the electronic vehicle guidance system according to the invention.

[0083] According to a further aspect of the invention, a computer program with instructions is specified. Upon execution of the further instructions by a data processing device, in particular a data processing device according to the invention, the instructions cause the data processing device to carry out a computer-implemented method for generating an approximate representation of a ground surface and / or a computer-implemented method according to the invention for determining the height of points of a point cloud above the ground surface and / or a computer-implemented method according to the invention for object recognition and / or a computer-implemented method according to the invention for calibrating an active optical sensor system and / or a method according to the invention for at least partially automatically guiding a motor vehicle.

[0084] The instructions can be provided, for example, as program code. The program code can be provided, for example, as binary code or assembly code and / or as source code of a programming language, for example, C, and / or as a program script, for example, Python.

[0085] According to a further aspect of the invention, a computer-readable storage medium is provided which stores a computer program according to the invention.

[0086] The computer program and the computer-readable storage medium are each computer program products containing the instructions.

[0087] Further features of the invention emerge from the claims, the figures and the description of the figures. The features and combinations of features mentioned above in the description as well as the features and combinations of features mentioned below in the description of the figures and / or shown in the figures can be encompassed by the invention not only in the respectively specified combination, but also in other combinations. In particular, the invention can also encompass embodiments and combinations of features that do not have all the features of an originally formulated claim. Furthermore, the invention can encompass embodiments and combinations of features that go beyond the combinations of features set out in the backreferences to the claims or deviate from them.

[0088] The invention is explained in more detail below using specific embodiments and associated schematic drawings. In the figures, identical or functionally equivalent elements may be provided with the same reference numerals. The description of identical or functionally equivalent elements may not necessarily be repeated for different figures.

[0089] The figures show

[0090] Fig. 1 is a schematic representation of a motor vehicle with an exemplary embodiment of an electronic vehicle guidance system according to the invention;

[0091] Fig. 2 is a schematic representation of an active optical sensor system of a further exemplary embodiment of an electronic vehicle guidance system according to the invention;

[0092] Fig. 3 schematically shows an object and a ground surface in the vicinity of an active optical sensor system;

[0093] Fig. 4 schematically shows the object and the ground surface according to Fig. 3 and a first plane according to an exemplary embodiment of a computer-implemented method according to the invention for generating an approximate representation of a ground surface;

[0094] Fig. 5 schematically shows the object and the ground surface according to Fig. 3 and a second plane according to another exemplary embodiment of a computer-implemented method according to the invention for generating an approximate representation of a ground surface; Fig. 6 schematically shows the object and the ground surface according to Fig. 3 and a second plane according to another exemplary embodiment of a computer-implemented method according to the invention for generating an approximate representation of a ground surface; and

[0095] Fig. 7 schematically shows an approximate representation of a ground surface according to a further exemplary embodiment of a computer-implemented method according to the invention for generating an approximate representation of a ground surface.

[0096] Fig. 1 schematically shows a motor vehicle 1 with an exemplary embodiment of an electronic vehicle guidance system 2 according to the invention. The electronic vehicle guidance system 2 has at least one computing unit 4 and an active optical sensor system 3.

[0097] An exemplary embodiment of the active optical sensor system 3 is shown schematically in Fig. 2. The active optical sensor system 3 has a housing 7 and an emitter unit 9 for emitting light 6a into an external environment of the active optical sensor system 3. The emitted light 6a passes, for example, through a window 8 of the housing 7. The active optical sensor system 3 has a control unit 11 which is configured to control a deflection device 18, for example a rotatable mirror, of the active optical sensor system 3, to deflect the light 6a in different directions and thereby scan the environment of the active optical sensor system 3.

[0098] The active optical sensor system 3 has a detector unit with a two-dimensional detector array 10, which has a plurality of detector pixels arranged, for example, in a plurality of rows and a plurality of columns. If portions 6b of the emitted light 6a are reflected by an object 5 in the environment, these portions 6b can be reflected back toward the active optical sensor system 3 and, for example, pass through the window 8 again, where they are directed by the deflection device 18 onto the detector array 10 and detected by it. Depending on the corresponding detector signals, the control unit 11 can then calculate a radial distance of the object 5 from the active optical sensor system 3, for example by measuring the light propagation time. The active optical sensor system 3 is designed, in particular, as a lidar sensor system in the manner of a laser scanner.The emitter unit 9 thus has one or more laser light sources, in particular laser diodes, for generating the light 6a. The detector pixels each have one or more photodetectors, which can be configured, for example, as photodiodes, avalanche photodiodes (APDs), or single-photon avalanche photodiodes (SPADs). Possible optical components in the optical path are representatively shown in Fig. 2 by a lens 12.

[0099] By means of the at least one computing unit 4, a computer-implemented method according to the invention for generating an approximate representation of a ground surface 17 in an environment of the active optical sensor system 3 can be carried out, as schematically illustrated in Figures 3 to 7 for various exemplary embodiments. The ground surface 17 corresponds in particular to a road surface of a roadway on which the motor vehicle 1 is located.

[0100] The active optical sensor system 3 generates a point cloud representing the surroundings of the active optical sensor system 3, and thus in particular of the motor vehicle 1, and containing a plurality of points, including a plurality of ground points corresponding to points on the ground surface 17. The at least one computing unit 4 receives the point cloud and constructs a first plane 13 that is parallel to the longitudinal axis x of a sensor coordinate system of the active optical sensor system 3. In some embodiments, the first plane 13, as schematically illustrated in Fig. 4, is parallel to the xz-plane spanned by the longitudinal axis x and the height axis y of the sensor coordinate system. The at least one computing unit 4 determines a first set of ground points from the plurality of ground points that lie at least approximately on the first plane 13 and determines a first regression curve 14 in the first plane 13 for the first set of ground points.

[0101] The at least one computing unit 4 constructs a second plane 15 which intersects the longitudinal axis x at a predetermined point of the longitudinal axis x, for example at a distance of 3 m to 20 m from the active optical sensor system 3. In some embodiments, the second plane 15 is, as schematically shown in Fig. 6, perpendicular to the longitudinal axis x and thus parallel to the yz-plane spanned by the transverse axis y of the sensor coordinate system and the height axis y of the sensor coordinate system.

[0102] The at least one computing unit 4 determines a second set of ground points from the plurality of ground points that lie at least approximately on the second plane 15. The at least one computing unit 4 determines a second regression curve 16 in the second plane 15 for the second set of ground points.

[0103] The approximate representation of the ground surface 17 is generated and stored depending on the first fitting curve 14 and the second fitting curve 16. For example, the approximate representation of the ground surface 17 can be stored as the first fitting curve 14 and the second fitting curve 16. However, the approximate representation of the ground surface 17 can also be determined as an approximate surface on which the first fitting curve 14 and the second fitting curve 16 lie.

[0104] In some embodiments, the first regression curve 14 and the second regression curve 16 are defined as segments of a circular arc, which includes straight lines as a limiting case of infinitely large radii. The approximate surface can then be defined, for example, as part of a torus surface.

[0105] Parts of the first compensation curve 14 can belong to an object 5, 5', 5" and not to the ground surface 17, as shown in Fig. 4. Accordingly, in various embodiments, a part of the compensation curve 14 can be selected that belongs to the ground surface 17. This can be done using predetermined extrinsic calibration data of the active optical sensor system 3, which data specify the position of the active optical sensor system 3, in particular of the sensor coordinate system x, y, z, in a reference coordinate system, for example a vehicle coordinate system of the motor vehicle 1. In this way, a section of the first compensation curve 14 can be selected that fits the ground surface 17 within predetermined tolerances.

[0106] In various embodiments, when the first compensation curve 14 crosses one or more objects 5, 5', 5", the first plane 13 can be shifted, in particular shifted parallel, until a sufficiently long curve 14' is found, as shown in Fig. 5. The second plane 15 can also be shifted analogously, in particular parallel to the longitudinal axis x, if it passes through corresponding objects 5, 5', 5".

[0107] Based on the approximate representation of the ground surface 17, various functions and methods can be performed that can be used for at least partially automatically guiding the motor vehicle 1. In particular, the at least one computing unit 4 can generate at least one control signal for at least partially automatically guiding the motor vehicle and / or driver assistance information to support a driver of the motor vehicle in guiding the motor vehicle, depending on the approximate representation of the ground surface 17.

[0108] The at least one control signal can, for example, be provided to one or more actuators of the motor vehicle 1, including, for example, one or more brake actuators and / or one or more steering actuators and / or one or more drive motors of the motor vehicle 1. The one or more actuators can influence a longitudinal and / or lateral control of the motor vehicle in order to guide the motor vehicle 1 at least partially automatically.

[0109] The driver assistance information can be output via an output device of the motor vehicle 1, for example a display and / or an audio output system and / or a haptic output system.

[0110] As described, in particular with reference to the figures, the invention makes it possible to reduce the computing and storage resources required for processing data generated by an active optical sensor system.

Claims

Patent claims 1. A computer-implemented method for generating an approximate representation of a ground surface (17) in an environment of an active optical sensor system (3), wherein a point cloud generated by means of the active optical sensor system (3) is obtained, which represents the environment and contains a plurality of points, comprising a plurality of ground points corresponding to points on the ground surface (17); a first set of ground points of the plurality of ground points is determined, which lie at least approximately on a first plane (13) which is parallel to a predetermined longitudinal axis of a predetermined sensor coordinate system of the active optical sensor system (3); a first regression curve (14) in the first plane (13) is determined for the first set of ground points;a second set of ground points of the plurality of ground points is determined which lie at least approximately on a second plane (15) which intersects the longitudinal axis at a predetermined point on the longitudinal axis; a second regression curve (16) is determined in the second plane (15) for the second set of ground points; and the approximate representation of the ground surface (17) is generated as a function of the first regression curve (14) and the second regression curve (16); 2. The computer-implemented method according to claim 1, wherein the first regression curve (14) is determined as a straight line or as a circular arc segment; and / or the second regression curve (16) is determined as a straight line or as a circular arc segment.

3. Computer-implemented method according to one of the preceding claims, wherein the approximate representation of the ground surface (17) is two-dimensional approximate surface is determined on which the first compensation curve (14) and the second compensation curve (16).

4. The computer-implemented method of claim 3, wherein the two-dimensional approximate surface is determined as a plane or part of a plane or as a toroid or part of a toroid.

5. A computer-implemented method according to any one of the preceding claims, wherein the first plane (13) is perpendicular to a predetermined transverse axis of the sensor coordinate system, which is perpendicular to the longitudinal axis; and / or the second plane (15) is perpendicular to the longitudinal axis.

6. A computer-implemented method according to any one of the preceding claims, wherein extrinsic calibration data of the active optical sensor system (3) are obtained, and the plurality of ground points are selected from the plurality of points depending on the extrinsic calibration data, and the first set of ground points and the second set of ground points are selected from the plurality of ground points; or the first set of ground points and the second set of ground points are selected from the plurality of points depending on the extrinsic calibration data.

7. The computer-implemented method of claim 6, wherein a local representation of the ground surface (17) is determined based on the extrinsic calibration data; the local representation of the ground surface (17) is extrapolated into the surroundings; and the plurality of ground points corresponds to a subset of the plurality of points that lie within a predetermined tolerance range around the extrapolated local representation of the ground surface (17).

8. Computer-implemented method according to one of claims 6 or 7, wherein the extrinsic calibration data comprises a position of the sensor coordinate system with respect to a vehicle coordinate system of a motor vehicle (1) on which the active optical sensor system (3) is mounted.

9. A computer-implemented method according to any one of the preceding claims, wherein a preliminary first plane (13) is determined which is parallel to the longitudinal axis; the point cloud contains a plurality of object points which correspond to points which do not lie on the ground surface (17); the preliminary first plane (13) is translated parallel to the longitudinal axis if it is determined that at least a portion of the plurality of object points lies at least approximately on the preliminary first plane (13) and the first plane (13) corresponds to the translated preliminary first plane (13), and otherwise the first plane (13) corresponds to the preliminary first plane (13).

10. A computer-implemented method for determining a height of points of a point cloud above a ground surface (17) in an environment of an active optical sensor system (3), wherein a computer-implemented method for generating an approximate representation of a ground surface (17) in the environment of the active optical sensor system (3) is carried out according to one of the preceding claims; each point of the plurality of points of the point cloud has a height position corresponding to a height axis of the sensor coordinate system that is perpendicular to the longitudinal axis; for at least some of the plurality of points, a height above the ground surface (17) is calculated depending on the respective height position and the approximate representation of the ground surface (17).

11. A computer-implemented method for object detection, wherein a computer-implemented method for determining a height of points of the point cloud above a ground surface (17) in an environment of the active optical sensor system (3) is carried out according to claim 10; an object is detected in the environment of the active optical sensor system (3) based on the calculated heights above the ground surface (17).

12. A computer-implemented method according to claim 11, wherein the detection of the object comprises a localization of the object in the environment and / or a classification of the object according to a predetermined object class.

13. A computer-implemented method for calibrating an active optical sensor system (3) mounted on a motor vehicle (1), wherein a computer-implemented method for generating an approximate representation of a ground surface (17) in the environment of the active optical sensor system (3) according to one of claims 1 to 9 is carried out; and a position of the sensor coordinate system with respect to a vehicle coordinate system of the motor vehicle (1) is corrected depending on the approximate representation of the ground surface (17).

14. A method for the at least partially automatic guidance of a motor vehicle (1), wherein a computer-implemented method according to one of the preceding claims is carried out and at least one control signal for the at least partially automatic guidance of the motor vehicle (1) is generated as a function of the point cloud and as a function of the approximate representation of the ground surface (17); and / or driver assistance information for supporting a driver of the motor vehicle (1) in guiding the motor vehicle is generated as a function of the point cloud and as a function of the approximate representation of the ground surface (17).

15. Data processing device with at least one computing unit (4, 11) which is designed to carry out a computer-implemented method according to one of the claims 1 to 13.

16. Electronic vehicle guidance system (2) for a motor vehicle (1), comprising a data processing device according to claim 15, wherein the at least one computing unit (4, 11) is configured to carry out a method according to claim 14.

17. Electronic vehicle guidance system (2) according to claim 16, which comprises the active optical sensor system (3).

18. The electronic vehicle guidance system (2) according to claim 17, wherein the active optical sensor system (3) comprises a detector array (10) having a plurality of pixels arranged corresponding to a plurality of columns and rows of the detector array (10); and a total number of rows of the detector array (10) is at least 100.

19. A computer program product comprising instructions which, when executed by a data processing device, cause the data processing device to perform a computer-implemented method according to any one of claims 1 to 13 and / or a method according to claim 14.

Images