Self-localization of a motor vehicle

Abstract

For the self-localization of a motor vehicle (1), a point cloud is generated by emitting light pulses and detecting reflected portions of the emitted light pulses by a detector array (3b) of the active optical sensor system (3). An image of the surroundings is generated by detecting ambient light incident on the detector array (3b). A first vehicle position of the motor vehicle (1) is determined as a function of the point cloud, and a second vehicle position of the motor vehicle (1) is determined as a function of the image. A position deviation of the first vehicle position from the second vehicle position is determined. A consolidated vehicle position (7b) is determined as the first vehicle position if the position deviation is smaller than a predefined threshold, and the consolidated vehicle position (7b) is determined as the second vehicle position if the position deviation is greater than the threshold.

Description

[0001] The present invention is directed to a method for self-localization of a motor vehicle, wherein a point cloud is generated for a frame interval by emitting light pulses into the surroundings of the motor vehicle by a transmitter unit of an active optical sensor system mounted on the motor vehicle, and by detecting reflected portions of the emitted light pulses by a detector arrangement of the active optical sensor system. The invention is also directed to a method for at least partially automatically guiding a motor vehicle, wherein such a method is carried out for self-localization. The invention is also directed to a corresponding active optical sensor system for a motor vehicle, to an electronic vehicle guidance system with such an active optical sensor system, and to corresponding computer program products.

[0002] Self-localization methods are used by autonomous vehicles or partially automated vehicles to track their own position in a reference coordinate system, for example, a coordinate system of a digital map. The tracked vehicle position can then be used for trajectory planning, collision avoidance, or various other applications for at least partially automatic guidance of the motor vehicle. For example, tracking algorithms based on Kalman filters or similar can be used to track the vehicle position based on environmental sensor data generated by one or more of the motor vehicle's environmental sensor systems. Methods for simultaneous localization and mapping (SLAM) can also be used for this purpose.

[0003] Due to the high safety relevance, it is generally desirable to validate or cross-check the tracked vehicle position to increase its reliability. One possible way to achieve this is to use multi-fusion approaches that utilize multiple sensors, such as cameras, radar systems, and / or lidar systems. However, this increases the complexity of the system and data evaluation, for example, with regard to temporal synchronization or calibration. Furthermore, inconsistencies between data from different sensors can cause problems. Due to the increased complexity, the risk of system failure also increases. Furthermore, especially for safety-relevant applications, it may be necessary for the reliability of certain tracking and object detection functions to be sufficiently high if only a single sensor system or a single type of sensor system is used.

[0004] An object of the present invention is to increase the reliability of self-localization of a motor vehicle, in particular without using multiple environmental sensor systems.

[0005] This object is achieved by the respective subject matter of the independent claims. Further embodiments and preferred embodiments are the subject matter of the dependent claims.

[0006] The invention is based on the idea of ​​using the same detector arrangement of an active optical sensor system to generate a point cloud based on the detection of reflected portions of emitted light pulses, on the one hand, and to generate an image based on detected ambient light, on the other. Respective vehicle positions are determined based on the point cloud and the image. A consolidated vehicle position is then determined based on a deviation between the two vehicle positions.

[0007] According to one aspect of the invention, a method for self-localization of a motor vehicle is provided. A point cloud is generated for a frame interval of an active optical sensor system mounted on the motor vehicle by emitting light pulses into the surroundings of the motor vehicle, in particular during the frame interval, by a transmitter unit of the active optical sensor system and by detecting reflected portions of the emitted light pulses by a detector array of the active optical sensor system. An image of the surroundings is generated by detecting ambient light incident on the detector array, in particular during the frame interval. A first vehicle position of the motor vehicle is determined based on the point cloud, and a second vehicle position of the motor vehicle is determined based on the image.A position deviation of the first vehicle position from the second vehicle position is determined. A consolidated vehicle position of the motor vehicle is determined. If the position deviation is less than a predefined threshold, the consolidated vehicle position is given by the first vehicle position. If the position deviation is greater than the threshold, the consolidated vehicle position is given by the second vehicle position.

[0008] Unless otherwise stated, the steps of the method that are not performed by the transmitter unit or the detector arrangement can be performed, for example, by a data processing device that has at least one computing unit, in particular a data processing device of the motor vehicle. For this purpose, the at least one computing unit can, for example, store a computer program with instructions that, when executed by the at least one computing unit, cause the at least one computing unit to perform the respective method steps.

[0009] All computing units of the at least one computing unit can be contained in the motor vehicle. However, it is also possible for all computing units of the at least one computing unit to be part of an external computing system external to the motor vehicle, for example, a backend server or a cloud computing system. It is also possible for the at least one computing unit to have at least one vehicle computing unit of the motor vehicle and at least one external computing unit contained in the external computing system. The at least one vehicle computing unit can, for example, be contained in one or more electronic control units (ECUs) and / or one or more zone control units (ZCUs) and / or one or more domain control units (DCUs) of the motor vehicle and / or by the active optical sensor system.

[0010] The term self-localization of a motor vehicle refers, for example, to the determination of a position of the vehicle—in the present method, a consolidated vehicle position—in a predefined reference coordinate system, for example, a map coordinate system of a digital map, also referred to as a global coordinate system, by the motor vehicle itself. Consequently, self-localization does not use images or similar that show the motor vehicle in its surroundings. In particular, the consolidated vehicle position and, for example, the first vehicle position and the second vehicle position are given in the reference coordinate system. Several approaches to self-localization are known, which can be used, for example, to determine the first vehicle position based on the point cloud and / or to determine the second vehicle position based on the image.These include SLAM approaches or approaches based on Kalman filters or other tracking algorithms, and so on.

[0011] Here and below, "light" may be understood to include electromagnetic waves in the visible, infrared, and / or UV ranges. Accordingly, the term "optical" may be understood to refer to light according to this meaning.

[0012] By definition, an active optical sensor system, in this case the transmitter unit, includes a light source for emitting light or light pulses. The light source can be implemented, for example, as a laser, in particular as an infrared laser. Furthermore, an active optical sensor system, in this case the detector arrangement, by definition includes at least one optical detector for detecting reflected portions of the emitted light. In particular, the active optical sensor system is configured to generate one or more sensor signals based on the detected portions of the light and to process and / or output the sensor signals. Lidar sensor systems, for example, are active optical sensor systems.

[0013] A laser scanner is a known embodiment of lidar sensor systems in which a laser beam is generated by one or more laser diodes of the lidar sensor system, particularly in the form of laser pulses, and is deflected by a light deflection arrangement so that different deflection angles of the laser beam can be realized. The light deflection arrangement can, for example, contain one or more rotatably mounted mirrors. Alternatively, the light deflection arrangement can include a mirror element with a tiltable and / or pivotable surface. The mirror element can, for example, be configured as a micro-electro-mechanical system (MEMS). The emitted laser beams can be partially reflected in the environment, and the reflected portions can in turn strike the laser scanner, particularly the light deflection arrangement, which can direct them onto the detector array of the laser scanner.In particular, each detector pixel of the detector array can generate a respective detector signal based on the detected light. Based on the spatial arrangement of the respective detector pixel together with the current position of the light-deflecting array, in particular its rotational position or its tilt and / or pivot position, it is thus possible to determine the direction of incidence of the detected reflected light components. The at least one control and / or evaluation unit of the laser scanner can, for example, perform a time-of-flight measurement or an indirect time-of-flight measurement to determine a radial distance of the reflecting object.

[0014] Reflected portions can be understood as the portion of emitted light that is reflected by surrounding objects, including the road surface. This does not necessarily mean specularly reflected light. Rather, the reflected portions can also include retroreflected and / or scattered light.

[0015] Other designs of lidar systems are flash lidar systems. These are non-scanning systems that do not require the aforementioned light deflection arrangement. The laser light generated by the light source is dispersed by an optical element to illuminate a wide angle in a single flash.

[0016] The detector array may include a plurality of detector pixels. A single detector pixel of the detector array does not necessarily consist of a single optical detector. Rather, it is also possible for a group of several adjacent optical detectors to form a detector pixel. The latter is particularly possible when single-photon avalanche diodes (SPADs) are used as optical detectors. In other embodiments, however, it is also possible for a pixel to consist of exactly one optical detector, for example, a single photodiode or a single avalanche photodiode (APD).

[0017] Since the detector array includes a plurality of detector pixels arranged, for example, according to a plurality of columns (e.g., several tens to several hundred columns) and a plurality of rows (e.g., several tens to several hundred rows), in addition to generating the point cloud, the detector array can also be used to capture two-dimensional images of the environment in the same or similar manner as a camera, such as a thermal camera. It can be understood that the image of the environment is generated by detecting light incident on the detector array during the frame interval.

[0018] In other words, no light pulses need to be emitted into the environment to generate the image, and no distance needs to be calculated using, for example, time-of-flight (ToF) measurement. In other words, the image is generated independently of the light pulses emitted during the frame interval or other respective frame intervals, and independently of the corresponding reflected portions. The image therefore corresponds, in particular, to a monochromatic image, for example, a grayscale image.

[0019] In particular, the image contains a plurality of image pixels, with each image pixel corresponding to one of the detector pixels. For each image pixel, the image stores a pixel value dependent on the amount of energy of the ambient light incident on the respective detector pixel during a corresponding exposure duration within the first frame interval. Compared to the images, each point in the point cloud also corresponds to one of the detector pixels; however, not every detector pixel necessarily has to detect reflected portions of the emitted light pulses. In particular, even the influence of ambient light can be eliminated to generate the point cloud using known ambient light suppression algorithms.For each point in the point cloud, in addition to the respective position of the detector pixel, further information can be stored, including, for example, the radial distance determined by direct or indirect time-of-flight measurement, an energy measure relating to the amount of optical energy of the respective reflected portions of light pulses, such as an echo pulse width (EPW), an area under a signal pulse, etc. Therefore, the image can be processed using conventional image processing algorithms or by well-known computer vision algorithms, for example, for object detection and / or classification.

[0020] The first vehicle position can be interpreted as a primarily determined vehicle position of the motor vehicle, in the sense that the first vehicle position is generally reliable and therefore corresponds to the consolidated vehicle position. If the error of the first vehicle position is relatively large, which can occur, for example, due to a drift in the position of the tracking instance, this can be identified by comparing it with the second vehicle position. Since the first vehicle position and the second vehicle position are determined based on independent data sources, namely the point cloud and the image, respectively, the reliability of the consolidated vehicle position is increased.

[0021] The position deviation is used as an indicator of the reliability of the first vehicle position. If the position deviation equals a predefined threshold, the consolidated vehicle position can be determined as the first vehicle position. In alternative embodiments, the consolidated vehicle position can be determined as the second vehicle position if the position deviation equals the predefined threshold.

[0022] Since the image of the surroundings is generated by exactly the same detector array used to generate the point cloud, no additional sensor system or sensor system type, such as a camera, is required to implement the inventive method. This reduces the complexity of the system and data analysis, and consequently also the risk of system failure.

[0023] The second vehicle position can be determined based on the image using methods known for cameras, such as visual odometry, SLAM, digital maps, and so on.

[0024] For applications or application situations that may arise in a method according to the invention and which are not explicitly described here, it may be provided that, according to the method, an error message and / or a request for user feedback is output and / or a standard setting and / or a predetermined initial state is set.

[0025] According to some embodiments, a previous vehicle position of the motor vehicle is provided for a previous frame interval preceding the frame interval, and the first vehicle position is determined depending on the previous vehicle position and the point cloud.

[0026] In particular, the frame interval immediately follows the previous frame interval. The previous vehicle position can be determined, for example, as a corresponding consolidated vehicle position of the previous frame interval, as described above for the frame interval.

[0027] For example, it can be determined how the point cloud or a specific part of the point cloud, such as a landmark, has changed, in particular moved, compared to the previous frame interval. Based on this change, the first vehicle position can be calculated depending on the previous vehicle position. This can also be iterated over more than two frame intervals. Consequently, the invention is particularly advantageous in such cases, as it can contribute to avoiding or limiting a drift in the vehicle position over multiple iterations.

[0028] According to some embodiments, a landmark is detected based on the point cloud, and a landmark position of the landmark relative to the motor vehicle is determined based on the point cloud. The first vehicle position is determined based on the landmark position and the previous vehicle position.

[0029] To increase the accuracy of the first vehicle position, such designs can also be extended by using more than one landmark accordingly.

[0030] For example, a previous landmark position of the landmark relative to the motor vehicle can be determined for the previous frame interval, for example, based on a corresponding previous point cloud generated by the detector array. A shift of the landmark position relative to the previous landmark position can be used to determine the first vehicle position based on the previous vehicle state by an analog shift.

[0031] Consequently, drift of the first vehicle position during the repeated updating of the first vehicle position can be avoided or limited by comparing the position deviation with the threshold value as described. In particular, it is not necessary to know a global position of the landmark in such embodiments, for example, from a digital map.

[0032] It is noted that in some embodiments, the landmark may be detected based on the point cloud and one or more other point clouds generated by the detector array during earlier or preceding frame intervals.

[0033] According to some embodiments, a landmark is detected based on the point cloud, and a landmark position of the landmark relative to the motor vehicle is determined based on the point cloud. The first vehicle position is determined based on the landmark position and digital map data containing a nominal global position of the landmark.

[0034] To increase the accuracy of the first vehicle position, such designs can also be extended by using more than one landmark accordingly.

[0035] In particular, the first vehicle position is then given by the nominal global position of the landmark and the landmark position of the landmark relative to the vehicle is determined depending on the point cloud.

[0036] According to some embodiments, one or more static objects in the environment are detected depending on the point cloud. Each static object of the one or more static objects is classified with respect to at least two predefined static object classes. A static object of the one or more static objects that belongs to a predefined subset of the at least two static object classes according to the classification is selected as a landmark. A landmark position of the landmark relative to the motor vehicle is determined depending on the point cloud. The first vehicle position is determined depending on the landmark position and the previous vehicle position.

[0037] A static object can be understood as an object whose position remains constant in the reference coordinate system. To identify an object as a static object, its position can be tracked over two or more frame intervals and / or an object classification algorithm can be used to assign the object a respective object class. It can then be determined whether the object is static or dynamic based on the object class. Non-limiting examples of static objects include buildings, guardrails, traffic signs, and so on. Dynamic objects can be, for example, other vehicles, pedestrians, etc.

[0038] To increase the accuracy of the first vehicle position, such designs can also be extended by using them as a landmark.

[0039] For example, a previous landmark position of the landmark relative to the motor vehicle can be determined for the previous frame interval, for example, based on a corresponding previous point cloud generated by the detector array. A shift of the landmark position relative to the previous landmark position can be used to determine the first vehicle position based on the previous vehicle state by an analog shift.

[0040] Consequently, drift of the first vehicle position during the repeated updating of the first vehicle position can be avoided or limited by comparing the position deviation with the threshold value as described. In particular, it is not necessary to know a global position of the landmark in such embodiments, for example, from a digital map.

[0041] In particular, the subset contains fewer static object classes than the two or more static object classes. Consequently, in such embodiments, not just any static landmark is used to determine the first vehicle position, but only certain predefined types of static landmarks that are given by the subset of static object classes. It has been shown that certain types of static landmarks are better suited for the purpose of self-localization than others or can be used for self-localization with less additional effort than others. In particular, well-located static landmarks, such as traffic signs or posts at the edge of a roadway, may be better suited for the purpose of self-localization than static landmarks that extend, for example, along the roadway, such as a lane marker or a guardrail.In particular, objects of the latter type have a more or less homogeneous appearance along their extension, making their reliable tracking more complicated, especially when vehicle motion is not explicitly accounted for by non-visual odometry or similar methods. By selecting the landmark from the subset of static object classes, the reliability and / or accuracy of the initial vehicle position is increased.

[0042] According to some implementations, the subset of the at least two static object classes includes a class corresponding to posts, and / or a class corresponding to light reflectors, and / or a class corresponding to buildings, and / or a class corresponding to traffic signs, and / or a class corresponding to floor markings, and / or a class corresponding to Botts dots.

[0043] These types of static objects are particularly suitable for the purpose of self-localization based on a point cloud. Light reflectors can be attached to other objects, such as guardrails, cement blocks, walls, posts, and so on, with the intention of reflecting light, especially from vehicle headlights.

[0044] According to some embodiments, the at least two static object classes include a further class corresponding to devices for structurally separating two adjacent lanes from each other and / or guardrails, wherein the subset of the at least two static object classes does not include the further class.

[0045] These types of static objects are less suitable for the purpose of self-localization based on a point cloud.

[0046] According to some embodiments, the landmark is detected based on the image, and a further landmark position of the landmark relative to the motor vehicle is determined based on the image. Alternatively, a further landmark in the environment is detected based on the image, and the further landmark position is determined as a further landmark position of the further landmark relative to the motor vehicle based on the image. The second vehicle position is determined based on the further landmark position and a digital map.

[0047] The digital map includes, in particular, the nominal global position of the landmark or a nominal global position of the additional landmark. The second vehicle position can then be calculated, for example, based on the nominal global position of the landmark or the additional landmark in combination with the additional landmark position relative to the motor vehicle.

[0048] In this way, the second vehicle position allows a reliable cross-check of the first vehicle position.

[0049] According to some implementations, the first vehicle position is determined using a SLAM algorithm and / or the second vehicle position is determined using another SLAM algorithm.

[0050] According to some embodiments, the plurality of detector pixels of the detector array are arranged according to a plurality of pixel groups that are exposed to detect the ambient light one after the other during respective successive subintervals of the frame interval to generate the image.

[0051] For example, the plurality of detector pixels is arranged according to the plurality of columns and rows of the detector array, and each of the pixel groups corresponds to one of the plurality of columns or at least two adjacent columns of the plurality of columns. In particular, the columns correspond to a scanning direction of the laser scanner when the active optical sensor system is implemented as a laser scanner.

[0052] The controlled exposure of the pixel groups one after the other can be achieved, for example, by an appropriate shutter mechanism, for example a mechanical shutter, but preferably an electronic shutter.

[0053] According to some embodiments, during each of the subintervals, a respective part of the point cloud is generated by emitting a respective portion of the light pulses into the environment and detecting the reflected portions of the emitted portion of the light pulses by the respective pixel group.

[0054] In other words, both the image and the point cloud are generated step by step, one pixel group at a time. During a given sub-interval, the same pixel group is used to generate a corresponding portion of the point cloud and a corresponding portion of the image based on the ambient light. This ensures that the image matches the points of the point cloud with high accuracy.

[0055] According to some embodiments, the portion of the emitted light pulses is emitted into the environment after the respective group of pixels has been illuminated in order to detect the ambient light.

[0056] In particular, the corresponding data for generating the part of the image and the part of the point cloud are not acquired simultaneously using the respective pixel group, but sequentially. Consequently, the image and the point cloud are generated based on independent data, so that the image is not influenced by the reflected portions of the emitted laser pulses.

[0057] In alternative embodiments, the respective group of pixels is illuminated to detect the ambient light after the portion of the emitted light pulses is emitted into the environment and after the reflected portions of the portion of the emitted light pulses are detected by the respective pixel group.

[0058] According to a further aspect of the invention, a method for at least partially automatically driving a motor vehicle is provided. A method according to the invention for self-localizing a motor vehicle is implemented. At least one control signal for at least partially automatically driving the motor vehicle is generated depending on the consolidated vehicle position, in particular by the at least one computing unit.

[0059] The at least one control signal can be provided, for example, to one or more actuators of the motor vehicle, including, for example, one or more brake actuators and / or one or more steering actuators and / or one or more drive motors of the motor vehicle. The one or more actuators can influence a longitudinal and / or lateral control of the motor vehicle in order to guide the motor vehicle at least partially automatically.

[0060] The motor vehicle may, for example, include an electronic vehicle guidance system for implementing the method for at least partially automatically driving a motor vehicle. The electronic vehicle guidance system may, for example, include the active optical sensor system and / or the at least one computing unit.

[0061] An electronic vehicle guidance system can be understood as an electronic system designed to guide a vehicle fully automatically or autonomously, and in particular, without requiring manual intervention or control by a driver or user of the vehicle. The vehicle automatically performs all required functions, such as steering maneuvers, deceleration maneuvers, and / or acceleration maneuvers, as well as monitoring and recording road traffic and corresponding reactions. In particular, the electronic vehicle guidance system can implement a fully automatic or fully autonomous driving mode according to Level 5 of the SAE J3016 classification. An electronic vehicle guidance system can also be implemented as a driver assistance system (ADAS), which assists a driver in partially automatic or partially autonomous driving.In particular, the electronic vehicle guidance system can implement a semi-automatic or semi-autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification. Here and below, SAE J3016 refers to the corresponding standard dated April 2021.

[0062] The at least partially automatic driving of the vehicle can therefore include driving the vehicle according to a fully automatic or fully autonomous driving mode according to Level 5 of the SAE J3016 classification. The at least partially automatic driving of the vehicle can also include a partially automatic or partially autonomous driving mode according to Levels 1 to 4 of the SAE J3016 classification.

[0063] According to some embodiments, an automation level for driving the motor vehicle is changed from a predefined first automation level to a predefined second automation level depending on the position deviation.

[0064] The automation levels can, for example, correspond to the levels specified in the SAE J3016 classification or another classification. In particular, the second automation level corresponds to a lower level of automation than the first level of automation. The second automation level can also correspond to manual driving.

[0065] Consequently, the criticality of the position deviation can be assessed and, if it is considered too high, the automation level is reduced, thus increasing safety.

[0066] According to some embodiments, the automation level is changed from the first automation level to the second automation level when the position deviation is greater than a predefined further threshold, in particular precisely when the position deviation is greater than the further threshold.

[0067] According to some embodiments, the further threshold is greater than the threshold.

[0068] In other words, an escalation of measures taken depending on the position deviation is carried out gradually. If the position deviation is smaller than the threshold, the first vehicle position is maintained as the consolidated vehicle position. If the position deviation is greater than the threshold but still smaller than the further threshold, the consolidated vehicle position is set or reset to the second vehicle position, but the automation level remains unchanged. If the position deviation is greater than the further threshold, the consolidated vehicle position is set or reset to the second vehicle position, and additionally the automation level is lowered. Consequently, the safety and reliability of self-localization are increased while the availability of the first automation level is kept high, if possible.

[0069] According to some embodiments, a warning is generated for a driver of the motor vehicle depending on the position deviation, in particular if the position deviation is greater than the further threshold value.

[0070] The warning can be generated alternatively or in addition to changing the automation level. This can further increase safety.

[0071] According to a further aspect of the invention, an active optical sensor system for a motor vehicle is provided. The active optical sensor system includes a transmitter unit, at least one computing unit, and a detector arrangement. The at least one computing unit is configured to control the transmitter unit to emit light pulses into an environment of the active optical sensor system. The detector arrangement is configured to detect reflected portions of the emitted light pulses and to generate detector signals depending on the detected reflected portions. The detector arrangement is configured to generate further detector signals depending on the ambient light incident on the detector arrangement. The at least one computing unit is configured to generate a point cloud depending on the detector signals and to generate an image of the environment depending on the further detector signals.The at least one computing unit is configured to determine a first vehicle position of the motor vehicle based on the point cloud and a second vehicle position of the motor vehicle based on the image. The at least one computing unit is configured to determine a positional deviation of the first vehicle position from the second vehicle position. The at least one computing unit is configured to determine a consolidated vehicle position of the motor vehicle as the first vehicle position if the positional deviation is smaller than a predefined threshold value, and to determine the consolidated vehicle position as the second vehicle position if the positional deviation is greater than the threshold value.

[0072] The at least one computing unit can be distributed across various positions in or on the motor vehicle. For example, the at least one computing unit or a part thereof can be integrated into a common housing of the active optical sensor system, which also includes the transmitter unit and the detector arrangement, and can, for example, also function as the aforementioned control and / or evaluation unit of the active optical sensor system. Alternatively or additionally, the at least one computing unit or a further part thereof can be implemented as one or more ECUs of the motor vehicle, etc.

[0073] In the present disclosure, a computing unit can be understood, for example, as a data processing device with processing circuits. A computing unit can thus perform computing operations for processing data. The computing operations can also include indexed accesses to a data structure, for example, a look-up table (LUT).

[0074] In particular, a computing unit can contain one or more computers, one or more microcontrollers and / or one or more integrated circuits, for example one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), and / or one or more single-chip systems (SoCs). The computing unit can also contain one or more processors, for example one or more microprocessors, one or more central processing units (CPUs), one or more graphics processing units (GPUs), and / or one or more signal processors, in particular one or more digital signal processors (DSPs). The computing unit can also include a physical or virtual network of computers or other of the aforementioned units.

[0075] A computing unit may also include one or more hardware and / or software interfaces and / or one or more memory units. A memory unit may be a volatile data memory, for example, a dynamic random access memory (DRAM) or a static random access memory (SRAM), or a non-volatile data memory, for example, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), a flash memory or flash EEPROM, or a ferroelectric random access memory (FRAM).be implemented as magnetoresistive random access memory (MRAM) or phase-change random access memory (PCRAM).

[0076] According to some embodiments, the detector array includes a plurality of detector pixels arranged according to a plurality of rows and a plurality of columns, wherein a total number of the plurality of rows is at least 100 and / or a total number of the plurality of columns is at least 100.

[0077] Further embodiments of the active optical sensor system according to the invention result directly from the various embodiments of the method according to the invention, and vice versa. In particular, individual features and corresponding explanations as well as advantages relating to the various embodiments of the method according to the invention can be transferred accordingly to the corresponding embodiments of the active optical sensor system according to the invention. In particular, the active optical sensor system according to the invention is designed or programmed to carry out a method according to the invention. In particular, the active optical sensor system according to the invention carries out a method according to the invention.

[0078] According to a further aspect of the invention, an electronic vehicle guidance system for a motor vehicle is provided. The electronic vehicle guidance system includes an active optical sensor system according to the invention. The at least one computing unit is configured to generate at least one control signal for at least partially automatically guiding the motor vehicle depending on the consolidated vehicle position.

[0079] Further embodiments of the electronic vehicle guidance system according to the invention result directly from the various embodiments of the methods according to the invention, and vice versa. In particular, individual features and corresponding explanations as well as advantages relating to the various embodiments of the methods according to the invention can be transferred accordingly to corresponding embodiments of the electronic vehicle guidance system according to the invention.

[0080] According to a further aspect of the invention, a computer program containing instructions is provided. When the instructions are executed by an active optical sensor system according to the invention, in particular by the at least one computing unit, the instructions cause the active optical sensor system to perform a method according to the invention for self-localizing a motor vehicle.

[0081] The instructions can be provided, for example, as program code. The program code can be provided, for example, as binary code or assembly code and / or as source code of a programming language, for example, C, and / or as a program script, for example, Python.

[0082] According to a further aspect of the invention, a further computer program containing further instructions is provided. When the further instructions are executed by an electronic vehicle guidance system according to the invention, in particular by the at least one computing unit, the instructions cause the electronic vehicle guidance system to carry out a method according to the invention for at least partially automatically guiding a motor vehicle.

[0083] The additional instructions can be provided, for example, as program code. The program code can be provided, for example, as binary code or assembly code and / or as source code of a programming language, for example, C, and / or as a program script, for example, Python.

[0084] According to a further aspect of the invention, a computer-readable storage medium, in particular a non-volatile computer-readable storage medium, which stores a computer program according to the invention is provided.

[0085] The computer program, the further computer program and the computer-readable storage medium are respective computer program products with the instructions.

[0086] Further features of the invention emerge from the claims, the figures and the description of the figures. The features and combinations of features mentioned above in the description as well as the features and combinations of features mentioned below in the description of the figures and / or shown in the figures can be encompassed by the invention not only in the respectively specified combination, but also in other combinations. In particular, embodiments and combinations of features can also be encompassed by the invention that do not have all of the features of an originally formulated claim. Furthermore, embodiments and combinations of features can be encompassed by the invention that go beyond or deviate from the combinations of features set out in the backreferences of the claims.

[0087] The invention is explained in detail below using specific exemplary embodiments and corresponding schematic drawings. In the drawings, identical or functionally equivalent elements may be designated by the same reference numerals. The description of identical or functionally equivalent elements is not necessarily repeated with reference to different figures.

[0088] The figures show: Fig. 1 schematically shows a motor vehicle with an exemplary embodiment of an electronic vehicle guidance system according to the invention; Fig. 2 is a schematic partial block diagram of an exemplary embodiment of a method according to the invention for self-localization of a motor vehicle; Fig. 3 schematically shows examples of landmarks for use in a further exemplary embodiment of a method according to the invention for self-localization of a motor vehicle; Fig. 4 schematic illustrations of steps of a further exemplary embodiment of a method according to the invention for self-localization of a motor vehicle; and Fig. 5 schematic illustrations of steps of a further exemplary embodiment of a method according to the invention for self-localization of a motor vehicle.

[0089] Fig. 1 schematically shows a motor vehicle 1 with an exemplary embodiment of an electronic vehicle guidance system 2 according to the invention. The electronic vehicle guidance system 2 includes an exemplary embodiment of an active optical sensor system 3 according to the invention, which is implemented, for example, as a lidar system, and at least one computing unit 3c, 4, which can include, for example, an ECU 4 and / or a control and / or evaluation unit 3c of the active optical sensor system 3.

[0090] The electronic vehicle guidance system 2 can implement a method according to the invention for at least partially automatically guiding a motor vehicle. For this purpose, the active optical sensor system 3 can implement a method according to the invention for self-localization of the motor vehicle 1.

[0091] The active optical sensor system 3 can generate a point cloud representing the surroundings of the motor vehicle 1 by emitting light pulses into the surroundings and detecting portions of the emitted light pulses that are reflected by objects 21 in the surroundings of the motor vehicle 1, using a detector array 3b with a plurality of detector pixels. The active optical sensor system 3 includes, in addition to the detector array 3b, a transmitter unit 3a and the control and / or evaluation unit 3c, which is configured to control the transmitter unit 3a to emit the light pulses into the surroundings of the motor vehicle 1. The detector array 3b is configured to generate detector signals depending on reflected portions of the emitted light pulses impinging on the detector array 3b.Furthermore, the detector arrangement 3b is configured to generate further detector signals depending on the ambient light incident on the detector arrangement 3b.

[0092] The control and / or evaluation unit 3c is configured to generate a point cloud depending on the detector signals and to generate an image, in particular a monochromatic image, of the environment depending on the further detector signals.

[0093] The control and / or evaluation unit 3c is configured to determine a first vehicle position of the motor vehicle 1 based on the point cloud and a second vehicle position of the motor vehicle 1 based on the image. The control and / or evaluation unit 3c is configured to determine a positional deviation of the first vehicle position from the second vehicle position. The control and / or evaluation unit 3c is configured to determine a consolidated vehicle position 7b of the motor vehicle 1 as the first vehicle position if the positional deviation is smaller than a predetermined threshold value, and to determine the consolidated vehicle position 7b as the second vehicle position if the positional deviation is greater than the threshold value.

[0094] For example, the ECU 4 can generate at least one control signal for at least partially automatically guiding the motor vehicle 1 depending on the consolidated vehicle position. The at least one control signal can be provided, for example, to one or more actuators of the motor vehicle 1, including, for example, one or more brake actuators and / or one or more steering actuators and / or one or more drive motors of the motor vehicle 1. The one or more actuators can influence a longitudinal and / or lateral control of the motor vehicle 1 in order to guide the motor vehicle 1 at least partially automatically.

[0095] It is noted that the distribution of tasks between the ECU 4 and the control and / or evaluation unit 3c is merely exemplary and may vary in other embodiments. In general, the steps performed by the ECU 4 and the control and / or evaluation unit 3c are executed by at least one processing unit of the electronic vehicle guidance system 2.

[0096] Fig. 2 shows a schematic partial block diagram of an exemplary embodiment of a method according to the invention for self-localization of a motor vehicle 1. Further steps of the method are shown in Fig. 4 and Fig. 5 illustrates.

[0097] For example, the point cloud and image can be described as described for each frame interval of a plurality of consecutive frame intervals T1, T2, T3. For each frame interval, the first vehicle position, the second vehicle position, and the consolidated vehicle position can be determined as described above. For example, a SLAM algorithm or a tracking algorithm can be used to determine and track the first vehicle position over the consecutive frame intervals. Whenever the position deviation is found to be greater than the threshold, the first vehicle position is set to the second vehicle position and can then be tracked further in subsequent frames.

[0098] To determine and track the first vehicle position, static landmarks 17 can be detected based on the point cloud, and the first vehicle position can be iteratively updated depending on the detected relative movement of the landmarks 17. To execute the steps of the method according to the invention, a dynamic position estimation module 5 can take landmark input data 6a indicating the current positions of the landmarks 17 and output the consolidated vehicle position 7b. Optionally, the dynamic position estimation module 5 can also output a status 7a of the method, for example, the current position deviation.In addition to the landmark input data 6a, the dynamic position estimation module 5, in some embodiments, may also receive vehicle telemetry input data 6b, GPS input data 6c, and / or lane input data 6d, which indicate a position of one or more lanes on the roadway relative to the motor vehicle 1, to calculate the consolidated vehicle position 7b. The dynamic position estimation module 5, in some embodiments, may also receive odometry input data, for example, wheel rotation data, gyroscopic measurements, and so on, to calculate the consolidated vehicle position 7b.

[0099] Fig. Figure 3 schematically shows various examples of static objects that are particularly suitable for use as landmarks 17. These include, for example, posts 8 at the edge of the roadway, light reflectors 9 of posts 8, light reflectors 11 of walls 10 or other barriers for structurally separating lanes from one another for structurally delimiting the roadway, light reflectors 13 of guardrails 12, mounting posts 14 of guardrails 12, light reflectors 16 of cement blocks 15, traffic signs, Botts dots, floor markings, and so on.

[0100] As in Fig.As indicated in Figure 4, some objects, including static objects suitable for use as landmarks 17 and, for example, dynamic objects 18, such as other vehicles, are detected based on the point cloud and represented, for example, by corresponding tracking instances 19, 20. Since the relative position of the landmarks 17 with respect to the motor vehicle 1 changes over the frame intervals T1, T2, T3, the first vehicle position can be updated accordingly.

[0101] As described in particular with reference to the figures, the invention increases the reliability of the self-localization of a motor vehicle.

[0102] During autonomous or semi-autonomous driving of a motor vehicle, it is desirable to dynamically verify the vehicle's position. In some implementations, this is done based on available landmarks in the environment detected by the active optical sensor system. Some implementations support autonomous driving near construction sites. Some implementations support a road curvature estimation algorithm and / or a lane departure warning system.

[0103] In some embodiments, the motor vehicle identifies the landmarks and dynamically checks their positions while driving in at least semi-autonomous mode. The landmarks can be classified using traditional and / or AI-based classification algorithms, for example, as post reflectors, road surface markings, guardrail posts, cement block reflectors, etc. Based on the landmarks, the motor vehicle can continuously self-check its position and, in some embodiments, assess the criticality of the identified position deviation in relation to other data.

Claims

[1] Method for self-localization of a motor vehicle (1), wherein for a frame interval - a point cloud is generated by emitting light pulses into the surroundings of the motor vehicle (1) by a transmitter unit (3a) of an active optical sensor system (3) which is mounted on the motor vehicle (1), and by detecting reflected portions of the emitted light pulses by a detector arrangement (3b) of the active optical sensor system (3); - an image of the surroundings is generated by detecting ambient light incident on the detector arrangement (3b); - a first vehicle position of the motor vehicle (1) is determined depending on the point cloud and a second vehicle position of the motor vehicle (1) is determined depending on the image; - a position deviation of the first vehicle position from the second vehicle position is determined; and - a consolidated vehicle position (7b) of the motor vehicle (1) is determined as the first vehicle position if the position deviation is smaller than a predefined threshold value, and the consolidated vehicle position (7b) is determined as the second vehicle position if the position deviation is greater than the threshold value. [2] Method according to claim 1, wherein a previous vehicle position of the motor vehicle (1) is provided for a previous frame interval preceding the frame interval and the first vehicle position is determined depending on the previous vehicle position and the point cloud. [3] Method according to claim 2, wherein - a landmark (17) is detected depending on the point cloud and a landmark position of the landmark (17) relative to the motor vehicle (1) is determined depending on the point cloud; and - the first vehicle position is determined depending on the landmark position and the previous vehicle position. [4] Method according to claim 2, wherein - one or more static objects in the environment are detected depending on the point cloud; - each of the one or more static objects is classified according to at least two predefined static object classes; - one of the one or more static objects which, according to the classification, belongs to a predefined subset of the at least two static object classes is selected as a landmark (17); - a landmark position of the landmark (17) relative to the motor vehicle (1) is determined depending on the point cloud; and - the first vehicle position is determined depending on the landmark position and the previous vehicle position. [5] Method according to claim 4, wherein the subset of the at least two static object classes - a class corresponding to posts; and / or - a class corresponding to light reflectors; and / or - a class corresponding to buildings; and / or - a class corresponding to traffic signs; and / or - a class corresponding to floor markings; and / or - contains a class corresponding to Botts-Dots. [6] Method according to one of claims 4 or 5, wherein the at least two static object classes include a further class corresponding to devices for the structural separation of two adjacent lanes from one another and / or guard rails, wherein the subset of the at least two static object classes does not include the further class. [7] Method according to one of claims 3 to 6, wherein - the landmark (17) or a further landmark (17) is detected depending on the image and a further landmark position of the landmark (17) or the further landmark (17) relative to the motor vehicle (1) is determined depending on the image; and - the second vehicle position is determined depending on the further landmark position and a digital map. [8] A method according to any one of the preceding claims, wherein the first vehicle position is determined using a simultaneous localization and mapping algorithm. [9] Method for at least partially automatically driving a motor vehicle (1), wherein - a method for self-localization of a motor vehicle (1) according to one of the preceding claims is carried out; and - at least one control signal for at least partially automatically guiding the motor vehicle (1) is generated depending on the consolidated vehicle position (7b). [10] Method according to claim 9, wherein an automation level for driving the motor vehicle (1) is changed from a predefined first automation level to a predefined second automation level depending on the position deviation. [11] The method of claim 10, wherein the automation level is changed from the first automation level to the second automation level when the position deviation is greater than a predefined further threshold value which is greater than the threshold value. [12] Method according to one of claims 9 to 11, wherein a warning for a driver of the motor vehicle (1) is generated depending on the position deviation. [13] Active optical sensor system (3) for a motor vehicle (1), comprising a transmitter unit (3a), at least one computing unit (3c, 4) which is designed to control the transmitter unit (3a) in order to emit light pulses into an environment of the active optical sensor system (3), and a detector arrangement (3b) which is designed to detect reflected portions of the emitted light pulses and to generate detector signals depending on the detected reflected portions and to generate further detector signals depending on ambient light incident on the detector arrangement (3b), wherein the at least one computing unit (3c, 4) is designed to - to generate a point cloud depending on the detector signals and to generate an image of the environment depending on the other detector signals; - to determine a first vehicle position of the motor vehicle (1) depending on the point cloud and a second vehicle position of the motor vehicle (1) depending on the image; - to determine a position deviation of the first vehicle position from the second vehicle position; and - to determine a consolidated vehicle position (7b) of the motor vehicle (1) as the first vehicle position if the position deviation is smaller than a predefined threshold value, and to determine the consolidated vehicle position (7b) as the second vehicle position if the position deviation is greater than the threshold value. [14] Active optical sensor system (3) according to claim 13, wherein the detector array (3b) comprises a plurality of detector pixels arranged according to a plurality of rows and a plurality of columns, wherein a total number of the plurality of rows is at least 100 and / or a total number of columns is at least 100. [15] Electronic vehicle guidance system (2) for a motor vehicle (1), which includes an active optical sensor system (3) according to one of claims 13 or 14, wherein the at least one computing unit (3c, 4) is configured to generate at least one control signal for at least partially automatically guiding the motor vehicle (1) depending on the consolidated vehicle position (7b). [16] Computer program product which - includes instructions which, when executed by an active optical sensor system (3) according to one of claims 13 or 14, cause the active optical sensor system (3) to carry out a method according to one of claims 1 to 8; and / or - further commands which, when executed by an electronic vehicle guidance system (2) according to claim 15, cause the electronic vehicle guidance system (2) to carry out a method according to one of claims 9 to 12.

Images