Self-localization of a motor vehicle

Abstract

For self-localization of a motor vehicle (1), a point cloud is generated by emitting light pulses and detecting reflected portions of the emitted light pulses by a detector array (3b) of the active optical sensor system (3). An image of the environment is generated by detecting ambient light impinging on the detector array (3b). A first vehicle position of the motor vehicle (1) is determined depending on the point cloud and a second vehicle position of the motor vehicle (1) is determined depending on the image. A position deviation of the first vehicle position from the second vehicle position is determined. A consolidated vehicle position (7b) is determined as the first vehicle position, if the position deviation is less than a predefined threshold value, and the consolidated vehicle position (7b) is determined as the second vehicle position, if the position deviation is greater than the threshold value.

Description

[0001] Self-localization of a motor vehicle

[0002] The present invention is directed to a method for self-localization of a motor vehicle, wherein for a frame interval a point cloud is generated by emitting light pulses into an environment of the motor vehicle by an emitter unit of an active optical sensor system, which is mounted to the motor vehicle, and detecting reflected portions of the emitted light pulses by a detector array of the active optical sensor system. The invention is also directed to a method for guiding a motor vehicle at least in part automatically, wherein such a method for self-localization is carried out. The invention is also directed to a corresponding active optical sensor system for a motor vehicle, to an electronic vehicle guidance system comprising such an active optical sensor system, and to corresponding computer program products.

[0003] Self-localization methods are used by autonomous vehicle or in part automatically guided vehicles to track their own vehicle position in a reference coordinate system, for example a coordinate system of a digital map. The tracked vehicle position may then be used for trajectory planning, collision avoidance or various other applications for guiding the motor vehicle at least in part automatically. For example, tracking algorithms based on Kalman filters or the like can be used for tracking the vehicle position based on environmental sensor data generated by one or more environmental sensor systems of the motor vehicle. Also SLAM (simultaneous localization and mapping) methods may be used for this purpose.

[0004] Being highly safety relevant, it is generally desirable to validate or cross-check the tracked vehicle position in order to increase its reliability. One possible way to do so drawbacks is to use multi-fusion approaches using multiple sensors, for example cameras, radar systems and / or lidar systems. This, however, increases the complexity of the system and the data evaluation, for example in view of temporal synchronization or calibration, and ambiguities between the data from different sensors may cause problems. Due to the increased complexity, also the risk for a system failure is increased. Furthermore, it may be required, in particular for safety relevant applications, that the reliability of certain tracking and object detection functions is sufficiently high when using only a single sensor system or a single type of sensor system. It is an objective of the present invention to increase the reliability of self-localization of a motor vehicle, in particular without using multiple environmental sensor systems.

[0005] This objective is achieved by the respective subject matter of the independent claims. Further implementations and preferred embodiments are subject matter of the dependent claims.

[0006] The invention is based on the idea to use the same detector array of an active optical sensor system one the one hand for generating a point cloud based on the detection of reflected portions of emitted light pulses and, on the other hand, for generating an image based on detected ambient light. Respective vehicle positions are determined based on the point cloud and based on the image. Then, a consolidated vehicle position is determined depending on a deviation between the two vehicle positions.

[0007] According to an aspect of the invention, a method for self-localization of a motor vehicle is provided. There, for a frame interval of an active optical sensor system, which is mounted to the motor vehicle, a point cloud is generated by emitting light pulses into an environment of the motor vehicle, in particular during the frame interval, by an emitter unit of the active optical sensor system and detecting reflected portions of the emitted light pulses by a detector array of the active optical sensor system, in particular during the frame interval. An image of the environment is generated by detecting ambient light impinging on the detector array, in particular during the frame interval. A first vehicle position of the motor vehicle is determined depending on the point cloud and a second vehicle position of the motor vehicle is determined depending on the image. A position deviation of the first vehicle position from the second vehicle position is determined. A consolidated vehicle position of the motor vehicle is determined. If the position deviation is less than a predefined threshold value, the consolidated vehicle position is given by the first vehicle position. If the position deviation is greater than the threshold value, the consolidated vehicle position is given by the second vehicle position.

[0008] Unless stated otherwise, the steps of the method, which are not carried out by the emitter unit or the detector array, may for example be performed by a data processing apparatus, which comprises at least one computing unit, in particular a data processing apparatus of the motor vehicle. For this purpose, the at least one computing unit may for example store a computer program comprising instructions which, when executed by the at least one computing unit, cause the at least one computing unit to carry out the respective method steps. All computing units of the at least one computing unit may be comprised by the motor vehicle. However, it is also possible that all computing units of the at least one computing unit are part of an external computing system external to the motor vehicle, for example a backend server or a cloud computing system. It is also possible that the at least one computing unit comprises at least one vehicle computing unit of the motor vehicle as well as at least one external computing unit comprised by the external computing system. The at least one vehicle computing unit may for example be comprised by one or more electronic control units, ECUs, and / or one or more zone control units, ZCUs, and / or one or more domain control units, DCUs, of the motor vehicle and / or by the active optical sensor system.

[0009] The term self-localization of a motor vehicle refers, for example, the determination of a position of the vehicle, in the presented method to consolidated vehicle position, in a predefined reference coordinate system, for example a map coordinate system of a digital map also denoted as global coordinate system, by the motor vehicle itself. Consequently, self-localization does, in particular, not make use of images or the like depicting the motor vehicle in the environment. In particular, the consolidated vehicle position and, for example, the first vehicle position and the second vehicle position, are given in the reference coordinate system. Several approaches for self-localization are known, which can be used, for example, for determining the first vehicle position depending on the point cloud and / or for determining the second vehicle position depending on the image. These include SLAM approaches or approaches based on Kalman filters or other tracking algorithms and so forth.

[0010] Here and in the following, "light" may be understood such that it comprises electromagnetic waves in the visible range, in the infrared range and / or in the ultraviolet range. Accordingly, the expression "optical" may be understood to be related to light according to this meaning.

[0011] By definition, an active optical sensor system, in the present case the emitter unit, comprises a light source for emitting light or light pulses, respectively. For example, the light source may be implemented as a laser, in particular as an infrared laser. Furthermore, an active optical sensor system, in the present case the detector array, comprises by definition at least one optical detector to detect reflected parts of the emitted light. In particular, the active optical sensor system is configured to generate one or more sensor signals based on the detected fractions of the light and process and / or output the sensor signals. For example, lidar sensor systems are active optical sensor systems.

[0012] A laser scanner is a known design of lidar sensor systems, in which a laser beam is generated by one or more laser diodes of the lidar sensor system, in particular in form of laser pulses, and deflected by means of a light deflection arrangement so that different deflection angles of the laser beam may be realized. The light deflection arrangement may, for example, contain one or more rotatably mounted mirrors. Alternatively, the light deflection arrangement may include a mirror element with a tiltable and / or pivotable surface. The mirror element may, for example, be configured as a micro-electro- mechanical system, MEMS. In the environment, the emitted laser beams can be partially reflected, and the reflected portions may in turn hit the laser scanner, in particular the light deflection arrangement, which may direct them to the detector array of the laser scanner. In particular, each detector pixel of the detector array may generate a respective detector signal based on the detected light. Based on the spatial arrangement of the respective detector pixel together with the current position of the light deflection arrangement, in particular its rotational position or its tilting and / or pivoting position, it is thus possible determine the direction of incidence of the detected reflected components of light. The at least one control and / or evaluation unit of the laser scanner may, for example, perform a time-of-flight measurement or an indirect time-of-flight measurement to determine a radial distance of the reflecting object. Reflected portions can be understood as portions of the emitted light reflected back from objects in the environment, including the road surface. This does not necessarily mean specularly reflected light. Rather, the reflected parts may also include retroreflected and / or scattered light.

[0013] Other designs of lidar systems are flash lidar systems. These are non-scanning systems, which do not require said light deflection arrangement. Therein, the laser light generated by the light source is diffused by an optical element to irradiate over a wide angle in a single flash.

[0014] The detector array may comprise a plurality of detector pixels. A single detector pixel of the detector array does not necessarily consist of a single optical detector. Rather, it is also possible that a group of several adjacent optical detectors form one detector pixel. The latter is particularly possible when using single photon avalanche diodes, SPADs, as optical detectors. In other embodiments, however, it is also possible that a pixel consists of exactly one optical detector, for example a single photodiode or a single avalanche photodiode, APD.

[0015] Since the detector array comprises a plurality of detector pixels, which are for example arranged according to a plurality of columns, for example tens to hundreds of columns, and a plurality of rows, for example tens to hundreds of rows, apart from generating the point cloud, the detector array can also be used to capture two-dimensional images of the environment in the same way or a similar way as a camera, for example a thermal camera. That the image of the environment is generated by detecting ambient light impinging on the detector array during the frame interval can be understood in this way.

[0016] In other words, for generating the image, no light pulses have to be emitted into the environment and no distance has to be computed using, for example, a ToF measurement. In yet other words, the image is generated independent of the light pulses emitted during the frame interval or other respective frame intervals and independent of corresponding reflected portions. Therefore, the image corresponds, in particular, to a monochromatic image, for example a grayscale image.

[0017] In particular, the image comprises a plurality of image pixels, each image pixel corresponding to one of the detector pixels. For each image pixel, the image stores a pixel value depending on the amount of energy of the ambient light impinging on the respective detector pixel during a corresponding exposure period within the first frame interval. Compared to the images, also each point of the point cloud corresponds to one of the detector pixels but not each detector pixel may necessarily detect reflected portions of the emitted light pulses. In particular, for generating the point cloud, influences of ambient light may even be removed by means of known ambient light suppression algorithms. For each point of the point cloud, apart from the respective position of the detector pixel, further information may be stored, including for example the radial distance determined by means of the direct or indirect ToF measurement, an energy measure concerning the amount of optical energy of the respective reflected portions of the light pulses, such as an echo pulse width, EPW, an area under a signal pulse et cetera. Therefore, the image can be processed by conventional image processing algorithms or by known computer vision algorithms, for example for object detection and / or classification.

[0018] The first vehicle position may be interpreted as a primarily determined vehicle position of the motor vehicle in the sense that in a regular case, the first vehicle position is reliable and therefore corresponds to the consolidated vehicle position. If the error of the first vehicle position is relatively large, which may for example happen due to a drift in the position of the tracking instance, this may be identified by comparing it with the second vehicle position. Since the first vehicle position and the second vehicle position are determined based on independent data sources, namely the point cloud and the image, respectively, the reliability of the consolidated vehicle position is increased.

[0019] Therein, the position deviation is used as an indicator for the reliability of the first vehicle position. If the position deviation is equal to the predefined threshold value, the consolidated vehicle position may be determined as the first vehicle position. In alternative implementations, the consolidated vehicle position may be determined as the second vehicle position, if the position deviation is equal to the predefined threshold value.

[0020] Since the image of the environment is generated by the very same detector array, which is also used for generating the point cloud, no further sensor system or further type of sensor system, such as a camera, needs to be used in order to carry out the method according to the invention. This reduces the complexity of the system and the data evaluation and, consequently, also the risk for a system failure.

[0021] The second vehicle position may be determined based on the image by using methods known for cameras, for example, such as visual odometry, SLAM, digital maps and so forth.

[0022] For use cases or use situations which may arise in a method according to the invention and which are not explicitly described herein, it may be provided that, in accordance with the method, an error message and / or a prompt for user feedback is output and / or a default setting and / or a predetermined initial state is set.

[0023] According to several implementations, for a previous frame interval preceding the frame interval, a previous vehicle position of the motor vehicle is provided, and the first vehicle position is determined depending on the previous vehicle position and the point cloud.

[0024] In particular, the frame interval follows directly after the previous frame interval. The previous vehicle position may for example be determined as a corresponding consolidated vehicle position of the previous frame interval as described above for the frame interval. For example, it may be determined how the point cloud or a specific part of the point cloud, for example a landmark, has changed, in particular moved, compared to the previous frame interval. Based on this change, the first vehicle position can be computed depending on the previous vehicle position. This may also be iterated over more than two frame intervals. Consequently, invention is particularly beneficial in such cases, since it may contribute to avoiding or limit a drift of the vehicle position over several iterations.

[0025] According to several implementations, a landmark is detected depending on the point cloud and a landmark position of the landmark relative to the motor vehicle is determined depending on the point cloud. The first vehicle position is determined depending on the landmark position and the previous vehicle position.

[0026] In order to increase the accuracy of the first vehicle position, such implementations may also be extended to use more than one landmark analogously.

[0027] For example, a previous landmark position of the landmark relative to the motor vehicle may be determined for the previous frame interval, for example based on a corresponding previous point cloud generated by the detector array accordingly. A shift of the landmark position with respect to the previous landmark position may be used to determine the first vehicle position based on the previous vehicle condition by an analog shift.

[0028] Consequently, a drift of the first vehicle position during the repeated update of the first vehicle position may be avoided or limited by comparing the position deviation to the threshold value as described. In particular, it is not necessary to know a global position of the landmark in such implementations, for example from a digital map.

[0029] It is noted that, in some implementations, the landmark may be detected based on the point cloud and one or more further point clouds generated by the detector array during earlier or preceding frame intervals.

[0030] According to several implementations, a landmark is detected depending on the point cloud and a landmark position of the landmark relative to the motor vehicle is determined depending on the point cloud. The first vehicle position is determined depending on the landmark position and digital map data containing a nominal global position of the landmark. In order to increase the accuracy of the first vehicle position, such implementations may also be extended to use more than one landmark analogously.

[0031] In particular, the first vehicle position is then given by the nominal global position of the landmark and the landmark position of the landmark relative to the motor vehicle determined depending on the point cloud.

[0032] According to several implementations, one or more static objects in the environment are detected depending on the point cloud. Each static object of the one or more static objects is classified according to at least two predefined static object classes. One static object of the one or more static objects, which belongs to a predefined subset of the at least two static object classes according to the classification, is selected as a landmark. A landmark position of the landmark relative to the motor vehicle is determined depending on the point cloud. The first vehicle position is determined depending on the landmark position and the previous vehicle position.

[0033] A static object can be understood as an object, whose position in the reference coordinate system remains constant. In order to identify an object as a static object, its position may be tracked over two or more frame intervals and / or an object classification algorithm may be used to assign a respective object class to the object. It may then be decided based on the object class whether the object is static or dynamic. Non limiting examples of static objects include buildings, guard rails, traffic signs and so forth. Dynamic objects may for example be further vehicles, pedestrians et cetera.

[0034] In order to increase the accuracy of the first vehicle position, such implementations may also be extended to use more than one landmark analogously.

[0035] For example, a previous landmark position of the landmark relative to the motor vehicle may be determined for the previous frame interval, for example based on a corresponding previous point cloud generated by the detector array accordingly. A shift of the landmark position with respect to the previous landmark position may be used to determine the first vehicle position based on the previous vehicle condition by an analog shift. Consequently, a drift of the first vehicle position during the repeated update of the first vehicle position may be avoided or limited by comparing the position deviation to the threshold value as described. In particular, it is not necessary to know a global position of the landmark in such implementations, for example from a digital map.

[0036] In particular, the subset comprises less static object classes than the two or more static object classes. Consequently, in such implementations, not any static landmark is used for determining the first vehicle position, but only certain predefined types of static landmarks given by the subset of static object classes. It has turned out that certain types of static landmarks are more suitable for the purpose of self-localization than others or can be used for self-localization with less additional effort. In particular, well localized static landmarks, such as for example traffic signs or poles on a roadside, may be more suitable for the purpose of self-localization than static landmarks, which are for example extended along the road, such as a road delimiter or a guardrail. In particular, objects of the latter type have a more or less homogeneous appearance along their extension, which complicates their reliable tracking, in particular without taking explicitly into account the vehicle motion from non-visual odometry or the like. By selecting the landmark from the subset of static object classes, the reliability and / or accuracy of the first vehicle position in increased.

[0037] According to several implementations, the subset of the at least two static object classes comprises a class corresponding to poles and / or a class corresponding to light reflectors and / or a class corresponding to buildings and / or a class corresponding to traffic signs and / or a class corresponding to ground markings and / or a class corresponding to botts dots.

[0038] These types of static objects are particularly suitable for the purpose of self-localization based on a point cloud. Therein, light reflectors may for example be fastened to other objects, such as guardrails, cement blocks, walls, poles, and so forth with the intention to reflect light, in particular of vehicle headlights.

[0039] According to several implementations, the at least two static object classes comprise a further class corresponding to devices for structurally separating two adjacent lanes from each other and / or guardrails, wherein the subset of the at least two static object classes does not comprise the further class. These types of static objects are less suitable for the purpose of self-localization based on a point cloud.

[0040] According to several implementations, the landmark is detected depending on the image and a further landmark position of the landmark relative to the motor vehicle is determined depending on the image. Alternatively, a further landmark in the environment is detected depending on the image and the further landmark position is determined as a further landmark position of the further landmark relative to the motor vehicle depending on the image. The second vehicle position is determined depending on the further landmark position and a digital map.

[0041] The digital map comprises, in particular, the nominal global position of the landmark or a nominal global position of the further landmark. The second vehicle position can then for example be computed based on the nominal global position of the landmark or the further landmark in combination with the further landmark position relative to the motor vehicle.

[0042] In this way, the second vehicle position allows for a reliable cross check of the first vehicle position.

[0043] According to several implementations, the first vehicle position is determined by using a SLAM algorithm and / or the second vehicle position is determined by using a further SLAM algorithm.

[0044] According to several implementations, the plurality of detector pixels of the detector array is arranged according to a plurality of pixel groups, which are exposed to detect the ambient light one after another during respective consecutive sub-intervals of the frame interval, in order to generate the image.

[0045] For example, the plurality of detector pixels is arranged according to the plurality of columns and rows of the detector array and each of the pixel groups corresponds to one of the plurality of columns or to at least two adjacent columns of the plurality of columns. The columns correspond, in particular, to a scanning direction of the laser scanner, if the active optical sensor system is implemented as a laser scanner. The controlled exposure of the pixel groups one after another may for example be achieved by a corresponding shutter mechanism, for example a mechanical shutter but, preferably, an electronic shutter.

[0046] According to several implementations, during each of the sub-intervals, a respective part of the point cloud is generated by emitting a respective fraction of the light pulses into the environment and detecting the reflected portions of the emitted fraction of the light pulses by the respective pixel group.

[0047] In other words, both the image as well as the point cloud are generated step by step one pixel group after the other, wherein during a given sub-interval the same pixel group is used to generate a corresponding part of the point cloud and to generate a corresponding part of the image based on the ambient light. In this way, it is achieved that the image matches the points of the point cloud with high accuracy.

[0048] According to several implementations, the fraction of the emitted light pulses is emitted into the environment after the respective group of pixels has been exposed to detect the ambient light.

[0049] In particular, the corresponding data for generating the part of the image and the part of the point cloud are not acquired at the same time by means of the respective pixel group but after another. Consequently, the image and the point cloud are generated based on independent data such that the image is, in particular, not affected by the reflected portions of the emitted laser pulses.

[0050] In alternative implementations, the respective group of pixels is exposed to detect the ambient light after the fraction of the emitted light pulses is emitted into the environment and after the reflected portions of the fraction of the emitted light pulses are detected by the respective pixel group.

[0051] According to a further aspect of the invention, a method for guiding a motor vehicle at least in part automatically is provided. Therein, a method for self-localization of a motor vehicle according to the invention is carried out. At least one control signal for guiding the motor vehicle at least in part automatically is generated depending on the consolidated vehicle position, in particular by the at least one computing unit. The at least one control signal may for example be provided to one or more actuators of the motor vehicle, including for example one or more braking actuators and / or one or more steering actuators and / or one or more propulsion motors of the motor vehicle. The one or more actuators may affect a longitudinal and / or lateral control of the motor vehicle in order to guide the motor vehicle at least in part automatically.

[0052] The motor vehicle may for example comprise an electronic vehicle guidance system for carrying out the method for guiding a motor vehicle at least in part automatically. The electronic vehicle guidance system may for example comprise the active optical sensor system and / or the at least one computing unit.

[0053] An electronic vehicle guidance system may be understood as an electronic system, configured to guide a vehicle in a fully automated or a fully autonomous manner and, in particular, without a manual intervention or control by a driver or user of the vehicle being necessary. The vehicle carries out all required functions, such as steering maneuvers, deceleration maneuvers and / or acceleration maneuvers as well as monitoring and recording the road traffic and corresponding reactions automatically. In particular, the electronic vehicle guidance system may implement a fully automatic or fully autonomous driving mode according to level 5 of the SAE J3016 classification. An electronic vehicle guidance system may also be implemented as an advanced driver assistance system, ADAS, assisting a driver for partially automatic or partially autonomous driving. In particular, the electronic vehicle guidance system may implement a partly automatic or partly autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification. Here and in the following, SAE J3016 refers to the respective standard dated April 2021 .

[0054] Guiding the vehicle at least in part automatically may therefore comprise guiding the vehicle according to a fully automatic or fully autonomous driving mode according to level 5 of the SAE J3016 classification. Guiding the vehicle at least in part automatically may also comprise guiding the vehicle according to a partly automatic or partly autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification.

[0055] According to several implementations, a level of automatization for guiding the motor vehicle is changed from a predefined first level of automatization to a predefined second level of automatization depending on the position deviation. The levels of automatization may for example correspond to said levels of the SAE J3016 classification or another categorization. In particular, the second level of automatization corresponds to a lower level of automatization than the first level of automatization. The second level of automatization can also correspond to manual driving.

[0056] Consequently, the criticality of the position deviation may be evaluated and, if it is considered too high, the level of automatization is lowered, which increases safety.

[0057] According to several implementations, the level of automatization is changed from the first level of automatization to the second level of automatization, if the position deviation is greater than a predefined further threshold value, in particular if and only if the position deviation is greater than the further threshold value.

[0058] According to several implementations, the further threshold value is greater than the threshold value.

[0059] In other words, an escalation of measures taken depending on the position deviation is carried out in steps. If the position deviation is less than the threshold value, the first vehicle position is kept as the consolidated vehicle position. If the position deviation is greater than the threshold value but still less than the further threshold value, the consolidated vehicle position is set or reset to the second vehicle position, but the level of automatization remains unchanged, the position deviation is greater than the further threshold value, then the consolidated vehicle position is set or reset to the second vehicle position, and, in addition, the level of automatization is lowered. Consequently, safety and reliability of the self-localization is increased while the availability of the first level of automatization is kept high, if possible.

[0060] According to several implementations, a warning message for a driver of the motor vehicle is generated depending on the position deviation, in particular if the position deviation is greater than the further threshold value.

[0061] The warning message may be generated alternatively or in addition to the change of the level of automatization. Thus, the safety may be further increased.

[0062] According to a further aspect of the invention, an active optical sensor system for a motor vehicle is provided. The active optical sensor system comprises an emitter unit, at least one computing unit, and a detector array. The at least one computing unit is configured to control the emitter unit to emit light pulses into an environment of the active optical sensor system. The detector array is configured to detect reflected portions of the emitted light pulses and to generate detector signals depending on the detected reflected portions. The detector array is configured to generate further detector signals depending on ambient light impinging on the detector array. The at least one computing unit is configured to generate a point cloud depending on the detector signals and to generate an image of the environment depending on the further detector signals. The at least one computing unit is configured to determine a first vehicle position of the motor vehicle depending on the point cloud and a second vehicle position of the motor vehicle depending on the image. The at least one computing unit is configured to determine a position deviation of the first vehicle position from the second vehicle position. The at least one computing unit is configured to determine a consolidated vehicle position of the motor vehicle as the first vehicle position, if the position deviation is less than a predefined threshold value, and determine the consolidated vehicle position as the second vehicle position, if the position deviation is greater than the threshold value.

[0063] The at least one computing unit may be distributed over different locations in or on the motor vehicle. For example, the at least one computing unit or a part of it may be integrated into a common housing of the active optical sensor system also enclosing the emitter unit and the detector array and may, for example, also act as said control and / or evaluation unit of the active optical sensor system. Alternatively or in addition, the at least one computing unit or a further part of it may be implemented as one or more ECUs of the motor vehicle et cetera.

[0064] In the present disclosure, a computing unit may for example be understood as a data processing device with processing circuitry. A computing unit can therefore perform computing operations in order to process data. The computing operations may also include indexed accesses to a data structure, for example a look-up table, LUT.

[0065] In particular, a computing unit may include one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example, one or more application-specific integrated circuits, ASIC, one or more field-programmable gate arrays, FPGA, and / or one or more systems on a chip, SoC. The computing unit may also include one or more processors, for example one or more microprocessors, one or more central processing units, CPU, one or more graphics processing units, GPU, and / or one or more signal processors, in particular one or more digital signal processors, DSP. The computing unit may also include a physical or a virtual cluster of computers or other of said units. A computing unit may also comprise one or more hardware and / or software interfaces and / or one or more memory units. Therein, a memory unit may be implemented as a volatile data memory, for example a dynamic random access memory, DRAM, or a static random access memory, SRAM, or as a non-volatile data memory, for example a readonly memory, ROM, a programmable read-only memory, PROM, an erasable programmable read-only memory, EPROM, an electrically erasable programmable readonly memory, EEPROM, a flash memory or flash EEPROM, a ferroelectric random access memory, FRAM, a magnetoresistive random access memory, MRAM, or a phase-change random access memory, PCRAM.

[0066] According to several implementations, the detector array comprises a plurality of detector pixels, which are arranged according to a plurality of rows and a plurality of columns, wherein a total number of the plurality of rows is at least 100 and / or a total number of the plurality of columns is at least 100.

[0067] Further implementations of the active optical sensor system according to the invention follow directly from the various embodiments of the methods according to the invention and vice versa. In particular, individual features and corresponding explanations as well as advantages relating to the various implementations of the methods according to the invention can be transferred analogously to corresponding implementations of the active optical sensor system according to the invention. In particular, the active optical sensor system according to the invention is designed or programmed to carry out a method according to the invention. In particular, the active optical sensor system according to the invention carries out a method according to the invention.

[0068] According to a further aspect of the invention, an electronic vehicle guidance system for a motor vehicle is provided. The electronic vehicle guidance system comprises an active optical sensor system according to the invention. The at least one computing unit is configured to generate at least one control signal for guiding the motor vehicle at least in part automatically depending on the consolidated vehicle position.

[0069] Further implementations of the electronic vehicle guidance system according to the invention follow directly from the various embodiments of the methods according to the invention and vice versa. In particular, individual features and corresponding explanations as well as advantages relating to the various implementations of the methods according to the invention can be transferred analogously to corresponding implementations of the electronic vehicle guidance system according to the invention.

[0070] According to a further aspect of the invention, a computer program comprising instructions is provided. When the instructions are executed by an active optical sensor system according to the invention, in particular by the at least one computing unit, the instructions cause the active optical sensor system to carry out a method for self-localization of a motor vehicle according to the invention.

[0071] The instructions may be provided as program code, for example. The program code can for example be provided as binary code or assembler and / or as source code of a programming language, for example C, and / or as program script, for example Python.

[0072] According to a further aspect of the invention, a further computer program comprising further instructions is provided. When the further instructions are executed by an electronic vehicle guidance system according to the invention, in particular by the at least one computing unit, the instructions cause the electronic vehicle guidance system to carry out a method according to the invention for guiding a motor vehicle at least in part automatically.

[0073] The further instructions may be provided as program code, for example. The program code can for example be provided as binary code or assembler and / or as source code of a programming language, for example C, and / or as program script, for example Python.

[0074] According to a further aspect of the invention, a computer-readable storage medium, in particular a non-transitory computer-readable storage medium, storing a computer program according to the invention is provided.

[0075] The computer program, the further computer program and the computer-readable storage medium are respective computer program products with the instructions.

[0076] Further features of the invention are apparent from the claims, the figures and the figure description. The features and combinations of features mentioned above in the description as well as the features and combinations of features mentioned below in the description of figures and / or shown in the figures may be comprised by the invention not only in the respective combination stated, but also in other combinations. In particular, embodiments and combinations of features, which do not have all the features of an originally formulated claim, may also be comprised by the invention. Moreover, embodiments and combinations of features, which go beyond or deviate from the combinations of features set forth in the recitations of the claims may be comprised by the invention.

[0077] In the following, the invention will be explained in detail with reference to specific exemplary implementations and respective schematic drawings. In the drawings, identical or functionally identical elements may be denoted by the same reference signs. The description of identical or functionally identical elements is not necessarily repeated with respect to different figures.

[0078] In the figures,

[0079] Fig. 1 shows schematically a motor vehicle with an exemplary implementation of an electronic vehicle guidance system according to the invention;

[0080] Fig. 2 shows a schematic partial block diagram of an exemplary implementation of a method for self-localization of a motor vehicle according to the invention;

[0081] Fig. 3 shows schematically examples of landmarks for use in a further exemplary implementation of a method for self-localization of a motor vehicle according to the invention;

[0082] Fig. 4 shows schematic illustrations of steps of a further exemplary implementation of a method for self-localization of a motor vehicle according to the invention; and

[0083] Fig. 5 shows schematic illustrations of steps of a further exemplary implementation of a method for self-localization of a motor vehicle according to the invention.

[0084] Fig. 1 shows schematically a motor vehicle 1 with an exemplary implementation of an electronic vehicle guidance system 2 according to the invention. The electronic vehicle guidance system 2 comprises an exemplary implementation of an active optical sensor system 3 according to the invention, which is for example implemented as a lidar system, and at least one computing unit 3c, 4, which may for example comprise an ECU 4 and / or a control and / or evaluation unit 3c of the active optical sensor system 3. The electronic vehicle guidance system 2 may carry out a method for guiding a motor vehicle at least in part automatically according to the invention. To this end, the active optical sensor system 3 may carry out a method for self-localization of the motor vehicle 1 according to the invention.

[0085] The active optical sensor system 3 may generate a point cloud representing the environment of the motor vehicle 1 by emitting light pulses into the environment and detecting portions of the emitted light pulses, which are reflected by objects 21 in the environment, by means of a detector array 3b with a plurality of detector pixels. The active optical sensor system 3 comprises, apart from the detector array 3b, an emission unit 3a and the control and / or evaluation unit 3c, which is configured to control the emission unit 3a to emit the light pulses into the environment of the motor vehicle 1 . The detector array 3b is configured to generate detector signals depending on reflected portions of the emitted light pulses impinging on the detector array 3b. Furthermore, the detector array 3b is configured to generate further detector signals depending on ambient light impinging on the detector array 3b.

[0086] The control and / or evaluation unit 3c is configured to generate a point cloud depending on the detector signals and to generate an image, in particular a monochromatic image, of the environment depending on the further detector signals.

[0087] The control and / or evaluation unit 3c is configured to determine a first vehicle position of the motor vehicle 1 depending on the point cloud and a second vehicle position of the motor vehicle 1 depending on the image. The control and / or evaluation unit 3c is configured to determine a position deviation of the first vehicle position from the second vehicle position. The control and / or evaluation unit 3c is configured to determine a consolidated vehicle position 7b of the motor vehicle 1 as the first vehicle position, if the position deviation is less than a predefined threshold value, and determine the consolidated vehicle position 7b as the second vehicle position, if the position deviation is greater than the threshold value.

[0088] For example, the ECU 4 may generate at least one control signal for guiding the motor vehicle 1 at least in part automatically depending on the consolidated vehicle position. The at least one control signal may for example be provided to one or more actuators of the motor vehicle 1 , including for example one or more braking actuators and / or one or more steering actuators and / or one or more propulsion motors of the motor vehicle 1 . The one or more actuators may affect a longitudinal and / or lateral control of the motor vehicle 1 in order to guide the motor vehicle 1 at least in part automatically.

[0089] It is noted that the distribution of tasks between the ECU 4 and the control and / or evaluation unit 3c is only exemplary and may be different in other implementations. In general, the steps carried out by the ECU 4 and the control and / or evaluation unit 3c are carried out by at least one computing unit of the electronic vehicle guidance system 2.

[0090] Fig. 2 shows a schematic partial block diagram of an exemplary implementation of a method for self-localization of a motor vehicle 1 according to the invention. Further steps of the method are illustrated in Fig. 4 and Fig. 5.

[0091] For example, the point cloud and the image may be generated as described for each frame interval of a plurality of consecutive frame intervals T1 , T2, T3. For each frame interval, the first vehicle position, the second vehicle position and the consolidated vehicle position may be determined as described above. For example, a SLAM algorithm or a tracking algorithm may be used to determine and track the first vehicle position over the consecutive frame intervals. Whenever it is found that the position deviation is greater than the threshold value, the first vehicle position is set to the second vehicle position and then may be tracked further in the following frames.

[0092] For determining and tracking the first vehicle position, static landmarks 17 may be detected based on the point cloud and the first vehicle position may be iteratively updated depending on the detected relative motion of the landmarks 17. For carrying out the steps of the method according to the invention, a dynamic position estimation module 5 may take landmark input data 6a indicating the current positions of the landmarks 17 and output the consolidated vehicle position 7b. Optionally, the dynamic position estimation module 5 may also output a status 7a of the method, for example the current position deviation. Apart from the landmark input data 6a, the dynamic position estimation module 5 may, in some implementations, also take vehicle telemetry input data 6b, GPS input data 6c and / or lane input data 6d indicating a position of one or more lanes on the road relative to the motor vehicle 1 , to compute the consolidated vehicle position 7b. The dynamic position estimation module 5 may, in some implementations, also take odometry input data, for example wheel rotation date, gyroscope measurements, and so forth, to compute the consolidated vehicle position 7b. Fig. 3 shows schematically various examples of static objects, which are particularly suitable for being used as landmarks 17. These include, for example, poles 8 on the roadside, light reflectors 9 of the poles 8, light reflectors 11 of walls 10 or other barriers for structurally separating lanes from each other for structurally delimiting the road, light reflectors 13 of guardrails 12, mounting poles 14 of guardrails 12, light reflectors 16 of cement blocks 15, traffic signs, botts dots, ground markings and so forth.

[0093] As indicated in Fig. 4, several objects including static objects that are suitable to be used as landmarks 17 and for example also dynamic objects 18, such as further vehicles, are detected based on the point cloud and represented, for example, by corresponding tracking instances 19, 20. Since the relative position of the landmarks 17 with respect to the motor vehicle 1 change over the frame intervals T1 , T2, T3, the first vehicle position can be updated accordingly.

[0094] As described, in particular with reference to the figures, by means of the invention, the reliability of the self-localization of a motor vehicle is increased.

[0095] When driving a motor vehicle autonomously or in part autonomously, it is desirable to cross check the vehicle position dynamically. In several implementations, this is done based on available landmarks in the environment, which are detected by the active optical sensor system. In several implementations, autonomous driving in the vicinity of construction zones is supported. In several implementations, a road curvature estimation algorithm and / or a lane keeping assistant are supported.

[0096] In several implementations, the motor vehicle identifies the landmarks and cross checks their positions dynamically while driving at least in part autonomously mode. The landmarks may be classified using traditional and / or Al-based classification algorithms, for example as pole reflectors, road pavement markers, guardrail poles, cement block reflectors, et cetera. Based on the landmarks, the motor vehicle may constantly carry out the self-check of its position and, in some implantation, may assess the criticality of the position deviation identified with respect to other data.

Claims

Claims1 . Method for self-localization of a motor vehicle (1), wherein for a frame interval a point cloud is generated by emitting light pulses into an environment of the motor vehicle (1) by an emitter unit (3a) of an active optical sensor system (3), which is mounted to the motor vehicle (1), and detecting reflected portions of the emitted light pulses by a detector array (3b) of the active optical sensor system (3); an image of the environment is generated by detecting ambient light impinging on the detector array (3b); a first vehicle position of the motor vehicle (1) is determined depending on the point cloud and a second vehicle position of the motor vehicle (1) is determined depending on the image; a position deviation of the first vehicle position from the second vehicle position is determined; and a consolidated vehicle position (7b) of the motor vehicle (1 ) is determined as the first vehicle position, if the position deviation is less than a predefined threshold value, vehicle position, if the position deviation is less than a predefined threshold value, and the consolidated vehicle position (7b) is determined as the second vehicle position, if the position deviation is greater than the threshold value.

2. Method according to claim 1 , wherein for a previous frame interval preceding the frame interval, a previous vehicle position of the motor vehicle (1) is provided and the first vehicle position is determined depending on the previous vehicle position and the point cloud.

3. Method according to claim 2, wherein a landmark (17) is detected depending on the point cloud and a landmark position of the landmark (17) relative to the motor vehicle (1) is determined depending on the point cloud; and the first vehicle position is determined depending on the landmark position and the previous vehicle position.

4. Method according to claim 2, wherein one or more static objects in the environment are detected depending on the point cloud; each of the one or more static objects is classified according to at least two predefined static object classes; one of the one or more static objects, which belongs to a predefined subset of the at least two static object classes according to the classification, is selected as a landmark (17); a landmark position of the landmark (17) relative to the motor vehicle (1 ) is determined depending on the point cloud; and the first vehicle position is determined depending on the landmark position and the previous vehicle position.

5. Method according to claim 4, wherein the subset of the at least two static object classes comprises a class corresponding to poles; and / or a class corresponding to light reflectors; and / or a class corresponding to buildings; and / or a class corresponding to traffic signs; and / or a class corresponding to ground markings; and / or a class corresponding to botts dots.

6. Method according to one of claims 4 or 5, wherein the at least two static object classes comprise a further class corresponding to devices for structurally separating two adjacent lanes from each other and / or guardrails, wherein the subset of the at least two static object classes does not comprise the further class.

7. Method according to one of claims 3 to 6, wherein the landmark (17) or a further landmark (17) is detected depending on the image and a further landmark position of the landmark (17) or the further landmark (17) relative to the motor vehicle (1 ) is determined depending on the image; and the second vehicle position is determined depending on the further landmark position and a digital map.

8. Method according to one of the preceding claims, wherein the first vehicle position is determined by using a simultaneous-localization-and-mapping algorithm.

9. Method for guiding a motor vehicle (1) at least in part automatically, wherein a method for self-localization of a motor vehicle (1 ) according to one of the preceding claims is carried out; and at least one control signal for guiding the motor vehicle (1) at least in part automatically is generated depending on the consolidated vehicle position (7b).

10. Method according to claim 9, wherein a level of automatization for guiding the motor vehicle (1) is changed from a predefined first level of automatization to a predefined second level of automatization depending on the position deviation.11 . Method according to claim 10, wherein the level of automatization is changed from the first level of automatization to the second level of automatization, if the position deviation is greater than a predefined further threshold value, which is greater than the threshold value.

12. Method according to one of claims 9 to 11 , wherein a warning message for a driver of the motor vehicle (1) is generated depending on the position deviation.

13. Active optical sensor system (3) for a motor vehicle (1) comprising an emitter unit (3a), at least one computing unit (3c, 4), which is configured to control the emitter unit (3a) to emit light pulses into an environment of the active optical sensor system (3), and a detector array (3b), which is configured to detect reflected portions of the emitted light pulses and to generate detector signals depending on the detected reflected portions and to generate further detector signals depending on ambient light impinging on the detector array (3b), wherein the at least one computing unit (3c, 4) is configured to generate a point cloud depending on the detector signals and to generate an image of the environment depending on the further detector signals; determine a first vehicle position of the motor vehicle (1 ) depending on the point cloud and a second vehicle position of the motor vehicle (1) depending on the image;determine a position deviation of the first vehicle position from the second vehicle position; and determine a consolidated vehicle position (7b) of the motor vehicle (1 ) as the first vehicle position, if the position deviation is less than a predefined threshold value, and determine the consolidated vehicle position (7b) as the second vehicle position, if the position deviation is greater than the threshold value.

14. Active optical sensor system (3) according to claim 13, wherein the detector array (3b) comprises a plurality of detector pixels, which are arranged according to a plurality of rows and a plurality of columns, wherein a total number of the plurality of rows is at least 100 and / or a total number of the plurality of columns is at least 100.

15. Electronic vehicle guidance system (2) for a motor vehicle (1 ) comprising an active optical sensor system (3) according to one of claims 13 or 14, wherein the at least one computing unit (3c, 4) is configured to generate at least one control signal for guiding the motor vehicle (1 ) at least in part automatically depending on the consolidated vehicle position (7b).

16. Computer program product comprising instructions, which, when executed by an active optical sensor system (3) according to one of claims 13 or 14, cause the active optical sensor system (3) to carry out a method according to one of claims 1 to 8; and / or further instructions, which, when executed by an electronic vehicle guidance system (2) according to claim 15, cause the electronic vehicle guidance system (2) to carry out a method according to one of claims 9 to 12.

Images