Generating guidance information for guiding a vehicle

Abstract

According to a computer-implemented method for generating guidance information for guiding a vehicle (1) at least in part automatically and / or for informing a user of the vehicle (1), lidar sensor data (8) representing an environment of the vehicle (1) is received, the lidar sensor data (8) comprising depth information and intensity information. A static object (9) in the environment is detected at least in part based on the depth information. At least one symbol (10, 10', 10", 11) on the static object (9) is detected and classified based on the lidar sensor data (8). The guidance information is generated depending on a result of the classification of the at least one symbol (10, 10', 10", 11).

Description

[0001] Generating guidance information for guiding a vehicle

[0002] The present invention is directed to a computer-implemented method for generating guidance information for guiding a vehicle at least in part automatically, wherein lidar sensor data representing an environment of the vehicle is received. The invention is further directed to a method for guiding a vehicle at least in part automatically, wherein such a computer-implemented method is carried out, to a data processing system configured to carry out a computer-implemented method, to an electronic vehicle guidance system comprising such a data processing system, and to a corresponding computer program product.

[0003] The recognition of text or other symbols in the environment of a vehicle is an important prerequisite for many vehicle functions, in particular for driver assistance functions or other functions for guiding the vehicle automatically or in part automatically. It is well known to carry out respective software algorithms based on camera images.

[0004] One drawback of this approach is that cameras tend to have a significantly lowered performance in low light scenarios. This may potentially render the recognition of text and symbols unreliable. Consequently, also the accuracy and / or reliability of guidance information for guiding the vehicle at least in part automatically or for informing a user of the vehicle is also reduced. Furthermore, in terms of functional safety, it is desirable, in particular for high levels of automation, to provide a redundancy for the data sources perceiving the environment of the vehicle.

[0005] US 2022 / 0107391 A1 describes a method for real-time landmark extraction from a sparse three-dimensional point cloud captured by a detection and ranging sensor mounted to a vehicle.

[0006] It is an objective of the present invention to increase the reliability of guidance information for guiding a vehicle at least in part automatically and / or for informing a user of the vehicle.

[0007] This objective is achieved by the subject matter of the independent claim. Further implementations and preferred embodiments are subject matter of the dependent claims. The invention based on the idea to use depth information of lidar sensor data to detect a static object in the environment of the vehicle and to detect and classify at least one symbol on the static object based on the lidar sensor data.

[0008] According to a first aspect of the invention, a computer-implemented method for generating guidance information for guiding a vehicle at least in part automatically and / or for informing a user of the vehicle is provided. Therein, lidar sensor data representing an environment, in particular an outer environment, of the vehicle is received. The lidar sensor data comprises depth information and intensity information. A static object in the environment is detected at least in part based on the depth information. At least one symbol on the static object, in particular on a surface of the static object, is detected and classified based on the lidar sensor data, in particular based on the depth information and / or the intensity information. The guidance information is generated depending on a result of the classification of the at least one symbol.

[0009] Unless stated otherwise, all steps of the computer-implemented method may be performed by a data processing system, which comprises at least one data processing device, in particular a data processing system of the vehicle. In particular, the at least one data processing device is configured or adapted to perform the steps of the computer-implemented method. For this purpose, the at least one data processing device may for example store a computer program comprising instructions which, when executed by the at least one data processing device, cause the at least one data processing device to carry out the computer-implemented method. The terms "data processing system" and "at least one data processing device" may be used interchangeably.

[0010] All data processing devices of the at least one data processing device may be comprised by the vehicle. However, it is also possible that all data processing devices of the at least one data processing device are part of an external computing system external to the vehicle, for example a backend server or a cloud computing system. It is also possible that the at least one data processing device comprises at least one vehicle data processing device of the vehicle as well as at least one external data processing device comprised by the external computing system. The at least one vehicle data processing device may for example be comprised by one or more electronic control units, ECUs, and / or one or more zone control units, ZCUs, and / or one or more domain control units, DCUs, of the vehicle and / or by a lidar sensor system. In case the at least one data processing device comprises two or more data processing devices, certain steps carried out by the at least one data processing device may be understood such that different data processing devices carry out different steps or different parts of a step. In particular, it is not required that each data processing device carries out the steps completely. In other words, carrying out the steps may be distributed amongst the two or more data processing devices.

[0011] From each implementation of the computer-implemented method, a respective implementation of a method for generating guidance information, which is not purely computer-implemented, is obtained by including respective steps of generating the lidar sensor data by the lidar sensor system. In other words, the lidar sensor data is or has been generated by a lidar sensor system, which may but is not necessarily a part of the method according to the invention.

[0012] The lidar sensor system may be mounted to the vehicle. However, it is also possible, that the lidar sensor system is arranged separately from the vehicle, for example a part of an infrastructure in the environment.

[0013] The lidar sensor system may for example be implemented as a so-called laser scanner, in which a laser beam is deflected by means of a light deflection arrangement so that different deflection angles of the laser beam may be realized. The light deflection arrangement may, for example, contain one or more rotatably mounted mirrors.

[0014] Alternatively, the light deflection arrangement may include a mirror element with a tiltable and / or pivotable surface. The mirror element may, for example, be configured as a micro-electro-mechanical system, MEMS. In the environment, the emitted laser beams can be partially reflected, and the reflected portions may in turn hit the laser scanner, in particular the light deflection arrangement, which may direct them to a detector array of the laser scanner comprising at least one optical detector. In particular, each optical detector of the detector array generates an associated detector signal based on the portions detected by the respective optical detector. Based on the spatial arrangement of the respective optical detector, together with the current position of the light deflection arrangement, in particular its rotational position or its tilting and / or pivoting position, it is thus possible to conclude the direction of incidence of the detected reflected components of light. A processing unit or an evaluation unit of the laser scanner may, for example, perform a time-of-flight measurement to determine a radial distance of the reflecting object. Alternatively or additionally, a method according to which a phase difference between the emitted and detected light is evaluated, may be used to determine the distance. In particular, the depth information corresponds to the determined distance and the intensity information corresponds to the respective intensity of the detected light portions, which may be represented by different parameters of the sensor signals generated by the optical detectors, for example a peak height of a detected signal pulse, also denoted as echo or echo pulse, an echo pulse width, EPW, of the signal pulse, an area under the signal pulse or a particular part of the signal pulse, and so forth.

[0015] The lidar sensor system may also be a flash lidar system. Flash lidar systems are nonscanning systems, which do not require said light deflection arrangement. Therein, the laser light generated by the light source is diffused by an optical element to irradiate over a wide angle in a single flash.

[0016] Since the lidar sensor system emits light, which is being detected after being reflected in the environment, said operation principle is also denoted as active sensing. It is possible, however, that the little sensor system is also capable of detecting ambient light impinging on the detector array, which has not been emitted by the lidar sensor system, similar to the operation of a camera. This may also be denoted as passive sensing. The depth information is generated by active sensing. The intensity information may be generated by active sensing and / or passive sensing.

[0017] The lidar sensor system may, for example, generate one or more point clouds representing the environment. A point cloud may be understood as a set of points, wherein each point is characterized by respective coordinates in a three-dimensional coordinate system. The three-dimensional coordinates may, for example, be determined by the direction of incidence of the reflected components of light and the corresponding time-of-flight or radial distance measured for this respective point. In other words, the three-dimensional coordinate system may be a three-dimensional polar coordinate system. However, the information may also be pre-processed to provide three-dimensional Cartesian coordinates for each of the points. In general, the points of a point cloud may be given in an orderless or unsorted manner, in contrast to, for example, a camera image. In addition to the spatial information, namely the two-dimensional or three-dimensional coordinates, the point cloud may also store additional information or measurement values for the parameters concerning the intensity of the detected light portions. The lidar sensor data may comprise the one or more point clouds, which inherently comprises the depth of information and the intensity information. The lidar sensor data may, however, alternatively or in addition, comprise data resulting from preprocessing the one or more point clouds. For example, the depth information and the intensity information may be provided as separate datasets, in particular two-dimensional datasets, which may also be considered as images.

[0018] Since the detector array of modern lidar sensor systems may comprise a large number of rows and columns of optical detectors, in particular up to hundreds of rows and columns, the spatial resolution of the depth information and intensity information may be very high, in particular comparable to the resolution of a camera image. For example, the number of rows and / or the number of columns, respectively, may be at least 20, for example at least 100, for example at least 200. For example, the number of rows and / or the number of columns, respectively, may lie in the range [20, 1000].

[0019] The computer-implemented method according to the invention exploits this fact to detect and classify the at least one symbol based on the lidar sensor data without suffering from the drawbacks of a camera-based approach. Alternatively or in addition, by using the lidar sensor data to detect and classify the at least one symbol and, consequently, to generate the guidance information, a level of redundancy may be established, which increases the functional safety for guiding the vehicle.

[0020] The static object may be detected at least in part based on the depth information by using a known object detection algorithm, which may for example be based on a trained machine learning model, MLM, such as a trained artificial neural network, ANN.

[0021] A trained MLM can be understood as an algorithm, in particular a computer-implemented algorithm, which can reproduce functions that are possible through human intellectual performance concretely or in a broader sense. A trained MLM can also be referred to as a "trained function", for example.

[0022] When training an MLM, parameters of the MLM are generally adjusted or updated. The training may be supervised, semi-supervised or unsupervised. The training may also include reinforcement learning or representation learning and / or other known training methods. In particular, the parameters of the MLM can be adapted iteratively over several training steps. In particular, a predefined loss function can be minimized for training. If the MLM is an artificial neural network, ANN, a backpropagation algorithm can be used to adjust the parameters.

[0023] In particular, an MLM may include an ANN, a support vector machine, a k-means clustering algorithm, a decision tree, and so on. In particular, an ANN may be or include a deep neural network and / or a convolutional neural network, CNN, in particular a deep CNN, and / or a recurrent neural network, RNN, in particular a recurrent CNN, and / or a transformer network and / or a generative adversarial network, GAN.

[0024] The static object may be understood as an object in the environment of the vehicle, which does not move or essentially not move in the environment. The static object may for example be a building or a part of a building, a traffic sign, a parked other vehicle, et cetera. It is well-known to differentiate static objects from dynamic objects, which move with respect to the vehicle, based on lidar sensor data. For example, the movement of the vehicle itself in the environment may be tracked and used to differentiate static objects from dynamic objects.

[0025] The static object being detected at least in part based on the depth information can be understood such that the static object is detected solely depending on the depth information or depending on the depth information and additional data or information including, in particular, the intensity information.

[0026] A symbol of the at least one symbol may, for example, be a letter, character, glyph, numerical digit, ideogram, pictogram, punctuation mark, et cetera.

[0027] In several embodiments, the at least one symbol comprises two or more symbols.

[0028] Consequently, the information content of the classified symbols and the corresponding guidance information is increased. The guidance information may therefore account for the information given by at least one symbol more thoroughly.

[0029] Detecting and classifying the at least one symbol may for example comprise detecting and classifying each of the two or more symbols individually. Detecting a symbol may for example comprise determining that the symbol is present, while classifying the symbol may comprise assigning one of a plurality of predefined symbol classes to the respective symbol. The symbol classes may for example correspond to the individual letters of an alphabet and / or a respective collection of other symbols. The result of the classification of the at least one symbol may for example comprise the respective symbol class for each of the at least one symbol and / or respective information derived from said classes including, but not limited to, a semantic meaning of all symbols of the at least one symbol in combination.

[0030] In some embodiments, the detection and classification may be carried out in at least two steps, a first step comprising a preliminary detection of the presence of the respective symbol on the static object and a second step comprising a validation of the presence of the static object and the classification. The first step may for example include an analysis of contrast in the region corresponding to the static object. If the first step reveals that it is likely that at least one symbol is present, the second step is carried out. The second step may for example include applying a respective symbol detection algorithm, for example text detection algorithm, which may for example be based on a further trained MLM.

[0031] In some embodiments, the semantic meaning of the classified two or more symbols is determined according to a predefined lexicon and the guidance information is generated depending on the semantic meaning.

[0032] Consequently, the information content of the guidance information is increased. The guidance information may therefore account for the information given by at least one symbol more thoroughly.

[0033] In some embodiments, the guidance information comprises, in particular for guiding the vehicle at least in part automatically, driver assistance information for assisting a driver of the vehicle at guiding the vehicle.

[0034] The driver assistance information may be output by means of an output device of the vehicle, for example a display and / or an audio output system and / or a haptic output system.

[0035] In some embodiments, the guidance information comprises, in particular for guiding the vehicle at least in part automatically, at least one control signal for at least one actuator of the vehicle for lateral control and / or longitudinal control of the vehicle.

[0036] The at least one control signal may for example be provided to the at least one actuator of the vehicle, including for example one or more braking actuators and / or one or more steering actuators and / or one or more propulsion motors of the vehicle. The at least one actuator may affect the longitudinal and / or lateral control of the vehicle in order to guide the vehicle at least in part automatically based on the at least one control signal.

[0037] In some embodiments, the guidance information comprises, in particular for informing the user of the vehicle, user information concerning the static object and / or the environment, such as landmarks, sights, monuments, shopping facilities et cetera.

[0038] According to several embodiments, at least one candidate object in the environment is detected at least in part based on the depth information. The at least one candidate object is classified according to two or more object classes including at least one static object class and at least one dynamic object class. The static object is selected as one of the at least one candidate object, which has been assigned to the at least one static object class, in particular by said classification.

[0039] In other words, dynamic objects are filtered out for the purpose of detecting and classifying the at least one symbol. It is noted that this does not necessarily imply that the guidance information is generated independent of dynamic objects.

[0040] According to several embodiments, the at least one static object class comprises at least one first static object class and at least one second static object class, wherein the sets of at least one first static object class and at least one second static object class are, in particular, disjoint. The static object is selected as one of the at least one candidate object, which has been assigned to the at least one first static object class.

[0041] In other words, static objects assigned to the at least one second static object class are filtered out for the purpose of detecting and classifying the at least one symbol. It is noted that this does not necessarily imply that the guidance information is generated independent of such static objects.

[0042] By excluding static objects assigned to the at least one second static object class for detecting and classifying the at least one symbol, particular types of static objects, which are not expected to carry relevant information for guiding the vehicle or informing the user, the computational effort is reduced.

[0043] The at least one first static object class may for example include a class for buildings or parts of buildings, traffic signs, standing vehicles, et cetera. The at least one second static object class may for example include a class for a ground surface, trees, other vegetation, et cetera.

[0044] According to several embodiments, a region of interest on the static object, in particular on the surface of the static object, is determined based on the lidar sensor data, for example at least partially based on the intensity information. A two-dimensional image depicting the region of interest is generated based on the lidar sensor data, for example at least partially based on the intensity information. Classifying the at least one symbol comprises applying a trained classification model to input data comprising the image.

[0045] The classification model is, in particular, an MLM, for example an ANN, which has been trained for a respective classification task using a conventional training method.

[0046] An advantage of such embodiments is that, in particular due to the high spatial resolution of the lidar sensor data, known classification models that have been designed for camera images can be reused for the two-dimensional image generated based on the lidar sensor data without adaptations or only with minor adaptions being necessary.

[0047] For example, the region of interest may be defined by a bounding box, for example a rectangular bounding box or another geometrical figure, enclosing the region of interest in the two-dimensional image. The region of interest may for example be determined by analyzing a contrast of the two-dimensional image, applying an edge detection algorithm to the two-dimensional image or the like.

[0048] According to several embodiments, classifying the at least one symbol comprises applying a trained classification model to input data comprising the lidar sensor data, for example the one or more point cloud or three-dimensional or 2.5-dimensional data derived from the one or more point clouds.

[0049] The classification model is, in particular, an MLM, for example an ANN, which has been trained for a respective classification task using a conventional training method.

[0050] According to several embodiments, wherein a semantic meaning of the classified two or more symbols is determined according to a predefined lexicon and the guidance information is generated depending on the semantic meaning. In such embodiments, the information content of the guidance information is increased. The guidance information may therefore account for the information given by at least one symbol more thoroughly.

[0051] The semantic meaning is, in particular, a semantic meaning of the two or more symbols in combination.

[0052] According to several embodiments, the intensity information corresponds to an intensity of light portions being emitted by the lidar sensor system, being reflected in the environment, and, after being reflected, being detected by the lidar sensor system.

[0053] In other words, the intensity information is determined by active sensing. Consequently, the accuracy and reliability of the computer-implemented method are increased in low light scenarios.

[0054] According to several embodiments, the intensity information is determined by active sensing and the lidar sensor data comprises further intensity information corresponding to an intensity of ambient light passively sensed by the lidar sensor system. The static object in the environment is detected at least in part based on the further intensity information and / or the at least one symbol is detected and classified based on the further intensity information.

[0055] In such embodiments, the accuracy and reliability of the computer-implemented method are further increased. The active sensing and the passive sensing may for example be carried out one after the other in each respective frame interval of the lidar sensor system.

[0056] According to several embodiments, the depth information corresponds to a time-of-flight of light portions being emitted by the lidar sensor system, being reflected in the environment, and, after being reflected, being detected by the lidar sensor system.

[0057] In other words, the depth information is determined by active sensing. Consequently, the accuracy and reliability of the computer-implemented method are increased in low light scenarios.

[0058] According to a second aspect of the invention, method for guiding a vehicle at least in part automatically is provided. Therein, a computer-implemented method according to the invention is carried out and the vehicle is guided at least automatically depending on the guidance information.

[0059] The data processing system and / or the lidar sensor system may for example be part of an electronic vehicle guidance system.

[0060] An electronic vehicle guidance system may be understood as an electronic system, configured to guide a vehicle in a fully automated or a fully autonomous manner and, in particular, without a manual intervention or control by a driver or user of the vehicle being necessary. The vehicle carries out all required functions, such as steering maneuvers, deceleration maneuvers and / or acceleration maneuvers as well as monitoring and recording the road traffic and corresponding reactions automatically. In particular, the electronic vehicle guidance system may implement a fully automatic or fully autonomous driving mode according to level 5 of the SAE J3016 classification. An electronic vehicle guidance system may also be implemented as an advanced driver assistance system, ADAS, assisting a driver for partially automatic or partially autonomous driving. In particular, the electronic vehicle guidance system may implement a partly automatic or partly autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification. Here and in the following, SAE J3016 refers to the respective standard dated April 2021.

[0061] Guiding the vehicle at least in part automatically may therefore comprise guiding the vehicle according to a fully automatic or fully autonomous driving mode according to level 5 of the SAE J3016 classification. Guiding the vehicle at least in part automatically may also comprise guiding the vehicle according to a partly automatic or partly autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification.

[0062] Guiding the vehicle at least in part automatically may, in particular, comprise generating and outputting the driver assistance information.

[0063] Guiding the vehicle at least in part automatically may, in particular, comprise generating the at least one control signal, providing the at least one control signal to the at least one actuator and carrying out or affecting the lateral control and / or longitudinal control of the vehicle by the at least one actuator.

[0064] According to a third aspect of the invention, a data processing system is provided, which is configured to carry out a computer-implemented method according to the invention. The terms "data processing system" and "at least one data processing device" may be used interchangeably in the present disclosure. In the present disclosure, a data processing device, also denoted as computing device, may for example be understood as a device with processing circuitry for processing data. A data processing device can therefore perform computing operations in order to process data. An indexed access to a data structure, for example a look-up table, LUT, or a database may also be considered as a computing operation. Data processing that is partially or fully implemented in hardware can also be considered a computing operation.

[0065] In particular, a data processing device may include one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example, one or more application-specific integrated circuits, ASIC, one or more field-programmable gate arrays, FPGA, and / or one or more systems-on-a-chip, SoC. A data processing device may also include one or more processors, for example one or more microprocessors, one or more central processing units, CPU, one or more graphics processing units, GPU, and / or one or more signal processors, in particular one or more digital signal processors, DSP. The data processing device may also include a physical or a virtual cluster of computers or other of said devices.

[0066] A data processing device may also comprise one or more hardware and / or software interfaces, for example for receiving and / or providing data, respectively.

[0067] A data processing device may also comprise one or more memory devices. Therein, a memory device may be implemented as a volatile memory, for example a dynamic random access memory, DRAM, or a static random access memory, SRAM, or as a nonvolatile data memory, for example a read-only memory, ROM, a programmable read-only memory, PROM, an erasable programmable read-only memory, EPROM, an electrically erasable programmable read-only memory, EEPROM, a flash memory or flash EEPROM, a ferroelectric random access memory, FRAM, a magnetoresistive random access memory, MRAM, or a phase-change random access memory, PCRAM.

[0068] According to a fourth aspect of the invention, an electronic vehicle guidance system is provided. The electronic vehicle guidance system comprises a data processing system according to the invention and the lidar sensor system, which is configured to generate the lidar sensor data.

[0069] Further implementations of the electronic vehicle guidance system according to the invention follow directly from the various embodiments of the methods according to the first and second aspect of the invention and vice versa. In particular, individual features and corresponding explanations as well as advantages relating to the various implementations of the methods according to the first and second aspect of the invention can be transferred analogously to corresponding implementations of the electronic vehicle guidance system according to the invention. In particular, the electronic vehicle guidance system according to the invention is designed or programmed to carry out a method according to the first or second aspect of the invention. In particular, the electronic vehicle guidance according to the invention carries out a method according to the first or second aspect of the invention.

[0070] According to a further aspect of the invention, a computer program comprising instructions is provided. When the instructions are executed by a data processing system, the instructions cause the data processing system to carry out a computer-implemented method according to the first aspect of the invention.

[0071] The instructions may be provided as program code, for example. The program code can for example be provided as binary code or assembler and / or as source code of a programming language, for example C, and / or as program script, for example Python.

[0072] According to a further aspect of the invention, a computer-readable storage medium storing a computer program according to the invention is provided.

[0073] The computer program and the computer-readable storage medium are respective computer program products comprising the instructions.

[0074] Further features of the invention are apparent from the claims, the figures and the figure description. The features and combinations of features mentioned above in the description as well as the features and combinations of features mentioned below in the description of figures and / or shown in the figures may be comprised by the invention not only in the respective combination stated, but also in other combinations. In particular, embodiments and combinations of features, which do not have all the features of an originally formulated claim, may also be comprised by the invention. Moreover, embodiments and combinations of features, which go beyond or deviate from the combinations of features set forth in the recitations of the claims may be comprised by the invention. In the following, the invention will be explained in detail with reference to specific exemplary implementations and respective schematic drawings. In the drawings, identical or functionally identical elements may be denoted by the same reference signs. The description of identical or functionally identical elements is not necessarily repeated with respect to different figures.

[0075] In the figures,

[0076] Fig. 1 shows schematically a vehicle with an exemplary implementation of an electronic vehicle guidance system according to the invention;

[0077] Fig. 2 shows schematically a histogram representing a number of detected light pulses as a function of time;

[0078] Fig. 3 shows schematically an amplitude of a signal pulse according to a detected light pulse as a function of time;

[0079] Fig. 4 shows a schematic flow diagram of an exemplary implementation of method for guiding a vehicle at least in part automatically according to the invention;

[0080] Fig. 5 shows a schematic representation of lidar sensor data;

[0081] Fig. 6 shows the schematic representation of lidar sensor data of Fig. 5 and a bounding box for a static object;

[0082] Fig. 7 shows the schematic representation of lidar sensor data of Fig. 5 and a region of interest including symbols; and

[0083] Fig. 8 shows the region of interest of Fig. 7 in more detail.

[0084] Fig. 1 shows schematically a vehicle 1 , in particular a motor vehicle, for example a car or another motorized land vehicle, with an exemplary implementation of an electronic vehicle guidance system 2 according to the invention.

[0085] The vehicle 1 , for example the electronic vehicle guidance system 2, comprises a lidar sensor system 3, which is configured to generate lidar sensor data 8, for example a point cloud, comprising depth information and intensity information. The electronic vehicle guidance system 2 comprises a data processing system 4 according to the invention, which is configured to generate guidance information for guiding the vehicle 1 at least in part automatically and / or for informing a user of the vehicle 1 depending on the lidar sensor data 8 by carrying out a computer-implemented method according to the invention.

[0086] The lidar sensor system 3 may for example be implemented as a laser scanner and may operate for example based on the principle of a direct or indirect time-of-flight measurement. The lidar sensor system 3 may emit light pulses 6 into the environment of the vehicle 1 and detect light pulses 7, which are portions of the emitted light pulses 6 reflected in the environment, for example by one or more objects 5 such as a further vehicle in the non-limiting example of Fig. 1. The lidar sensor system 3 may determine the time-of-flight of the detected light pulses 7 and compute the distance d of the reflecting object 5 from the these are sensor system 3 based on the time-of-flight as d = T / (2c), wherein T denotes the time-of-flight and c denotes the speed of light. The distance d may also be denoted as depth.

[0087] Fig. 2 shows schematically a histogram representing a number # of detected light pulses 7 as a function of time t. The squares in the lowest row in the histogram corresponds for example to detected ambient light, while the remaining squares correspond to the detected light pulses 7.

[0088] Fig. 3 shows schematically an amplitude A of a signal pulse generated by an optical detector of the lidar sensor system 3 based on a detected light pule 7 as a function of time t. The depth information may for example correspond to the distance d of the various points in the point cloud 8 (see for example Fig. 5). The intensity information may for example correspond to different parameters that may be extracted from the signal pulse, for example a height H, given by a maximum of the amplitude A, or an EPW, given by the time between a first time instance ti, where the amplitude first exceeds a predefined threshold amplitude, and a second time instance t2, where the amplitude drops below the threshold amplitude. Another suitable parameter may be an area under the signal pulse, for example between the first time instance ti and the second time instance t2.

[0089] To carry out the computer-implemented method for generating guidance information, the data processing system 4 receives the lidar sensor data 8 representing the environment of the vehicle 1 , and detects a static object 9 in the environment at least in part based on the depth information, as depicted in Figs. 5 and 6. In the example of Fig. 5, the static object 9 is for example a building. The data processing system 4 detects and classifies at least one symbol 10, 10', 10", 11 , for example letters 10, 10', 10" or other symbols 11 , on the static object 9 based on the lidar sensor data 8, as depicted in Figs. 7 and 8, and generates the guidance information depending on a result of the classification of the at least one symbol 10, 10', 10", 11.

[0090] Fig. 4 shows a schematic flow diagram of an exemplary implementation of a method for guiding the vehicle 1 at least in part automatically according to the invention including an exemplary implementation of a computer-implemented method for generating guidance information according to the invention.

[0091] In steps 400, the lidar sensor system 3 generates the lidar sensor data 8. In steps 410 to 430, the data processing system 4 generates the guidance information. For example, the guidance information may comprise driver assistance information for assisting a driver of the vehicle 1 at guiding the vehicle 1 and / or at least one control signal for at least one actuator of the vehicle 1 for lateral control and / or longitudinal control of the vehicle 1 and / or user information concerning the static object 9 and / or the environment. In step 440, the vehicle 1 is guided at least automatically depending on the guidance information.

[0092] In particular, in step 410, the static object 9 in the environment is detected at least in part based on the depth information, for example by determining a two-dimensional or three-dimensional bounding box 12 for the static object 9, as depicted in Fig. 6. In step 420, a region of interest 13 on the static object 9 may be determined based on the lidar sensor data 8, wherein the region of interest 13 potentially contains the at least one symbol 10, 10', 10", as depicted in Fig. 7. In step 430, the presence of the at least one symbol 10, 10', 10" may be validated and the at least one symbol 10, 10', 10" may be classified based on the region of interest 13, as depicted in Fig. 8.

[0093] According to several embodiments, the static objects 9 in the environment, for example infrastructure components, are isolated from the rest of the environment based on the depth information. Considering the intensity information, the presence of characteristic patterns on the static objects 9 may be inferred. Once the patterns are segregated, the intensity information may be used to separate text and / or other symbols 10, 10', 10", 11 from the static objects 9. Then, using image processing techniques, the text and / or other symbols 10, 10', 10", 11 are identified, classified, and their semantic meaning may be extracted. In particular, level 3, level 4 and level 5 autonomous driving may require or benefit from a redundancy for the sources of information, which may be achieved by combining the use of lidar sensor data 8 according to the invention with other sensor data, for example from cameras and / or radar systems. In particular, the lidar sensor system 3 may provide similar depth information as radar systems with a resolution and contrast information comparable to cameras using a different sensing principle.

[0094] Exemplary use cases of the embodiments of the invention include, for example, the detection of readable signs on infrastructures, which adds a layer of additional information that may be used by a fusion system to localize the vehicle 1 and control it accordingly. It is also possible to create high definition maps actively in low light conditions and / or to identify landmarks or monuments to provide general information to the user of the vehicle 1.

Claims

Claims1. Computer-implemented method for generating guidance information for guiding a vehicle (1 ) at least in part automatically and / or for informing a user of the vehicle (1 ), whereinlidar sensor data (8) representing an environment of the vehicle (1) is received, the lidar sensor data (8) comprising depth information and intensity information;a static object (9) in the environment is detected at least in part based on the depth information;at least one symbol (10, 10', 10", 11 ) on the static object (9) is detected and classified based on the lidar sensor data (8); andthe guidance information is generated depending on a result of the classification of the at least one symbol (10, 10', 10", 11 ).

2. Computer-implemented method according claim 1 , whereinat least one candidate object in the environment is detected at least in part based on the depth information;the at least one candidate object is classified according to two or more object classes including at least one static object class and at least one dynamic object class; andthe static object (9) is selected as one of the at least one candidate object, which has been assigned to the at least one static object class.

3. Computer-implemented method according to claim 2, whereinthe at least one static object class comprises at least one first static object class and at least one second static object class; andthe static object (9) is selected as one of the at least one candidate object, which has been assigned to the at least one first static object class.

4. Computer-implemented method according to claim 3, wherein the second static object class comprises a class corresponding to vegetation and / or a class corresponding to a ground surface.

5. Computer-implemented method according to one of the preceding claims, wherein a region of interest (13) on the static object (9) is determined based on the lidar sensor data (8);a two-dimensional image depicting the region of interest (13) is generated based on the lidar sensor data (8); andclassifying the at least one symbol (10, 10', 10", 11 ) comprises applying a trained classification model to input data comprising the image.

6. Computer-implemented method according to one of claims 1 to 3, wherein classifying the at least one symbol (10, 10', 10", 11 ) comprises applying a trained classification model to input data comprising the lidar sensor data (8).

7. Computer-implemented method according to one of the preceding claims, wherein the at least one symbol (10, 10', 10", 11 ) comprises two or more symbols.

8. Computer-implemented method according to claim 7, wherein detecting and classifying the at least one symbol (10, 10', 10", 11) comprises detecting and classifying each of the two or more symbols individually.

9. Computer-implemented method according to one of claims 7 or 8, wherein a semantic meaning of the classified two or more symbols is determined according to a predefined lexicon and the guidance information is generated depending on the semantic meaning.

10. Computer-implemented method according to one of the preceding claims, wherein the intensity information corresponds to an intensity of light portions (6, 7) being emitted by a lidar sensor system (3), being reflected in the environment, and being detected by the lidar sensor system (3);the lidar sensor data (8) comprises further intensity information corresponding to an intensity of ambient light passively sensed by the lidar sensor system (3); andthe static object (9) in the environment is detected at least in part based on the further intensity information and / or the at least one symbol (10, 10', 10", 11) is detected and classified based on the further intensity information.

11. Computer-implemented method according to one of the preceding claims, wherein the guidance information comprisesdriver assistance information for assisting a driver of the vehicle (1 ) at guiding the vehicle (1); and / orat least one control signal for at least one actuator of the vehicle (1 ) for lateral control and / or longitudinal control of the vehicle (1); and / oruser information concerning the static object (9) and / or the environment.

12. Method for guiding a vehicle (1) at least in part automatically, wherein a computer- implemented method according to one of the preceding claims is carried out and the vehicle (1) is guided at least automatically depending on the guidance information.

13. Data processing system (4) configured to carry out a computer-implemented method according to one of the preceding claims.

14. Electronic vehicle guidance system (2) comprising a data processing system (4) according to claim 13 and a lidar sensor system (3), which is configured to generate the lidar sensor data (8).

15. Computer-program product comprising instructions, which, when executed by a data processing system (4), cause the data processing system (4) to carry out a computer-implemented method according to one of claims 1 to 12.

Images