Method for operating a lidar sensor system of a motor vehicle and lidar sensor system for a motor vehicle

Abstract

To operate a lidar sensor system (3) of a motor vehicle (1), an image (5) is generated for each of at least two frame intervals, including a final frame interval, by detecting ambient light incident on a detector array of the lidar sensor system (3). For the final frame interval, a point cloud is generated by emitting laser pulses into the environment and detecting reflected components of the emitted laser pulses by the detector array. A static object (6, 7, 8) is detected in each of the images (5), and a change in the pose of the lidar sensor system (3) relative to the motor vehicle (1) is determined depending on the static object (6, 7, 8). The point cloud is corrected by applying a geometric transformation that includes a translation dependent on the change in pose and / or a rotation dependent on the change in pose.

Description

[0001] The present invention is directed to a method for operating a lidar sensor system, to a method for at least partially automatically guiding a motor vehicle, wherein such a method is carried out for operating a lidar sensor system, to a corresponding lidar sensor system for a motor vehicle, to an electronic vehicle guidance system and to corresponding computer program products.

[0002] Lidar sensor systems can be used for a wide range of driver assistance functions and other functions for the at least partially automatic guidance of a motor vehicle. For this purpose, a lidar sensor system can be mounted on the motor vehicle in such a way that point clouds generated by the lidar sensor system represent the external environment of the motor vehicle. Based on the point cloud or based on a plurality of consecutively recorded point clouds of corresponding consecutive frame intervals, control signals for at least partially automatically guiding the motor vehicle and / or assistance information for supporting a driver of the motor vehicle can be generated. For example, known object detection, object tracking, and / or classification algorithms can be applied to the point clouds, and their results can be used to guide the motor vehicle.

[0003] In particular, point clouds from lidar sensor systems can contain information about the position of objects in the vicinity of the lidar sensor system, and in particular, about their distance from the lidar sensor system. To do this, the lidar sensor system emits laser pulses and detects portions of the emitted laser pulses that are reflected by the objects in the vicinity using the detector array. Known methods for direct or indirect time-of-flight (ToF) measurement can be used to determine the distance of the objects accordingly.

[0004] During the lifetime of the lidar sensor system, i.e., while the vehicle with the lidar sensor system is traveling on roads, the position and / or orientation of the lidar sensor system relative to the vehicle may change. These changes can be caused, for example, by vibrations due to uneven road surfaces or when the vehicle drives over bumps, etc. As a result, the point clouds generated by the lidar system are less accurate, and functions that use the point clouds are less reliable.

[0005] An object of the present invention is to provide a possibility for at least partially correcting a point cloud of a lidar sensor system in the event of a pose change of the lidar sensor system relative to the motor vehicle.

[0006] This object is achieved by the respective subject matter of the independent claims. Further embodiments and preferred embodiments are the subject matter of the dependent claims.

[0007] The invention is based on the idea of ​​using the same detector array as used to generate the point cloud to detect ambient light, using the resulting images of the environment for two or more frames to identify a pose change of the lidar sensor system relative to the motor vehicle based on a detected static object in the images, and correcting the point cloud by applying a corresponding geometric transformation based on the pose change.

[0008] According to one aspect of the invention, a method for operating a lidar sensor system, in particular a laser scanner, of a motor vehicle is provided. For each of the two frame intervals, including a final frame interval, a respective image of the surroundings of the motor vehicle is generated by detecting ambient light incident on a detector array of the lidar sensor system during the respective frame interval. A point cloud is generated for the final frame interval by emitting laser pulses into the surroundings, in particular by a transmitting unit of the lidar sensor system, which is controlled by at least one control and / or evaluation unit of the lidar sensor system, and detecting reflected portions of the laser pulses emitted by the detector array.A static object is detected in each of the images generated for the at least two frames, in particular by the at least one control and / or evaluation unit, and a pose change of the lidar sensor system relative to the motor vehicle is determined, in particular by at least one control and / or evaluation unit, depending on the static object, in particular the static object detected for the at least two frame intervals. The point cloud is corrected by applying a geometric transformation to the point cloud, in particular by the at least one control and / or evaluation unit, which includes a translation depending on the pose change and / or a rotation depending on the pose change.

[0009] Here and below, "light" can be understood to include electromagnetic waves in the visible, infrared, and / or ultraviolet ranges. Accordingly, the term "optical" can be understood to refer to light in this sense.

[0010] A laser scanner is a known embodiment of lidar sensor systems in which a laser beam is generated by one or more laser diodes of the lidar sensor system, particularly in the form of laser pulses, and is deflected by a light deflection arrangement so that different deflection angles of the laser beam can be realized. The light deflection arrangement can, for example, contain one or more rotatably mounted mirrors. Alternatively, the light deflection arrangement can include a mirror element with a tiltable and / or pivotable surface. The mirror element can, for example, be configured as a microelectromechanical system (MEMS). The emitted laser beams can be partially reflected in the environment, and the reflected portions can in turn strike the laser scanner, particularly the light deflection arrangement, which can direct them onto the detector array of the laser scanner.In particular, each detector pixel of the detector array can generate a respective detector signal based on the detected light.

[0011] Based on the spatial arrangement of the respective detector pixel together with the current position of the light deflection arrangement, in particular its rotational position or its tilt and / or pivoting position, it is thus possible to determine the direction of incidence of the detected reflected light components. The at least one control and / or evaluation unit of the laser scanner can, for example, perform a time-of-flight measurement or an indirect time-of-flight measurement to determine a radial distance of the reflecting object. Reflected components can be understood as components of the emitted light that are reflected back by the objects in the surrounding area, including the road surface. This does not necessarily mean specularly reflected light. Rather, the reflected components can also include retroreflected and / or scattered light.

[0012] The final frame interval is referred to here and below as "final" to express that one or more further frame intervals of the at least two frame intervals precede the final frame interval. In particular, the term "final" does not imply that no other frame interval follows the final frame interval. Furthermore, it is also possible that respective further point clouds are generated analogously for other frame intervals of the at least two frame intervals besides the final frame interval.

[0013] The detector array has a plurality of detector pixels. A single detector pixel of the detector array does not necessarily consist of a single optical detector. Rather, it is also possible for a group of several adjacent optical detectors to form a detector pixel. The latter is particularly possible when single-photon avalanche diodes (SPADs) are used. In other embodiments, however, it is also possible for a pixel to consist of exactly one optical detector, for example, a single photodiode or avalanche photodiode (APD).

[0014] Since the detector array includes a plurality of detector pixels arranged, for example, according to a plurality of columns (e.g., several tens to several hundred columns) and a plurality of rows (e.g., several tens to several hundred rows), in addition to generating the point cloud, the detector array can also be used to capture two-dimensional images of the environment in the same or similar manner as a camera, such as a thermal camera. The image of the environment is generated by detecting ambient light incident on the detector array during the initial frame interval.

[0015] In other words, no laser pulses need to be emitted into the environment to generate the image, and no distance needs to be calculated using, for example, a ToF measurement. In other words, the image is generated independently of the laser pulses emitted during the final frame interval or other respective frame intervals, and independently of the corresponding reflected portions. The image is therefore, in particular, represented by a monochromatic image, for example, a grayscale image.

[0016] In particular, the images each contain a plurality of image pixels, with each image pixel corresponding to one of the detector pixels. For each image pixel, the image stores a pixel value dependent on the amount of ambient light energy incident on the respective detector pixel during a corresponding exposure duration within the first frame interval. Compared to the images, each point of the point cloud also corresponds to one of the detector pixels, but it may be that not every detector pixel detects reflected portions of the emitted laser pulses. In particular, the influence of ambient light can even be removed using known ambient light suppression algorithms to generate the point cloud.

[0017] For each point of the point cloud, apart from the respective position of the detector pixel, further information can be stored, including, for example, the radial distance determined by means of the direct or indirect ToF measurement, an energy measure concerning the amount of optical energy of the respective reflected portions of the laser pulses, such as an echo pulse width, EPW, an area under a signal pulse, etc.

[0018] Therefore, the image can be processed by conventional image processing algorithms or by known computer vision algorithms, for example for object detection and classification, in particular to detect and classify the static object as such. In particular, the static object can be detected and classified independently of the point cloud. For example, an object detection algorithm can detect an object in the respective image and assign one of a plurality of object classes to the detected object. If the assigned object class corresponds to a static object class, then the detected object is a static object. Alternatively or in addition to the object detection algorithm, a semantic segmentation algorithm or a panoptic segmentation algorithm can also be used to detect and classify the static object.

[0019] A static object corresponds to an object type that does not move relative to the roadway on which the motor vehicle is traveling. For example, the road surface itself or parts of the road surface, traffic signs, lane markings, guardrails, buildings, etc., can represent static objects.

[0020] In particular, upon detecting the static object in each of the images, a position and / or orientation of the static object in the respective image is determined, and the pose change is determined depending on the determined positions and / or orientations of the static object. For example, the pose change of the lidar sensor system can be determined based on the determined sequence of position and / or orientation of the static object and a movement of the motor vehicle during the respective at least two frame intervals. The movement of the motor vehicle can be determined, for example, based on odometric methods, positioning based on global navigation satellite systems (GNSS), etc.

[0021] The pose change of the lidar sensor system corresponds to a change in the position and / or orientation of the lidar sensor system relative to the motor vehicle. Since the images are two-dimensional, it is possible that not every position change and not every orientation change of the lidar sensor system with respect to the vehicle can be detected based on the images. Consequently, the point cloud may only be partially corrected if the pose change of the lidar sensor system is detectable based on the images. Pose changes that cannot be detected based on the images can be ignored or determined using alternative methods.

[0022] In particular, to determine the pose change of the lidar sensor system, the positions and / or orientations of the static object detected in the images can be compared, and the result of the comparison can be checked with respect to the movement of the motor vehicle. Changes in the position and / or orientation of the static object in the images that cannot be correlated with the movement of the motor vehicle can be interpreted as the corresponding pose change of the lidar sensor system relative to the motor vehicle.

[0023] Instead of directly comparing the positions and / or orientations of the static object in the images, an object tracking algorithm can be used to track the position and / or orientation of the static object in the images, and the tracked position and / or orientation of the static object can be evaluated with respect to the movement of the motor vehicle to detect the pose change of the lidar sensor system with respect to the motor vehicle.

[0024] In other words, if it is determined that the change in the position and / or orientation of the static object corresponds to the movement of the motor vehicle, then, for example, no pose change of the lidar sensor system relative to the motor vehicle is identified. Otherwise, the geometric transformation can be understood as a reversal of the detected pose change.

[0025] By means of the method according to the invention, the point cloud can be at least partially corrected without requiring an analysis of several point clouds for more than a single frame interval, which reduces the required computing resources.

[0026] Since the images of the environment are generated by exactly the same detector array used to generate the point cloud, no additional sensor system or type of sensor system, such as a camera, needs to be used to implement the inventive method. This reduces the complexity of the system and data analysis, and consequently also the risk of system failure.

[0027] In application cases or application situations that may arise in a method according to the invention and which are not explicitly described here, it may be provided that, according to the method, an error message and / or a request for user feedback is output and / or a standard setting and / or a predetermined initial state is set.

[0028] According to some embodiments, inertial sensor data is generated by an inertial sensor system, in particular an inertial measurement unit (IMU) of the lidar sensor system, and the determination of the pose change depending on the static object is triggered when, for example, only when a movement of the detector array is detected based on the inertial sensor data, in particular by the at least one control and / or evaluation unit.

[0029] In particular, the inertial sensor data represent respective accelerations of the detector array according to different spatial directions and / or respective rotation rates of the detector array about different rotation axes. Here and in the following, “acceleration” refers to an acceleration of a translational motion as opposed to a centripetal acceleration, which causes a rotational motion in the corresponding reference coordinate system.

[0030] The inertial sensor system is rigidly connected to the detector array in the sense that any force acting on the detector array also acts on the inertial sensor system. For example, the detector array and the inertial sensor system may be arranged within the same housing of the lidar sensor system. Consequently, if the lidar sensor system is mounted on the motor vehicle, the accelerations and / or rotation rates detected by the inertial sensor system may result from movement of the vehicle relative to the roadway and / or from movement of the detector array relative to the motor vehicle. However, if there is movement of the detector array relative to the motor vehicle, it is likely that movement of the detector array will also be detected based on the inertial sensor data.By using the motion of the detector array detected based on the inertial sensor data as a trigger to determine the pose change based on the images, the computational resources used can be reduced.

[0031] According to some embodiments, the pose change is validated depending on the static object, in particular by the at least one control and / or evaluation unit, depending on the inertial sensor data, and the point cloud is corrected depending on a result of the validation.

[0032] In particular, depending on the vehicle's movement, it is checked whether the inertial sensor data matches the determined pose change of the static object. If this is the case, the point cloud can be corrected as described. Otherwise, for example, no correction of the point cloud may be made. This improves the reliability and robustness of the method.

[0033] According to some embodiments, the inertial sensor data corresponds to a respective acceleration and / or rotation of the detector array for at least two measurement intervals of the inertial sensor system.

[0034] Consequently, the inertial sensor data can be used to estimate an expected movement of the detector array over the at least two measurement intervals.

[0035] According to some embodiments, the total measurement duration of the at least two measurement intervals is shorter than a frame duration of one of the at least two frame intervals.

[0036] In other words, the measurement rate of the inertial sensor system is greater than the frame rate for generating the images, and in particular the corresponding point clouds. For example, the frame rate can be on the order of several tens of Hertz, while the measurement rate, i.e., the number of measurement intervals per unit time, can be on the order of a thousand or several thousand Hertz. Consequently, the sensor data contains detailed information regarding the accelerations and / or rotations for each frame and, consequently, for each of the images.

[0037] According to some embodiments, the lidar sensor system includes a housing, with the detector array and the inertial sensor system disposed within the housing.

[0038] For example, the detector array and the inertial sensor system are both rigidly mounted to the same housing and / or to each other. In other words, any movement of the housing coincides with a respective movement of the detector array and the inertial sensor system.

[0039] According to some embodiments, an image position and / or an image orientation of the static object in the respective image is determined for the final frame and for a further frame interval of the at least two frame intervals. The pose change of the static object is determined based on a comparison of the determined image positions and / or image orientations.

[0040] In this way, the pose change can be easily determined without requiring extensive computational resources.

[0041] According to some embodiments, an object tracking algorithm is applied to the respective images of the at least two frame intervals to track the static object, and the pose change of the static object is determined based on a result of the tracking algorithm.

[0042] In this way, the pose change can be determined more precisely.

[0043] According to some embodiments, the static object is detected by applying an object detection algorithm to the respective image and classified as static.

[0044] The object detection algorithm can be a known object detection algorithm for two-dimensional images, based, for example, on a trained artificial neural network, in particular a convolutional neural network (CNN). This achieves fast and reliable detection and classification of the object, increasing the reliability of the method according to the invention.

[0045] According to some embodiments, the plurality of detector pixels of the detector array are arranged according to a plurality of pixel groups that are illuminated to detect the ambient light one after the other during respective successive subintervals of the final frame interval to generate the respective image. This may apply analogously to the further frame intervals of the at least two frame intervals.

[0046] For example, the plurality of detector pixels are arranged according to a plurality of columns and rows of the detector array, and each of the pixel groups corresponds to one of the plurality of columns or at least two adjacent columns of the plurality of columns. In particular, the columns correspond to a scanning direction of the laser scanner.

[0047] The controlled exposure of the pixel groups one after the other can be achieved, for example, by an appropriate aperture mechanism, for example a mechanical aperture, but preferably an electronic aperture.

[0048] According to some embodiments, during each of the subintervals, a respective part of the point cloud is generated by emitting a respective portion of the laser pulses into the environment and detecting the reflected portions of the emitted portion of the laser pulses by the respective pixel group.

[0049] In other words, both the image and the point cloud are generated step by step, one pixel group at a time. During a given sub-interval, the same pixel group is used to generate a corresponding portion of the point cloud and a corresponding portion of the image based on the ambient light. This ensures that the image matches the points of the point cloud very accurately.

[0050] According to some embodiments, the portion of the emitted laser pulses is emitted into the environment after the respective pixel group has been exposed to detect the ambient light.

[0051] In particular, the corresponding data for generating the image part and the point cloud part using the respective pixel group are not acquired at the same time, but one after the other. Consequently, the image and the point cloud are generated based on independent data, so that the image is not influenced by the reflected portions of the emitted laser pulses.

[0052] In alternative embodiments, the respective group of pixels is illuminated to detect the ambient light after the portion of the emitted laser pulses is emitted into the environment and after the reflected portions of the portion of the emitted laser pulses are detected by the respective pixel group.

[0053] According to some embodiments, the plurality of detector pixels are arranged according to a plurality of columns and rows of the detector array, and each of the pixel groups corresponds to one of the plurality of columns or the at least two adjacent columns of the plurality of columns.

[0054] According to some embodiments, the images are generated as respective monochromatic images.

[0055] According to a further aspect of the invention, a method for at least partially automatically guiding a motor vehicle is provided. A method according to the invention for operating a lidar sensor system of a motor vehicle is performed. At least one control signal for at least partially automatically guiding the motor vehicle is generated, in particular by at least one processing unit, for example, of the motor vehicle, and / or assistance information for assisting a driver of the motor vehicle is generated, in particular by the at least one processing unit.

[0056] The at least one control signal can be provided, for example, to one or more actuators of the motor vehicle, including, for example, one or more brake actuators and / or one or more steering actuators and / or one or more drive motors of the motor vehicle. The one or more actuators can influence a longitudinal and / or lateral control of the motor vehicle in order to guide the motor vehicle at least partially automatically.

[0057] The assistance information can be output by means of an output device of the motor vehicle, for example a display and / or an audio output system and / or a haptic output system.

[0058] A computing unit can be understood, in particular, as a data processing device that has processing circuits. The computing unit can therefore, in particular, process data to perform computing operations. This can also include operations for performing indexed access to a data structure, for example, a look-up table (LUT).

[0059] A computing unit can, in particular, contain one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), and / or one or more single-chip systems (SoCs). The computing unit can also contain one or more processors, for example one or more microprocessors, one or more central processing units (CPUs), one or more graphics processing units (GPUs), and / or one or more signal processors, in particular one or more digital signal processors (DSPs). The computing unit can also include a physical or virtual network of computers or other of the aforementioned units.

[0060] In various embodiments, the computing unit includes one or more hardware and / or software interfaces and / or one or more memory units.

[0061] A memory unit can be a volatile data memory, for example a dynamic random access memory (DRAM) or a static random access memory (SRAM), or a non-volatile data memory, for example a read-only memory (ROM), a programmable read-only memory (PROM), an erasable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), a flash memory or flash EEPROM, a ferroelectric random access memory (FRAM), a magnetoresistive random access memory (MRAM), or a phase-change random access memory.PCRAM (phase-change random access memory) is implemented.

[0062] All computing units of the at least one computing unit can be contained in the motor vehicle. However, it is also possible for all computing units of the at least one computing unit to be part of an external computing system external to the vehicle, for example, a backend server or a cloud computing system. It is also possible for the at least one computing unit to comprise at least one vehicle computing unit of the motor vehicle and at least one external computing unit contained in the external computing system. The at least one vehicle computing unit can, for example, be contained in one or more electronic control units (ECUs) and / or one or more zone control units (ZCUs) and / or one or more domain control units (DCUs) of the motor vehicle and / or in the lidar sensor system.

[0063] According to a further aspect of the invention, a lidar sensor system, in particular a laser scanner, is provided for a motor vehicle. The lidar sensor system includes a transmitting unit and a detector array configured to generate detector signals depending on ambient light incident on the detector array during each of the two frame intervals. The lidar sensor system includes at least one control and / or evaluation unit configured to generate a respective image of the surroundings of the motor vehicle for each of the at least two frame intervals depending on the respective detector signals.The at least one control and / or evaluation unit is configured to control the transmitting unit to emit laser pulses into an environment of the motor vehicle during a final frame interval of the at least two frame intervals, and the detector array is configured to generate further detector signals depending on reflected portions of the emitted laser pulses that impinge on the detector array during the respective frame interval.

[0064] The at least one control and / or evaluation unit is configured to generate a point cloud depending on the further detector signals, to detect a static object in each of the images and to determine a pose change of the lidar sensor system relative to the motor vehicle depending on the static object and to correct the point cloud by applying a geometric transformation to the point cloud, which includes a translation depending on the pose change and / or a rotation depending on the pose change.

[0065] The at least one control and / or evaluation unit can in particular contain one or more computing units or be contained by one or more computing units according to the above definition.

[0066] According to some versions, the lidar sensor system is a laser scanner.

[0067] According to some embodiments, the plurality of detector pixels are arranged according to a plurality of columns and rows of the detector array. A total number of columns of the detector array is at least 100 and / or a total number of rows of the detector array is at least 100.

[0068] Further embodiments of the lidar sensor system according to the invention result directly from the various embodiments of the method according to the invention, and vice versa. In particular, individual features and corresponding explanations as well as advantages relating to the various embodiments of the method according to the invention can be transferred accordingly to respective embodiments of the lidar sensor system according to the invention. In particular, the lidar sensor system according to the invention is designed or programmed to carry out a method according to the invention. In particular, the lidar sensor system according to the invention carries out a method according to the invention.

[0069] According to a further aspect of the invention, an electronic vehicle guidance system for a motor vehicle is provided. The electronic vehicle guidance system includes a lidar sensor system according to the invention and at least one computing unit configured to generate, depending on the corrected point cloud, at least one control signal for at least partially automatically guiding a motor vehicle and / or assistance information for assisting a driver of the motor vehicle.

[0070] An electronic vehicle guidance system can be understood as an electronic system designed to guide a vehicle fully automatically or fully autonomously, in particular without requiring manual intervention or control by a driver or user of the vehicle. The vehicle automatically performs all required functions, such as steering maneuvers, braking maneuvers, and / or acceleration maneuvers, as well as monitoring and recording road traffic and corresponding reactions. In particular, the electronic vehicle guidance system can implement a fully automatic or fully autonomous driving mode according to Level 5 of the SAE J3016 classification. An electronic vehicle guidance system can also be implemented as a driver assistance system (ADAS), which assists a driver in partially automatic or partially autonomous driving.In particular, the electronic vehicle guidance system can implement a semi-automatic or semi-autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification. Here and below, SAE J3016 refers to the corresponding standard dated April 2021.

[0071] The at least partially automatic driving of the vehicle can therefore include driving the vehicle according to a fully automatic or fully autonomous driving mode according to Level 5 of the SAE J3016 classification. The at least partially automatic driving of the vehicle can also include driving the vehicle according to a partially automatic or partially autonomous driving mode according to Levels 1 to 4 of the SAE J3016 classification.

[0072] According to a further aspect of the invention, a computer program containing instructions is provided. When the instructions are executed by a lidar sensor system according to the invention, in particular by the at least one control and / or evaluation unit, the instructions cause the lidar sensor system to execute a method according to the invention for operating a lidar sensor system of a motor vehicle.

[0073] The instructions can be provided, for example, as program code. The program code can be provided, for example, as binary code or assembly code and / or as source code of a programming language, for example, C, and / or as a program script, for example, Python.

[0074] According to a further aspect of the invention, a further computer program containing further instructions is provided. When the further instructions are executed by an electronic vehicle guidance system according to the invention, in particular by the at least one control and / or evaluation unit and / or at least one computing unit, the instructions cause the electronic vehicle guidance system to carry out a method according to the invention for at least partially automatically guiding a motor vehicle.

[0075] The additional instructions can be provided, for example, as program code. The program code can be provided, for example, as binary code or assembly code and / or as source code of a programming language, for example, C, and / or as a program script, for example, Python.

[0076] According to a further aspect of the invention, a computer-readable storage medium is provided which stores a computer program according to the invention.

[0077] The computer program, the further computer program and the computer-readable storage medium are respective computer program products with the instructions.

[0078] Further features of the invention emerge from the claims, the figures and the description of the figures. The features and combinations of features mentioned above in the description as well as the features and combinations of features mentioned below in the description of the figures and / or shown in the figures can be encompassed by the invention not only in the respectively specified combination, but also in other combinations. In particular, embodiments and combinations of features can also be encompassed by the invention which do not have all of the features of an originally formulated claim. Furthermore, embodiments and combinations of features can be encompassed by the invention which go beyond or deviate from the combinations of features set out in the backreferences of the claims.

[0079] The invention is explained in detail below using specific exemplary embodiments and corresponding schematic drawings. In the drawings, identical or functionally equivalent elements may be designated by the same reference numerals. The description of identical or functionally equivalent elements is not necessarily repeated with reference to different figures.

[0080] The figures show: Fig. 1 schematically shows a motor vehicle with an exemplary embodiment of an electronic vehicle guidance system according to the invention; and Fig. 2 a schematic flow diagram of an exemplary embodiment of a method according to the invention for operating a lidar sensor system.

[0081] Fig. 1 schematically shows a motor vehicle 1 with an exemplary embodiment of an electronic vehicle guidance system 2 according to the invention. The electronic vehicle guidance system 2 includes an exemplary embodiment of a lidar sensor system 3 according to the invention and at least one computing unit 4.

[0082] The lidar sensor system 3 can generate a point cloud representing the surroundings of the motor vehicle 1 by emitting laser pulses into the surroundings and detecting portions of the emitted laser pulses that are reflected by objects in the surroundings using a detector array with a plurality of detector pixels. In addition to the detector array, the lidar sensor system 3 has a transmitting unit and at least one control and / or evaluation unit configured to control the transmitting unit to transmit the laser pulses into the surroundings of the motor vehicle 1 for at least a final frame interval of at least two frame intervals. The detector array is configured to generate detector signals depending on the reflected portions of the emitted laser pulses that impinge on the detector array.Furthermore, the detector array is configured to generate additional detector signals depending on ambient light that strikes the detector array during each of the two or more frame intervals.

[0083] The at least one control and / or evaluation unit is configured to generate a point cloud for the final frame interval depending on the detector signals and, as in Fig. 2, to generate a respective image 5 of the environment depending on the further detector signals.

[0084] The at least one control and / or evaluation unit is configured to detect a static object 6, 7, 8 in each of the images 5 and to determine a pose change of the lidar sensor system 3 relative to the motor vehicle 1 depending on the static object 6, 7, 8. The at least one control and / or evaluation unit is configured to correct the point cloud by applying a geometric transformation to the point cloud, which includes a translation depending on the pose change and / or a rotation depending on the pose change.

[0085] Depending on the corrected point cloud, the at least one computing unit 4 can generate at least one control signal for at least partially automatically driving the motor vehicle 1 and / or assistance information for assisting a driver of the motor vehicle 1.

[0086] In particular, the lidar sensor system 3 can carry out a method according to the invention. A schematic flow diagram of an exemplary embodiment of such a method is shown schematically in Fig. 2 shown.

[0087] In step 200, for each of the at least two frame intervals, a respective image 5 of the environment is generated by detecting the ambient light incident on the detector array during the respective frame interval, and for the final frame interval, a point cloud is generated by emitting laser pulses into the environment and detecting reflected portions of the emitted laser pulses by the detector array.

[0088] In step 210, the at least one control and / or evaluation unit applies at least one computer vision algorithm, for example, an object detection algorithm that yields bounding boxes 10, 11, and / or a semantic segmentation algorithm to the image 5 of an initial frame of the two or more frames. In step 220, the at least one control and / or evaluation unit checks whether a static object 6, 7, 8, for example, a road surface 6, a traffic sign 7, or traffic lights 8, can be identified in the image 5 based on the result of the at least one computer vision algorithm. Depending on the actual embodiment, it is possible that only static objects of certain object classes are used, but not every static object.If no such static object 6, 7, 8 is found, then the at least one computing unit 4 applies the at least one computer vision algorithm to the image 5 of the next frame interval and so on until the said static object 6, 7, 8 is found.

[0089] In optional step 230, inertial sensor data 9 is received from an inertial sensor system of the motor vehicle 1. It is advantageous that the inertial sensor system can be arranged in the motor vehicle 1 and / or in the lidar sensor system 3. In step 240, a change in the pose of the lidar sensor system relative to the motor vehicle is determined by the at least one control and / or evaluation unit depending on the static object 6, 7, 8 and the positions and / or orientations determined for the at least two frames.

[0090] For example, the image position and / or an image orientation of the static object 6, 7, 8 in the respective image 5 is determined for the final frame and for a further frame interval, and the pose change of the static object 6, 7, 8 is determined based on a comparison of the determined image positions and / or image orientations. The pose change can be validated, for example, using the inertial sensor data 9. In step 250, the at least one control and / or evaluation unit corrects the point cloud as described.

[0091] As described in particular with reference to the figures, the invention provides an approach for at least partially correcting a point cloud of a lidar sensor system in the event of a pose change of the lidar sensor system relative to the motor vehicle. For this purpose, images based on ambient light are generated by the same detector array used to generate the point cloud. The inventive solution offers a real-time solution using a single environmental sensor and only one point cloud for a single frame interval. This eliminates the need to process the point clouds of multiple frame intervals.

Claims

[1] Method for operating a lidar sensor system (3) of a motor vehicle (1), wherein - for each of at least two frame intervals, including a final frame interval, a respective image (5) of an environment of the motor vehicle (1) is generated by detecting ambient light incident on a detector array of the lidar sensor system (3); - a point cloud is generated for the final frame interval by emitting laser pulses into the environment and detecting reflected portions of the emitted laser pulses by the detector array; - a static object (6, 7, 8) is detected in each of the images (5) and a change in pose of the lidar sensor system (3) relative to the motor vehicle (1) is determined as a function of the static object (6, 7, 8); - the point cloud is corrected by applying a geometric transformation to the point cloud that includes a translation dependent on the pose change and / or a rotation dependent on the pose change. [2] The method according to claim 1, wherein inertial sensor data (9) is generated by an inertial sensor system of the lidar sensor system (3) and the determination of the pose change depending on the static object (6, 7, 8) is triggered when a movement of the detector array is detected based on the inertial sensor data (9). [3] Method according to claim 2, wherein the pose change determined as a function of the static object (6, 7, 8) is validated as a function of the inertial sensor data (9) and the point cloud is corrected as a function of a result of the validation. [4] Method according to one of claims 2 or 3, wherein the inertial sensor data (9) correspond to a respective acceleration and / or rotation of the detector array for at least two measuring intervals of the inertial sensor system. [5] The method according to claim 4, wherein a total measurement duration of the at least two measurement intervals is shorter than a frame duration of one of the at least two frame intervals. [6] Method according to one of claims 2 to 5, wherein the lidar sensor system (3) comprises a housing, wherein the detector array and the inertial sensor system are arranged in the housing. [7] Method according to one of the preceding claims, wherein - an image position and / or an image orientation of the static object (6, 7, 8) in the respective image (5) is determined for the final frame and for a further frame interval of the at least two frame intervals; and - the pose change of the static object (6, 7, 8) is determined based on a comparison of the determined image positions and / or image orientations. [8] Method according to one of claims 1 to 6, wherein an object tracking algorithm is applied to the respective images (5) of the at least two frame intervals to track the static object (6, 7, 8), and the pose change of the static object (6, 7, 8) is determined based on a result of the tracking algorithm. [9] Method according to one of the preceding claims, wherein - a plurality of detector pixels of the detector array are arranged according to a plurality of pixel groups which are exposed to detect the ambient light one after the other during respective successive sub-intervals of the final frame interval to generate the respective image (5); and - during each of the subintervals, a respective part of the point cloud is generated by emitting a respective portion of the laser pulses into the environment and detecting the reflected portions of the emitted portion of the laser pulses by the respective pixel group. [10] The method of claim 9, wherein - the proportion of the emitted laser pulses is emitted into the environment after the respective group of pixels has been illuminated in order to detect the ambient light; or - the respective group of pixels is illuminated to detect the ambient light after the portion of the emitted laser pulses is emitted into the environment and after the reflected portions of the portion of the emitted laser pulses are detected by the respective pixel group. [11] A method according to any one of claims 9 or 10, wherein the plurality of detector pixels are arranged according to a plurality of columns and rows of the detector array and each of the pixel groups corresponds to one of the plurality of columns or at least two adjacent columns of the plurality of columns. [12] Method according to one of the preceding claims, wherein the image (5) is generated as a monochromatic image (5). [13] Method for at least partially automatically driving a motor vehicle (1), wherein a method according to one of the preceding claims is carried out and depending on the point cloud - at least one control signal is generated for at least partially controlling the motor vehicle (1); and / or - assistance information for assisting a driver of the motor vehicle (1) is generated. [14] Lidar sensor system (3) for a motor vehicle (1), which comprises a transmitting unit and a detector array which is designed to generate detector signals depending on ambient light which strikes the detector array during each of the at least two frame intervals, wherein - the lidar sensor system (3) includes at least one control and / or evaluation unit which is configured to generate a respective image (5) of an environment of the motor vehicle (1) for each of the at least two frame intervals depending on the respective detector signals; - the at least one control and / or evaluation unit is configured to control the transmitting unit to emit laser pulses into the surroundings of the motor vehicle (1) during a final frame interval of the at least two frame intervals, and the detector array is configured to generate further detector signals depending on reflected portions of the emitted laser pulses that impinge on the detector array; and - at least one control and / or evaluation unit is designed to - to generate a point cloud depending on the other detector signals; - detecting a static object (6, 7, 8) in each of the images (5) and determining a change in the pose of the lidar sensor system (3) relative to the motor vehicle (1) depending on the static object (6, 7, 8); and - correct the point cloud by applying a geometric transformation to the point cloud that includes a translation dependent on the pose change and / or a rotation dependent on the pose change. [15] Lidar sensor system (3) according to claim 14, wherein the lidar sensor system (3) is a laser scanner. [16] Lidar sensor system (3) according to one of claims 14 or 15, wherein the detector array comprises a plurality of detector pixels arranged according to a plurality of columns and rows of the detector array, and a total number of columns of the detector array is at least 100 and / or a total number of rows of the detector array is at least 100. [17] Electronic vehicle guidance system (2) for a motor vehicle (1), comprising a lidar sensor system (3) according to one of claims 14 to 16 and at least one computing unit (4) which is configured to, depending on the corrected point cloud - to generate at least one control signal for at least partially automatically driving the motor vehicle (1); and / or - to generate assistance information for assisting a driver of the motor vehicle (1). [18] Computer program product that - includes commands which, when executed by a lidar sensor system (3) according to any one of claims 14 to 16, cause the lidar sensor system (3) to perform a method according to any one of claims 1 to 12; and / or - further instructions which, when executed by an electronic vehicle guidance system (2) according to claim 17, cause the electronic vehicle guidance system (2) to carry out a method according to claim 13.

Images