Method and apparatus for classifying objects

Abstract

A method (100) for classifying objects is proposed, the method comprising extracting (107) features from measurement data of one or more receiving elements of a sensor (11) by a neural network (14a) or a Gaussian mixture model (GMM; 14b), wherein the measurement data of each of at least one receiving element of the sensor (11) is related to at least a portion of a photon histogram (40), and the neural network (14a) comprises a fully connected neural network (FCN) or a convolutional neural network (CNN).

Description

[Technical field]

[0001] The invention relates to a method and a device for classifying objects according to the independent claims. [Background technology]

[0002] It is known from the prior art to perform optical distance measurements using Lidar sensors. These are based on the so-called time-of-flight principle. In the process, a scanning sensor is used which periodically emits a pulse. The pulse is reflected by an object and the reflected pulse is detected. By determining the transit time of the pulse from the sensor to the object and back, the distance to said object can be estimated using the speed of light. The object may relate to an object, for example a person or an object, or to an environmental condition, for example fog. In the prior art, the goal is always to detect the object reliably.

[0003] To detect objects and estimate distances here, classifiers are usually trained using sensor data. This results in a position-dependent estimation of the object. The disadvantage of known signal processing chains is the high computational load required for the individual processing steps. Since the complexity of the problem is by its very nature higher here, a special computational load is introduced by the operations in 3D.

[0004] Therefore, current solutions use simplified models and filter approaches to filter out noise and report only valid detections. However, this introduces the problem that pulses are reflected by both objects and environmental conditions, such as rain or fog. Prior art systems are not suitable to distinguish between valid object reflections and corresponding environmental condition reflections, so false positives can occur. This can then lead to potentially dangerous reactions of vehicles operating autonomously or semi-autonomously based on this information.

[0005] US Pat. No. 5,399,433 describes a LIDAR system for use in vehicles. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] US Patent Application Publication No. 2019 / 324147 Summary of the Invention

[0007] As a result, the object of the present invention is to improve a method and an apparatus for classifying objects in such a way that it reduces the computational load and allows for semantic segmentation or object detection of the vehicle's environment, in particular with the aim of identifying whether an object relates to an object or to an environmental condition.

[0008] The object is achieved by a method for classifying objects, comprising extracting features from measurement data of one or more receiving elements of a sensor by a neural network or a Gaussian mixture model (GMM). The measurement data of each of at least one receiving element of the sensor relates to at least a portion of a photon histogram. The neural network comprises a fully connected neural network (FCN) or a convolutional neural network (CNN). Moreover, the above steps can be performed for all receiving elements of the sensor.

[0009] In particular, the neuronal network of the GMM is designed to detect patterns in the measurement data. Patterns should be understood as meaning in particular rising parts, so-called peaks. In particular, the neuronal network or GMM can analyze the shape of the peaks. The GMM or neuronal network is used in particular to model and thus evaluate the corresponding distribution underlying the rising parts. For example, the neuronal network or GMM can detect the rising and / or falling flanks and their gradient and / or their curvature, for example exponentially. The width of the peaks is also detected.

[0010] In particular, the FCN comprises at least one fully connected layer. Each layer comprises a number of artificial neurons. In particular, the FCN models a nonlinear function. The FCN is designed to directly detect the corresponding nonlinear function in at least a portion of the photon histogram and thus in the raw data of the sensor. In other words, the FCN attempts to detect the nonlinear function based on the input values, in particular the values ​​of the photon histogram, that correspond to the desired output values ​​of the nonlinear function. In particular, the FCN comprises at least one fully connected layer, in particular a number of fully connected layers. In particular, the FCN comprises at least 5 layers, preferably at least 10 layers, in particular at least 15 layers.

[0011] In particular, the CNN comprises at least one convolutional layer. Each layer comprises a number of artificial neurons. In particular, each layer of the CNN may be designed as a convolutional layer. In particular, the convolutional layer performs convolution. In particular, the CNN comprises at least two, preferably at least three, in particular at least five convolutional layers. Each of all layers (preferably except the final layer) is activated, in particular, by a nonlinear function, in particular the so-called ReLU function. The ReLU function here comprises a rectified linear unit. In other words, it comprises a function that is linearly designed in the positive X domain but 0 in the negative X domain, with X as the input value of the neuron. The final layer preferably does not have an activation function, so as to allow all numerical values ​​as the final step.

[0012] Furthermore, the CNN may include one or more pooling layers. In particular, a pooling layer is placed after each convolution layer. The pooling layer is used to reduce the dimension of the output transmitted to the subsequent layer. In particular, the pooling layer is designed to perform "max pooling", in other words, to always take only the maximum value of multiple output values ​​and transmit it to the subsequent layer.

[0013] Furthermore, the CNN may comprise at least one, in particular two, fully connected layers. The CNN may also comprise at least one dropout layer. In particular, the network is constructed such that the dropout layer is placed between the two fully connected layers.

[0014] FCN and CNN are easy to implement, as the features extracted from CNN are invariant to translations and rotations, i.e., slight displacements or rotations that may affect the input photon histogram are not destructive.

[0015] In particular, the GMM is a function of various functions. At least one, some or all of these functions may be Gaussian functions here. The GMM is used to model and thus evaluate the peak shape. In other words, the GMM identifies an analytical description of the histogram. In particular, the GMM is fitted without supervision. The GMM can be very easily combined with any kind of classifier.

[0016] In particular, the one or more receiving elements may include an avalanche photodetector, for example a single photon avalanche diode SPAD.

[0017] In particular, the photon histogram is designed as a one-dimensional array. In other words, it contains values, in other words numbers, which are recorded in so-called bins. A bin is a portion of the previously defined measurement interval. In particular, the values ​​of the photon histogram are the amount of detected photons in each bin.

[0018] In particular, the sensor includes a solid-state Lidar sensor, and the method includes performing optical transit time measurements for at least one defined measurement interval by the sensor. The method includes generating a photon histogram for each measurement interval based on the optical transit time measurements.

[0019] The optical transit time measurement is based on the principle of time of flight. Thus, at least one transmitting element, preferably several transmitting elements, of the transmitting part emits at least one measurement pulse, and each transmitting element of the transmitting part is assigned to a respective receiving element of the receiving elements. Furthermore, each pair of transmitting and receiving elements is assigned to a sector of the field of view of the sensor. As a result, the field of view can be divided into different sectors based on the transmitting and receiving elements. Thus, the principle of time of flight and the local assignment of each receiving element to a sector of the field of view described above can be used to determine the distance to a reflection point of an object from which a pulse is reflected, and the position of the reflection point.

[0020] Here, it is not necessary that all transmitting elements emit at least one measurement pulse within the measurement interval. For example, the area of ​​the transmitter and thus the target area, target region of the field of view, where a group of transmitting elements emits at least one transmission pulse, can be measured, such that only a single group of receiving elements receives the corresponding reflected pulse. The definition of the measurement interval is determined, in particular, by the so-called frame rate. As a result, at least a part of the field of view is illuminated within the measurement interval. Thus, a single image, in other words a frame, of at least a part of the field of view can occur within the measurement interval.

[0021] In particular, generating the photon histogram includes time-correlated single photon counting. In particular, a photon histogram is generated for each measurement interval and for each receiver. Thus, the measurement data provided to the neuronal network or GMM does not include measurement data observing reflections from the field of view of the sensor for time spans exceeding the measurement interval.

[0022] In particular, the individual detection values ​​of photons of the receiving elements are accumulated over the measurement interval defined above for the time until the measurement interval ends. This is carried out in particular for each receiving element.

[0023] The principle of time of flight is used to determine the distance to the corresponding reflection point for each detected photon. The measurement interval is now divided into different parts (bins) of preferably equal length, which represent different distance ranges to the sensor based on the principle of time of flight. The determined distances of the detected photons are now recorded in the bins. Thus, a detection number in each bin is assigned to each bin (i.e. based on reflections at reflection points within the assigned distance range).

[0024] The photon histogram represents the distribution of detection values ​​over the time of the measurement interval or over the distance of the removal area covered by the measurement interval. The structure of the distribution depends on the actual distance of the detection object, the object itself, e.g. the reflectivity of its surface, and the angle of the object. The photon histogram represents the value of the detection values ​​for the individual bins. The photon histogram is obtained in the manner described above for each receiving element that has received at least one measurement pulse. A sector is assigned to each receiving element in relation to the field of view, so that a histogram with 3D position information about the field of view can be determined using the distance information obtained in the respective photon histogram. The object mainly relates to an object or an environmental condition.

[0025] Apart from the transmitter, the sensor also comprises a receiver comprising receiving elements. The transmitter is arranged to emit pulses periodically or randomly, which are reflected by objects in the field of view of the sensor. The transmitter mainly comprises a transmission matrix, and the receiver in particular a reception matrix. The transmission and reception elements can here be regularly arranged in a regular pattern in a corresponding matrix, in particular in a uniform grid. In particular, the reception matrix is ​​designed as a SPAD array. SPADs are semiconductor diodes that use the effect of avalanche breakdown. SPADs have very high sensitivity and high time resolution. Areas of high electric field strength can be generated in the avalanche diode, and the number of free charge carriers in the conduction or valence band is multiplied by impact ionization, exceeding the breakdown voltage and causing an avalanche. This allows single photons to be detected with a small measurement volume.

[0026] As a result, the photon histogram contains the raw data of the sensor. Much more information can be obtained by directly processing the photon histogram. For example, specific areas of interest (regions of interest) and objects, especially objects, can be detected which are only detectable at very high loads even with known methods. The application of neuronal networks or GMMs allows the existing information content to be used more efficiently. This results in a more accurate and comprehensive impression of the field of view of the sensor.

[0027] In particular, the sensor is arranged in the vehicle and the method serves to detect and classify objects located in the vehicle's environment. This brings more advantageous effects to the vehicle environment, reducing the computational load to enable semantic segmentation of the environment, in particular object or object detection. The term "semantic segmentation" should be understood to mean that the field of view can be segmented into areas that contain objects. The object points and their distances are identified not only based on measurements on a single point, but also taking into account measurements of neighboring points.

[0028] In particular, neuronal networks or GMMs can be implemented in the sensor ASIC, which already allows segmentation and / or pre-processing for subsequent ADAS (advanced driver-assistance systems) algorithms in the sensor ASIC.

[0029] In particular, for CNNs, the method may include converting the histogram from a one-dimensional array to a 2D image. The one-dimensional array is specifically converted here to a 2D representation. In other words, the values ​​assigned to corresponding bins are not represented as numbers, but are illustrated as "columns" that correspond to those numbers. This results in a kind of column chart with no separating lines between the individual columns or bins, which represents the 2D image.

[0030] CNNs can process inputs in the form of 2D images. In other words, CNNs model visible features, such as lines, edges, corners, etc., and recognize them in 2D images. CNNs are designed to extract simple features, such as lines, in any orientation, in the first layer; the deeper the network, the more complex the features extracted, such as L-shaped features, circular features, star-shaped features, or higher-dimensional features. One particular advantage of CNNs is that they are invariant to slight misalignments that affect two-dimensional images. Moreover, CNNs can also process inputs in the form of one-dimensional arrays.

[0031] Furthermore, CNNs can also take into account neighbor correlations. This allows semantic segmentation. In other words, CNNs can be understood as filter banks implementing 2D filters. These filters are slid over the 2D image, thereby performing a folding process. CNNs operate on the entire 2D image, but can take neighbor correlations into account. As a result, CNNs only consider neighboring pixels, and do not link any pixels to each other. One particular advantageous effect of CNNs is that they can directly model geometric features. Since correlations exist between adjacent receiving elements, more robust detection and classification of objects, mainly objects, can be achieved. Since an attempt is made to adjust the function over the peaks, the GMM also operates over the peaks, so that neighbor correlations are taken into account here as well.

[0032] In particular, the method comprises the provision of measurement data to a neuronal network or a GMM. The method can preferably be used to provide only at least a portion of the photon histogram, the distance comprising only one peak. As a result, the method can comprise the selection of at least a portion of the photon histogram for each measurement interval and for each receiving element. The at least a portion is selected here to comprise only one peak. In particular, a portion is provided for each peak. In other words, the information of the photon histogram is reduced to the portion representing the peak, and thus reduced to its essential elements. Alternatively, the entire photon histogram can be provided to the neuronal network or the GMM.

[0033] If only a portion of the photon histogram is provided with the peaks, the complexity of the network or GMM can be reduced since they will be processing less information, or in other words, information can be identified from the peaks that may not have been previously extracted by conventional signal processing, which may entail a higher computational load.

[0034] In particular, the output of a neuronal network or a GMM contains features, in particular in the form of tensors, i.e. multidimensional vectors. Furthermore, features associated with a GMM can be present as features of the corresponding used functions, e.g. Gaussian functions, e.g. the mean or standard deviation.

[0035] In particular, the extracted features are indicative of the properties of the photon histogram, in particular the progression of the photon histogram, mainly the smoothness. Furthermore, the extracted features are indicative of the smoothness, shape, height and / or width of peaks in the photon histogram.

[0036] In particular, the method includes the detection of objects, mainly objects, based on the extracted features. Furthermore, the classification of objects, especially objects, can be based on previously extracted features. In particular, in the case of objects, classification refers to the assignment of at least one object class, especially one previous object class, to a detected object. Examples of object classes include, in particular, humans, bicycles, motorcycles, cars, trucks, trees or road signs. In particular, the classification is performed by a neuronal network, which can be designed as a fully connected neuronal network (FCN). Furthermore, the classification can be performed by simple, linear or non-linear layers.

[0037] The method may include detecting environmental conditions, in particular determining a statement about the environmental conditions, in particular the weather conditions, in a corresponding field of view of a receiving element of the sensor.

[0038] The indication may relate to the presence of the environmental condition, the type of environmental condition and / or the intensity of the environmental condition.

[0039] The representation may relate to the fundamental presence of certain environmental conditions. The method may distinguish between an increase and an increase, in particular whether the increase is due to reflections on real objects or whether it results from reflections on environmental conditions, for example fog, snowfall or rainfall. What is relevant here are the environmental conditions in the immediate environment of the vehicle.

[0040] Furthermore, the representation may relate to which environmental conditions are present, for example, rainfall, snowfall or fog. As a result, the method can classify the environmental conditions, i.e. indicate the type of the environmental condition. Again, type refers to the assignment of at least one environmental condition class, in particular only one environmental condition class, to the detected environmental condition. Examples of classes include fog, snowfall and rainfall.

[0041] In particular, the classification of the environmental condition is made based on the extracted features. For example, the features may indicate the progression of a peak. The extracted features may be compared to previously stored profiles of the environmental conditions, in other words previously stored features, allowing the assignment to be made.

[0042] For example, the smoothness of a histogram or peak may be relevant as a feature for classification. The smoothness determines the extent and range of the roughness, in other words the noise, present in the course of the photon histogram or peak, the roughness being significantly smaller than the peak. If there is a very smooth histogram or peak, i.e. very little roughness, then mainly fog can be identified. This is because the corresponding histogram or peak will be very smooth due to the high homogeneity of the fog. By comparison, a histogram or peak that can be traced back to rainfall will be rougher, in other words noisier. This is because the size of the raindrops does not allow homogeneity at the spatial scale relative to the light of the measurement pulse used. Similarly, the roughness of the histogram or peak will be even greater for snowfall.

[0043] As a result, it is possible to estimate whether the lift is due to, for example, fog, snowfall or rainfall, and thus arrive at a representation for the environmental condition. Furthermore, the representation may relate to the intensity of the environmental condition, for example the thickness of the fog or the intensity of the rainfall.

[0044] The distinction between reflections on objects and reflections on environmental conditions and / or the classification of environmental conditions is based on the fact that environmental conditions result in a characteristic shape of a peak for objects and also between environmental conditions. The peak of an environmental condition preferably has a very broad peak at the beginning of the photon histogram and has exponentially declining sides. This is due to the fact that photons are already reflected by particles, e.g. raindrops, very early on.

[0045] As a result, evaluating the shape of the peak makes it possible to classify the environmental condition and thus arrive at a statement about the type of environmental condition. The intensity of the environmental condition also influences the distribution and thus the shape of the peak. Therefore, based on the shape of the peak, a statement can be made about whether the environmental condition is present or not, and more preferably about the type and / or intensity of the environmental condition. It is further preferred if a distance can be assigned to the environmental condition based on its position in the photon histogram.

[0046] Furthermore, noise-based peaks can also be classified. They also have a characteristic shape and are therefore distinguishable from the peaks of objects and environmental conditions. For example, very strong solar illumination results in a noise floor for the entire histogram. Mainly the width and shape of the corresponding peaks are relevant features when distinguishing between objects and noise. Object peaks usually have the width and shape of the measurement pulse, while noise peaks arise in the presence of a given time-invariant solar light and therefore only arise due to statistical variance. As a result, their width is larger than that of object peaks.

[0047] Therefore, the method is particularly suitable for detecting all peaks in the photon histogram and dividing the corresponding objects on which the peaks are based into objects and environmental conditions and classifying them respectively.

[0048] In particular, neuronal networks do not include long short-term memory type networks. LSTMs are much harder to learn than the proposed neuronal networks or GMMs. In particular, they observe sensor data over a period that far exceeds the measurement interval. The disadvantage to this is that if environmental conditions occur as "disturbances" in every instantaneous snapshot, the LSTM does not identify them, but tracks them over time similarly to objects. As a result, the LSTM is unable to extract additional information, e.g., a representation for the environmental conditions. Thus, the LSTM cannot distinguish whether an object is related to an object or an environmental condition. Thus, the LSTM is not robust with respect to disturbances such as environmental conditions that occur regularly in multiple frames, such as fog. The present invention can detect disturbances based on their shape in the histogram, and thus detect and classify repetitive disturbances such as noise or environmental conditions that appear over a longer period of time.

[0049] In particular, a neuronal network is trained or a GMM is fitted. Furthermore, the method may include said training or fitting.

[0050] During training, data is made available to the neuronal network. The data has already been manually evaluated, in other words labeled. And classes have already been assigned to the dataset. This represents the "ground truth". To identify the data, photon histograms occurring in real driving situations can be acquired, collected and manually classified in a manual classification step to generate a set of training data based on the acquired photon histograms and the manually acquired classification. In an additional step, the neuronal network can be trained by a training dataset to generate a neuronal network trained on the corresponding dataset. Since the dataset here relates to objects, the neuronal network or GMM is trained on various object classes. Furthermore, the dataset may also contain data recorded under corresponding environmental conditions, for example according to their type and intensity, so that the neuronal network is also trained on the environmental conditions. The neuronal network or GMM can be trained on environmental conditions, for example to detect rainfall and / or rainfall intensity, to detect fog and / or fog density, and / or to detect snowfall and / or snowfall amount.

[0051] Thus, as a result, a large amount of training data is presented to the neuron network during the training process. The manually performed correspondence assignments, i.e. classifications, can be learned by the neuron network and used by the neuron network during the classification process. The neuron network here learns the feature extraction and classification, i.e. together with the classifier, i.e. in one step. Preferably, the above-mentioned labeled data is provided to the neuron network and the extracted features are transmitted to the classifier. The weights of the neuron network are adjusted based on a comparison between the previously assigned classes of the data set and the generated classes. The training process mainly involves a loss function.

[0052] During the training process of a neuron network, primarily nonlinear functions are learned. In particular, in the first step a forward pass of the current data is performed, in the second step the loss calculation is performed, in the third step the gradient calculation and the backward pass and thus the optimization of the weights used are performed.

[0053] In the case of GMM, the GMM functions are fitted, in other words their properties are adjusted.

[0054] In particular, the GMM is fitted by an expectation-maximization approach. In a first step, an expectation step is performed in which non-existent data points are estimated or predicted. In a second step, a maximization step, the variables of the distribution of the GMM function are adjusted so that they correspond as best as possible to the shape of the distribution, i.e. the peak. The fitting is mainly done unsupervised. The fitted function is then made available to a classifier, which is trained using previously labeled training data.

[0055] Since artificial neural networks (similar to biological neural networks) and GMMs have high pattern recognition capabilities, the trained artificial neural network can easily identify detected objects even under adverse environmental conditions. Furthermore, the neural network may autonomously detect which environmental conditions are dominant and communicate this result.

[0056] Additionally, probabilities for one or more environmental conditions and / or for one or more object classes may also be indicated. For example, multiple object classes can be assigned to an object with corresponding probabilities of how well the corresponding class fits. The same is true for environmental conditions. Multiple possible environmental conditions, e.g., rain, snow, fog, can be assigned to a known environmental condition with corresponding probabilities. For example, it may be output that the environmental condition is most likely to include rain, followed by snow.

[0057] Additionally, the method may identify representations for the object's reflectivity and object angle. These two features are also derived from the peak shape. Learning may also take this information into account.

[0058] In particular, the method includes identifying and assigning a distance to each detected object, primarily to each detected object and / or to each detected environmental condition. For example, a distance may be assigned to an elevation, for example, due to a particular environmental condition. In one example, an object, for example, a vehicle, is located within 100 m, while fog is present within 50 m. The fog triggers a corresponding peak in the photon histogram, which is differentiated, classified, and assigned a distance by the detection of the object based on the method.

[0059] In particular, the method may be used for autonomously or semi-autonomously controlling a vehicle. Furthermore, the method may be applied within the framework of driver assistance for autonomous or semi-autonomous driving. Furthermore, the method may be applied within the framework of traffic monitoring. For example, corresponding devices may here be arranged on bollards.

[0060] In particular, the extraction and / or classification is performed by an evaluation unit, which mainly comprises a neuronal network or a GMM. The evaluation unit may take into account the classified objects and / or the environmental conditions and transmit a corresponding control signal to the control unit of the vehicle. Furthermore, the determined distances for the objects and / or the environmental conditions may be taken into account. For example, if a certain environmental condition is detected, for example in case of dense fog, the fog lights may be automatically activated and / or the driving may be automatically slowed down and / or automatic braking may be performed, for example in case of rainy weather.

[0061] In particular, the control unit comprises a number of modules, for example a control module for a braking system, a control module for a drivetrain, a control module for a steering system of the vehicle and a control module for autonomous driving, As a result, the braking system, the drivetrain and / or the steering system can be controlled based on information of the evaluation unit.

[0062] Accurate environment detection is particularly important for applications in the area of ​​autonomous driving, and the method or apparatus can provide corresponding reliable data, which can act as the basis for corresponding control signals.

[0063] In another aspect, the invention comprises an apparatus for classifying an object, the apparatus comprising a sensor having a number of receiving elements and an evaluator having a neuronal network or a GMM, the neuronal network comprising an FCN or a CNN, the neuronal network or the GMM being designed to extract features from measurement data of one or more receiving elements of the sensor, the measurement data of at least one receiving element of the sensor relating to at least one respective portion of a photon histogram.

[0064] In particular, the device is designed to implement the above-mentioned method. In particular, the evaluation unit comprises a computation unit, for example a hardware accelerator, configured to execute the steps of the above-mentioned method, in particular to extract features, classify objects, mainly objects and environmental conditions, and / or arrive at a representation on the environmental conditions, based on a neuronal network or a GMM. Furthermore, the evaluation unit may comprise a memory and / or an input / output interface.

[0065] In another aspect, the present invention relates to a vehicle comprising the device according to the invention, and preferably the control unit as described above. Furthermore, the vehicle may be equipped with a user interface, which may include, for example, an electronic display for displaying a specified representation of the environmental conditions.

[0066] Furthermore, the present invention comprises a computer program product, which comprises a computer-readable storage medium having stored thereon a program which, when loaded into the memory of a computer, can cause the computer to carry out the above-mentioned method, possibly together with the above-mentioned apparatus.

[0067] Furthermore, the present invention relates to a computer-readable storage medium having stored thereon a program which, when loaded into the memory of a computer, enables the computer to carry out the above-mentioned method as well as the above-mentioned apparatus.

[0068] A purely schematic diagram is shown below: [Brief description of the drawings]

[0069] [Figure 1] FIG. 1 is a process diagram of the method according to the present invention. [Diagram 2] FIG. 2 shows an apparatus according to the present invention. [Diagram 3] FIG. 3 is a block diagram showing the configuration of a vehicle having the device according to the present invention. [Figure 4] FIG. 4 shows a photon histogram as a one-dimensional array. [Diagram 5] FIG. 5 shows a photon histogram as a 2D image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0070] FIG. 1 shows a method 100 for classifying objects according to the present invention.

[0071] The method 100 preferably includes performing 101 an optical transit time measurement by the sensor 11. A photon histogram 40 may be generated 102 in particular based on the optical transit time measurement. To this end, the method may include time-correlated single photon counting 103.

[0072] The method may further include providing 106 measurement data for the neuronal network 14a, in particular the CNN or FCN, or the GMM 14b. The providing 106 may include converting 104 the photon histogram 40 from a one-dimensional array 41 to a 2D image 44. Furthermore, the photon histogram 40 may be provided 106 as a one-dimensional array 41. The providing 106 may include selecting 105 a portion of the photon histogram 40. In particular, the portion comprises a peak 45.

[0073] The method includes extracting 107 features based on the photon histogram 40 by a neuronal network 14a or a GMM 14b. Furthermore, the method may include classifying 108 the objects based on the previously extracted features. Based on the extracted features, the method 100 may include identifying 109 a representation related to the environmental conditions. Furthermore, the method may include identifying and assigning 110 a distance to each detected object based on the extracted features.

[0074] Fig. 2 shows a device 10 according to the invention, which comprises a sensor 11 with a number of receiving elements. The sensor 11 comprises a receiving part 13 with receiving elements and a transmitting part 12 with transmitting elements. Furthermore, the device 10 comprises a neuronal network 14a or a GMM 14b, which are located in the evaluation part 14. The neuronal network 14a or the GMM 14b are designed to extract features from the measurement data of the sensor 11, the measurement data relating to at least a part of the photon histogram 40. The evaluation part 14 further comprises a classifier 14c for classifying objects and environmental conditions based on the extracted features.

[0075] 3 shows a block diagram of the configuration of a vehicle 21 comprising the device 10 according to the invention. In other words, FIG.

[0076] The vehicle 21 includes multiple electronic components interconnected via a vehicle communication network 22. For example, the vehicle communication network 22 can be a standard vehicle communication network built into the vehicle 21, such as a CAN bus.

[0077] The vehicle 21 is equipped with a control unit 15, which comprises a number of modules, in particular a control module 16 for the braking system, a control module 17 for the drivetrain, a control module 19 for the steering system of the vehicle 21, and a control module 18 for autonomous driving.

[0078] The vehicle 21 comprises an apparatus 10, in particular a sensor 11 and an evaluation unit 14, which is able to perform an environment classification that enables a control unit 15 for autonomous driving to take into account the classification results, for example, when making its decisions regarding controlling the vehicle 21. The evaluation unit 14 is designed to transmit control signals to the control unit 15 based on the results of the classification unit 14c.

[0079] The vehicle 21 further comprises a wireless communication interface 23, here in particular a mobile communication interface, which is designed according to the LTE / UMTS standard and enables the evaluation unit 14 to communicate with external services such as cloud services, in particular navigation services, here in particular with an external emergency call centre. For this purpose, for example, the mobile communication interface comprises a subscriber identity module (SIM), by which the vehicle 21 can report to the mobile network so as to enable communication via the mobile network, in particular an emergency call centre (if the mobile network does not enable communication with emergency numbers without SIM authentication).

[0080] Furthermore, the vehicle 21 is equipped with a satellite navigation unit 24 for obtaining the position of the vehicle 21 .

[0081] The vehicle 21 further comprises a user interface 20 that allows an occupant to interact with one or more vehicle systems. The user interface 20 may comprise an electronic display for outputting graphical, symbolic and / or textual content, as well as an input interface for receiving input (e.g., manual input, voice input, and input via gestures, head or eye movements). For example, the input interface may comprise a keyboard, switches, a touch-sensitive screen (touch screen), an eye tracker, etc.

[0082] FIG. 4 shows an exemplary photon histogram 40 acquired from a receiving element and designed as a one-dimensional array 41. It comprises bins 42, here for example 30, in which detection values ​​43 detected within a defined measurement interval 46 are recorded. A distance measurement range can be assigned to the measurement interval 46. The photon histogram 40 is exemplarily specified with a measurement interval 46 of one tenth of a second. The distance measurement range is divided into 30 bins (n=1...30), which are recorded on the x-axis of the histogram and numbered 1 to 30. Each bin corresponds to a predefined distance range. As an example, it is assumed that the distance measurement range covers a distance range of 0 to 90 m and that the bins are set at equal intervals. In this exemplary case, each bin corresponds to a distance range of 3 m.

[0083] The number of detections for the measurement interval 46 is recorded as a number in each bin. For example, 4 detections occur in bin 21, which corresponds for example to a distance of 60-63 m. 6 detections occur in bin 22, 17 detections in bin 23, 30 detections in bin 24 and 6 detections in bin 25. As a result, bins 21-25 define a maximum peak 45 in bin 24, which corresponds here to a distance of 69-72 m. Furthermore, 20 detections occur in bin 7, 21 detections in bin 8 and 24 detections in bin 9. This defines a further peak 45. The neuronal network 14a or the GMM 14b is designed to detect the peaks 45 and evaluate their shape.

[0084] The number of detections per bin indicates the intensity of the reflected signal in the respective distance range. As is evident in the photon histogram 40 of FIG. 4, a first object is located in the field of view of the receiving element, which results in measurements in bins 21-25 with a maximum value in bin 24, so that a distance of the detected object of about 71 m can be estimated. Since peak 45 shows a characteristic shape for an object, it can be concluded that the object is related to the object. Furthermore, based on peak 45 in bins 7-9, it can be concluded that fog is present in the field of view of the receiving element. In other words, the object responsible for peak 45 can be classified as fog. The fog is located at a distance of about 21-27 m. This can be determined from the shape of peak 45. This makes it possible to arrive at a representation for the environmental conditions.

[0085] FIG. 5 shows the photon histogram 40 of FIG. 4 in a similar manner, but in the form of a 2D image 44. Instead of the corresponding detection values ​​in the form of numbers as in FIG. 4, the values ​​at the corresponding bin positions are plotted as columns in the Y direction. As a result, a 2D image 44 is included. Two peaks 45 are again visible. Based on the method, the shape of the peaks can be used to identify that the rear peak 45, which has a maximum in bin 24, is due to reflections on objects, whereas the front peak 45 is due to reflections on fog. Based on the shape of the individual peaks, the detection objects can therefore be classified, even if they relate to environmental conditions.

[0086] Example of a neuron network 1 shows an exemplary architecture of a CNN network that processes one-dimensional input and estimates the distance of an object. Convolutional layer 1: - Input shape: 2016×1 -Initialize: Xavier -Kernel size: 5×1 -Stride: 0 -Padding:Zero padding -Feature Map Out: 2007×16 -Activation:ReLU Max Pooling -Kernel size: 2×1 -Stride: 0 -Output shape: 1003×16 Convolutional layer 2: - Input shape: 1003×16 -Initialize: Xavier -Kernel size: 5×1 -Stride: 0 -Padding:Zero padding -Feature Map Out: 999 x 32 -Activation:ReLU Max Pooling -Kernel size: 2×1 -Stride: 0 -Output shape: 499×32 Convolutional layer 3: -Initialize: Xavier -Kernel size: 5×1 -Stride: 0 -Padding:Zero padding -Feature Map Out: 497x64 -Activation:ReLU Max Pooling -Kernel size: 2×1 -Stride: 0 -Output format: 248x64 ·Fully connected layer: - Input shape: 15872 x 1 -Initialize: Xavier -Size:50 -Activation:ReLU Dropouts: - Dropout rate: 0.3 ·Fully connected layer: - Input shape: 50×1 -Size:1 -Activation: None [Explanation of symbols]

[0087] 100 ways 101 Performing Optical Transit Time Measurements with Sensors 102 Photon Histogram Generation Based on Optical Transit Time Measurements 103 Time-correlated single photon counting 104 Converting a Photon Histogram to a 2D Image 105 Selecting a portion of the photon histogram 106 Providing measurement data for neuronal networks or GMMs 107 Extraction of features from measurement data of one or more receiving elements of a sensor by means of a neural network or GMM 108 Object Classification Based on Previously Extracted Features 109 Identifying expressions related to environmental conditions 110 Identifying and assigning distance to each detected object 10 equipment 11 Sensors 12 Transmitter 13 Receiving section 14 Evaluation Section 14a Neuronal Network 14b GMM 14c Classification section 15 Control section 16 Control module for brake system 17 Control module for drivetrain 18 Control module for autonomous driving 19 Control module for steering system 20 User Interface 21 Vehicles 22 Vehicle Communication Network 23 Wireless communication interface 24 Satellite Navigation Unit 40 Photon Histogram 41 1D Array 42 Bottles 43 Detection value 44 2D Images 45 Peak 46 Measurement Interval

Claims

1. A method (100) for classifying objects, comprising: Extraction (107) of features from measurement data of one or more receiving elements of the sensor (11) by means of a neuronal network (14a) or a Gaussian Mixture Model (GMM; 14b), said measurement data of each of at least one of said receiving elements of said sensor (11) relates to at least a portion of a photon histogram (40), said photon histogram (40) being designed as a one-dimensional array (41); The neuron network (14a) is a fully connected neuron network (FCN) or a convolutional neuron network (CNN), The method (100) may further comprise: providing (106) the measurement data to the neuronal network (14a) or the Gaussian Mixture Model (GMM; 14b); Classifying (108) the object based on the previously extracted features; Including, the measurement data provided are data of a selection of a portion of the photon histogram (40) for each measurement interval and receiving element, one portion being selected for each peak of the photon histogram such that each selected portion includes only one peak (45), Method (100).

2. The sensor (11) is a solid-state Lidar sensor, and the method (100) comprises: performing (101) an optical transit time measurement by said sensor (11) for at least one defined measurement interval (46); generating (102) the photon histogram (40) for each measurement interval (46) based on the optical transit time measurements; The method (100) of claim 1 , comprising:

3. The method (100) includes providing (106) the measurement data to the Gaussian Mixture Model (GMM; 14b); The method (100) according to claim 1 or 2, comprising extracting (107) features from measurement data of one or more receiving elements of the sensor (11) by means of a Gaussian Mixture Model (GMM; 14b).

4. The method (100) of any one of claims 1 to 3, comprising transforming (104) the one-dimensional array (41) of the photon histogram (40) into a 2D image (44).

5. The classification (108) is performed by an additional neuronal network, The method (100) of claim 1, wherein the neuronal network is designed as a fully connected neuronal network (FCN).

6. The method (100) of any one of claims 1 to 5, comprising identifying (109) a representation relating to an environmental condition.

7. The method (100) of claim 6, wherein the representation relates to the presence of the environmental condition, a type of the environmental condition and / or an intensity of the environmental condition.

8. The method (100) of any one of claims 1 to 7, comprising identifying and assigning (110) a distance to each detected object.

9. Once loaded into the computer's memory, providing (106) measurement data of one or more receiving elements of the sensor (11) to a neuronal network (14a) or a Gaussian mixture model (GMM; 14b); Extracting (107) features from the measurement data of the sensor (11) by the neuronal network (14a) or the Gaussian Mixture Model (GMM; 14b); selecting at least a portion of the photon histogram (40) for each measurement interval and receiving element; performing a selection of one portion for each peak of said photon histogram such that each selected portion contains only one peak (45); Classifying (108) the object based on the previously extracted features; A computer-readable storage medium storing a program for causing a computer to execute the above, said measurement data of each of at least one of said receiving elements of said sensor (11) relates to at least a portion of a photon histogram (40), said photon histogram (40) being designed as a one-dimensional array (41); The neuron network (14a) is a fully connected neuron network (FCN) or a convolutional neuron network (CNN). A computer-readable storage medium on which a program is stored.

Images