METHOD AND DEVICE FOR CLASSIFYING TARGETS

Description

[0001] The present invention relates to a method and a device for classifying targets according to the independent claims. State of the art

[0002] It is known from the state of the art to perform optical distance measurements using lidar sensors. These are based on the so-called time-of-flight principle. This involves a scanning sensor that periodically emits pulses. The pulses are reflected by targets, and the reflected pulses are detected. By determining the time of flight of the pulses from the sensor to the target and back, the distance to this target can be determined using the speed of light. The target can be an object, such as people or objects, or an environmental condition, such as fog. In the state of the art, the goal is always to reliably detect objects.

[0003] To perform object detection and distance estimation, classifiers are typically trained with the sensor data. The result is a position-dependent estimation of objects. The disadvantage of the conventional signal processing chain is the high computational effort required for the individual processing steps. Computations in 3D require a particularly high level of computational effort, as the problem complexity is inherently higher here.

[0004] Current solutions therefore use simplified models and filtering approaches to filter out noise and report only valid detections. However, this poses the problem that the pulses are reflected by both objects and environmental conditions such as rain or fog. Because current systems are not capable of distinguishing reflections from valid objects and corresponding environmental conditions, false detections can occur, which can then lead to potentially dangerous reactions from a vehicle driving autonomously or semi-autonomously based on this information.

[0005] US 2019 / 324 147 A1 describes a LIDAR system for use in a vehicle. Description of the invention: task, solution, advantages

[0006] The present invention is therefore based on the object of improving a method and a device for classifying targets in such a way that the computational effort is reduced and semantic segmentation or target recognition of a vehicle's surroundings is enabled. In particular, a distinction should be made between whether a target is an object or an environmental condition.

[0007] The aforementioned problem is solved by a computer-implemented method for classifying targets, which involves extracting features from measurement data from multiple receiving elements of a sensor using a neural network or a Gaussian Mixture Model (GMM). The measurement data from each receiving element of the sensor represents one or more sections of the respective photon histogram. The neural network is a fully connected neural network (FCN) or a convolutional neural network (CNN). Furthermore, the above steps can be performed for all receiving elements of the sensor.

[0008] The neural network or GMM is specifically designed to recognize patterns in the measurement data. A pattern refers, in particular, to increases, so-called peaks. The neural network or GMM is particularly capable of analyzing the shape of peaks. In particular, the GMM or neural networks serve to model and thus evaluate the corresponding distribution underlying an increase. The neural network or GMM can, for example, recognize the rising and / or falling edge and its gradient and / or its curve shape, e.g., exponential. The width of the peak is also recognized.

[0009] The FCN comprises, in particular, at least one fully connected layer. Each layer contains a plurality of artificial neurons. The FCN models, in particular, non-linear functions. The FCN is designed to recognize corresponding non-linear functions directly in the at least one section of the respective photon histogram, and thus in the raw data of the sensor. In other words, the FCN attempts to recognize non-linear functions based on the input values, namely the values ​​of the respective photon histogram, which correspond to the desired output values ​​of the non-linear functions. The FCN comprises, in particular, at least one fully connected layer, in particular, several fully connected layers. In particular, the FCN comprises at least five, preferably at least ten, in particular at least 15, layers.

[0010] The CNN comprises, in particular, at least one convolutional layer. Each layer contains a plurality of artificial neurons. In particular, each layer of the CNN can be designed as a convolutional layer. A convolutional layer performs, in particular, convolution. In particular, the CNN comprises at least two, preferably at least three, in particular at least five, convolutional layers. All layers (except preferably the last layer) are activated primarily by a non-linear function, in particular a so-called ReLU function. The ReLU function is a Rectified Linear Unit. In other words, it is a function that is linear in the positive X range, but 0 in the negative X range, where X represents an input value of a neuron. The last layer preferably has no activation function in order to allow all numerical values ​​as a final step.

[0011] Furthermore, the CNN can include one or more pooling layers. Specifically, a pooling layer is located after each convolutional layer. A pooling layer serves to reduce the dimension of the output passed on to the subsequent layer. Specifically, the pooling layers are designed to perform "max pooling," in other words, they always take only the maximum value from a large number of output values ​​and pass it on to the subsequent layer.

[0012] Furthermore, the CNN can comprise at least one, in particular two, fully connected layers. The CNN can also comprise at least one dropout layer. In particular, the network is constructed such that a dropout layer is arranged between two fully connected layers.

[0013] FCNs and CNNs are easy to implement. The extracted features of a CNN are invariant to transitions and rotations, so slight shifts or rotations that may affect the input, i.e., the respective photon histogram, do not cause interference.

[0014] The GMM is, in particular, a function consisting of various functions. At least one, several, or all of these functions can be Gaussian functions. The GMM is used to model and thus evaluate the peak shape. In other words, the GMM determines an analytical description of the histogram. The GMM is fitted in an unsupervised manner. GMMs can be easily combined with any type of classification unit.

[0015] The multiple receiving elements may in particular be avalanche photodetectors, for example single-photon avalanche diodes (SPADs).

[0016] According to the invention, each photon histogram is designed as a one-dimensional array. In other words, a photon histogram consists of values, or numbers, that are entered into so-called bins. A bin is a section of a previously defined measurement interval. The values ​​of the respective photon histogram are, in particular, the number of detected photons in the respective bin.

[0017] The sensor is in particular a solid-state lidar sensor, wherein the method comprises performing time-of-flight measurements for at least one defined measurement interval by means of the sensor, and wherein the method comprises creating a respective photon histogram per measurement interval based on the time-of-flight measurements.

[0018] Time-of-flight measurements are based on the time-of-flight principle. Thus, at least one transmitting element, preferably a plurality of transmitting elements, of a transmitting unit, preferably a laser, of the sensor emits at least one measurement pulse, with each transmitting element of the transmitting unit being assigned to a receiving element of the receiving element. Furthermore, the pairs of transmitting elements and receiving elements are each assigned to a sector of a field of view of the sensor. The field of view can thus be classified into different sectors based on the transmitting elements and receiving elements. Using the time-of-flight principle and the previously described spatial assignment of each receiving element to a sector of the field of view, a distance to a reflection point of a target at which a pulse was reflected, and the position of the reflection point, can be determined.

[0019] Not all transmitting elements always have to have sent at least one measuring pulse within a measuring interval. For example, a region of interest of the field of view can be measured. For this purpose, an area of ​​the transmitting unit, and thus a group of transmitting elements, emits at least one transmitted pulse, so that only a group of receiving elements receives corresponding reflected pulses. The definition of the measuring interval is determined in particular by the so-called frame rate. Thus, at least a section of the field of view is illuminated within a measuring interval. Thus, an individual image, in other words a frame, of at least a section of the field of view can be created within a measuring interval.

[0020] The creation of each photon histogram specifically involves time-correlated single-photon counting. Specifically, one photon histogram is created per measurement interval and per receiving element. The measurement data provided to the neural network or GMM thus does not include measurement data that observe reflections from the sensor's field of view over a period of time that extends beyond the measurement interval.

[0021] In particular, the individual detections of photons from a receiving element are accumulated over the previously defined measurement interval until the expiration of the measurement interval. This is performed for each receiving element.

[0022] For each detected photon, the distance to the corresponding reflection point is determined using the time-of-flight principle. The measurement interval is divided into different, preferably equally long, sections (bins), which, based on the time-of-flight principle, represent different distance ranges from the sensor. The determined distances of the detected photons are then entered into the bins. Each bin is thus assigned the number of detections in the respective bin (i.e., based on a reflection at a reflection point within the assigned distance range).

[0023] The photon histogram represents a distribution of detections over the time of the measurement interval or over the distance of the distance range covered by the measurement interval. The structure of the distribution depends on the actual distance of the detected target, the target itself (e.g., the reflectivity of its surface), and the angle of the target. The photon histogram represents the detection values ​​relative to the individual bins. In the manner described above, a photon histogram is obtained for each receiving element that has received at least one measurement pulse. Since each receiving element is assigned a sector in relation to the field of view, a histogram with 3D location information for the field of view can be determined using the distance information obtained in the respective photon histogram. A target is primarily an object or an environmental condition.

[0024] In addition to the transmitting unit, the sensor comprises a receiving unit with receiving elements. The transmitting unit is designed to emit pulses either periodically or randomly, which are reflected by a target in the sensor's field of view. The transmitting unit is primarily a transmitting matrix, and the receiving unit is particularly a receiving matrix. The transmitting elements and the receiving elements can be arranged in a regular pattern, for example, a uniform grid, on the corresponding matrices. The receiving matrix is ​​particularly designed as a SPAD array. SPADs are semiconductor diodes that exploit the effect of avalanche breakdown. SPADs exhibit very high sensitivity and high temporal resolution.Avalanche diodes can create regions of high electric field strength in which the number of free charge carriers in the conduction or valence band multiplies through impact ionization and increases avalanche-like above the breakdown voltage. This makes it possible to detect a single photon in a small measurement volume.

[0025] The photon histograms are therefore the raw data from the sensor. By directly processing the photon histograms, greater information gain is achieved. For example, regions of interest and targets, especially objects, can be detected that cannot be identified using conventional methods or can only be identified with significantly greater effort. The use of a neural network or a GMM offers the possibility of using the available information more efficiently. This provides a more accurate and comprehensive impression of the sensor's field of view.

[0026] In particular, the sensor is mounted on a vehicle, and the method is used to detect and classify targets located in the vehicle's surroundings. This provides a more favorable impression of the vehicle's surroundings, while reducing computational effort to enable semantic segmentation, particularly target or object recognition, of the surroundings. The term "semantic segmentation" refers to the ability to segment the field of view into areas containing a target. Points of a target and its distance are determined not only based on the measurement of the single point, but also taking into account measurements of neighboring points.

[0027] In particular, it is possible to implement the neural network or GMM on an ASIC of the sensor. This allows segmentation and / or preprocessing for subsequent ADAS algorithms (ADAS = Advanced Driver Assistance Systems) to be implemented directly on the sensor's ASIC.

[0028] Especially for the CNN, the method can involve converting the histogram from a one-dimensional array into a 2D image. In doing so, the one-dimensional array is specifically converted into a 2D representation. In other words, the values ​​assigned to the corresponding bins are not represented as numerical values, but rather as "columns" according to their numerical value. This results in a type of bar chart without dividing lines between the individual columns or bins, representing a 2D image.

[0029] The CNN is capable of processing input in the form of a 2D image. In other words, the CNN models visual features, such as lines, edges, corners, etc., and recognizes them in the 2D image. The CNN is trained to extract simple features in the first layers, such as lines in any orientation. The deeper the network becomes, the more complex the features extracted become, such as L-features, circle features, star features, or high-dimensional features. A particular advantage of the CNN is that it is invariant to slight displacements that affect the two-dimensional image. Furthermore, the CNN can also process input in the form of a one-dimensional array.

[0030] Furthermore, CNNs can incorporate neighborhood relationships, thus enabling semantic segmentation. In other words, CNNs can be understood as filter banks that implement 2D filters. These filters are placed over the 2D image, thus performing a convolution. The CNN operates on the entire 2D image but can take neighborhood relationships into account. The CNN therefore only considers neighboring pixels and does not link all pixels together. A particular advantage of CNNs is their ability to directly model geometric features. Because there is a relationship between neighboring receiving elements, more robust detection and classification of targets, especially objects, can be achieved. GMMs also operate on the entirety of a peak, attempting to adapt the function to the entire peak, so neighborhood relationships are also taken into account here.

[0031] According to the invention, the method comprises providing the measurement data for the neural network or the GMM, wherein one or more sections of the respective photon histogram are provided by means of the method, wherein each section comprises exactly one peak. The method comprises selecting one section or more sections of the respective photon histogram per measurement interval and per receiving element. The section or sections are selected such that each contains only one peak. In other words, each section consists of a respective peak. A section is provided for each peak. In other words, the information of the respective photon histogram is reduced to the sections that represent peaks, and thus reduced to the essentials.

[0032] By providing only sections of the photon histograms with peaks, the complexity of the network or GMM can be reduced, as they have less information to process. Furthermore, information can be extracted from the peak that could previously not be extracted using classical signal processing, as this requires more computational effort.

[0033] The output of the neural network or GMM is, in particular, features, particularly in the form of tensors, i.e., multidimensional vectors. Furthermore, with regard to the GMM, the features can be present as features of the corresponding functions used, for example, Gaussian functions, such as the mean or standard deviation.

[0034] According to the invention, the extracted features describe shapes of the peaks of the photon histograms.

[0035] In particular, the method comprises the detection of targets, especially objects, based on the extracted features. According to the invention, the method comprises classifying the targets, in particular objects, based on the previously extracted features, namely a shape of the peaks. Classification, in particular in the case of an object, means an assignment of at least one object class, in particular exactly one object class, to a detected object. Examples of object classes are, in particular, a person, bicycle, motorcycle, car, truck, tree or road sign. In particular, the classification is carried out using a neural network, whereby the neural network can be designed as a fully connected neural network (FCN). Furthermore, the classification can be carried out using simple, linear or non-linear layers.

[0036] The method may include detecting an environmental condition. In particular, a statement is made about an environmental condition, in particular a weather condition, within the corresponding field of view of the sensor's receiving element.

[0037] The statement may concern the presence of the environmental condition and / or the type of the environmental condition and / or the strength of the environmental condition.

[0038] The statement may refer to the fact that an environmental condition is present. The method can distinguish between elevations, namely whether an elevation originates from a reflection off an actual object or from a reflection off an environmental condition, such as fog, snow, or rain. These are environmental conditions in the immediate vicinity of the vehicle.

[0039] Furthermore, the statement can concern which environmental condition is present, for example, rain, snow, or fog. Thus, the method can classify the environmental condition, i.e., specify the type of environmental condition. Here, too, the t means an assignment of at least one environmental condition class, in particular exactly one environmental condition class, to a detected environmental condition. Examples of classes are fog, snow, and rain.

[0040] The classification of the environmental condition is based primarily on the extracted features. The extracted features can be compared with a previously stored profile—in other words, previously stored features—of an environmental condition, which allows for the assignment.

[0041] In addition, the smoothness of a histogram can be extracted as a feature. For example, the smoothness of the histogram or peak can be relevant as a feature with regard to classification. Smoothness describes the extent to which unevenness, in other words noise, is present in the course of the respective photon histogram or peak, whereby the unevenness is significantly smaller than the peak. A very smooth histogram or peak, i.e. with very few unevenness, can be used primarily to detect fog, as the corresponding histograms or peaks are very smooth due to the high homogeneity of the fog. In comparison, histograms or peaks that are attributable to rain are more uneven, in other words noisier, because the size of the raindrops prevents homogeneity on the spatial scale of the light of the measurement pulses used. Analogously, the unevenness of the histograms or peaks is even greater with regard to snow.

[0042] This makes it possible to assess whether an elevation was caused by, for example, fog, snow, or rain, and thus make a statement about the environmental conditions. Furthermore, a statement can be made about the severity of the environmental condition, e.g., the density of fog or the intensity of rain.

[0043] The distinction between reflections from an object and reflections from an environmental condition, and / or the classification of an environmental condition, is based on the fact that environmental conditions, unlike objects and also environmental conditions among themselves, cause a characteristic peak shape. Peaks from environmental conditions preferably exhibit a very broad peak at the beginning of the photon histogram with an exponentially declining slope. This is because photons are reflected very early on by particles, such as raindrops.

[0044] Thus, by evaluating the shape of the peak, the environmental condition can be classified and a statement can be made about the type of environmental condition. The intensity of the environmental condition also influences the distribution and thus the shape of the peak. Thus, based on the shape of the peak, it can be determined whether an environmental condition is present and, preferably, the type and / or intensity of the environmental condition. Furthermore, a distance can preferably be assigned to the environmental condition based on its position in the photon histogram.

[0045] Furthermore, peaks based on noise can also be classified. These also have a characteristic shape and can thus be distinguished from peaks from objects and environmental conditions. For example, very strong solar radiation leads to a noise floor that affects the entire histogram. When distinguishing between objects and noise, the width and shape of the corresponding peaks are particularly relevant as features. Peaks from objects typically have the width and shape of a measurement pulse, whereas noise peaks occur in temporally invariant sunlight and therefore arise only from statistical variances. Their width is thus greater than that of object peaks.

[0046] In particular, the method is thus suitable for detecting all peaks of a photon histogram and for dividing the corresponding targets on which the peaks are based into objects and environmental conditions and for classifying them respectively.

[0047] In particular, the neural network is not a long-short-term memory network. LSTMs are much more difficult to train than the proposed neural networks or GMMs. In particular, they observe sensor data over a period of time that extends far beyond a single measurement interval. This has the disadvantage that if an environmental condition occurs as a "disturbance" in all snapshots, the LSTM will not only identify it but also track it over time like an object. Thus, LSTMs are unable to extract further information, such as a statement about an environmental condition. LSTMs cannot distinguish whether a target is an object or an environmental condition. LSTMs are therefore not robust to disturbances, such as environmental conditions that occur regularly across multiple frames, such as fog.The present invention can recognize these based on their shape in the histogram and can thus detect and classify repetitive disturbances, such as noise or environmental conditions that occur over a longer period of time.

[0048] In particular, the neural network is trained or the GMM is fitted. Furthermore, the method can include training or fitting, respectively.

[0049] During training, data is provided to the neural network. The data may already have been manually evaluated, in other words, labeled. A class has then already been assigned to a dataset. This represents the "ground truth." To determine the data, photon histograms occurring in real driving situations can be captured, collected, and manually classified in a manual classification step to generate a set of training data based on the captured photon histograms and the manually obtained classifications. In a further step, the neural network can be trained with the training dataset to generate a neural network trained on the corresponding dataset. The dataset can refer to objects, so that the neural network or GMM is trained with respect to the various object classes.Furthermore, the dataset can also contain data recorded under corresponding environmental conditions, e.g., depending on their type and intensity, so that the neural network is also trained on environmental conditions. The neural network or GMM can be trained on the environmental conditions, for example, to detect rain and / or rain intensity and / or fog and / or fog density and / or snow and / or snow thickness.

[0050] During the training process, a large amount of training data is presented to the neural network. The corresponding assignments, i.e. classifications, which were made manually, are learned by the neural network and can then be used in the classification process by the neural network. In this process, the neural network learns the feature extraction and the classification, i.e. the classification unit, together, i.e. in a single step. The labeled data described above is preferably made available to the neural network, with the extracted features being passed on to the classification unit. The weights of the neural network are adjusted based on a comparison of the previously assigned class of a data set and the generated class. The training process primarily involves a loss function.

[0051] When learning neural networks, nonlinear functions are primarily learned. In a first step, a forward pass of the current data is performed, followed by a loss calculation in a second step. In a third step, a gradient calculation and a backward pass are performed, thus optimizing the weights used.

[0052] In a GMM, the functions of the GMM are fitted, in other words their properties are adjusted.

[0053] The GMM is fitted, in particular, using the Expectation Maximum approach. The first step, an Expectation Step, involves estimating or guessing at nonexistent data points. In a second step, the Maximization Step, the variables are adjusted to the distribution of the GMM functions as best as possible so that they match the distribution, i.e., the shape of a peak. Fitting is primarily unsupervised. The fitted functions are then provided to a classification unit, which is trained using previously labeled training data.

[0054] Since artificial neural networks (like biological neural networks) and the GMM possess high pattern recognition capabilities, a trained artificial neural network can reliably identify detected objects even in adverse environmental conditions. Furthermore, the neural network can independently detect the prevailing environmental conditions and forward this result.

[0055] Furthermore, it is also possible to specify probabilities for one or more environmental conditions and / or for one or more object classes. For example, an object can be assigned multiple object classes, each with a corresponding probability indicating how well the corresponding class applies. The same applies to the environmental condition. A detected environmental condition can be assigned multiple possible environmental condition classes, e.g., rain, snow, fog, with a corresponding probability. For example, it can be output that the environmental condition is most likely rain, and the second most likely snow.

[0056] Furthermore, the method can determine information about the object's reflectivity and its angle. These two features are also derived from the shape of the peaks. This information can also be incorporated into training.

[0057] The method specifically includes determining and assigning a distance to each detected target, especially each detected object and / or each detected environmental condition. For example, a distance can be assigned to an elevation resulting from a specific environmental condition. An example might be that an object, such as a vehicle, is located 100 m away, while fog is located 50 m away. The fog triggered a corresponding peak in the photon histogram, which was differentiated from the detection of an object based on the method, classified, and assigned a distance.

[0058] In particular, the method can be used for autonomous or semi-autonomous vehicle control. Furthermore, the method can be used as part of driver assistance for autonomous or semi-autonomous driving. Furthermore, the method can be used in traffic monitoring. A corresponding device can be arranged, for example, on a bollard.

[0059] The extraction and / or classification are performed, in particular, by an evaluation unit. The evaluation unit primarily comprises the neural network or the GMM. The evaluation unit can forward a corresponding control signal to a vehicle control unit, taking into account the classified objects and / or environmental conditions. Furthermore, the distances determined to the objects and / or environmental conditions can be taken into account. For example, upon detection of a specific environmental condition, such as dense fog, the fog lights can be automatically activated and / or the vehicle can be automatically slowed down, for example, in rain, and / or automatically braked.

[0060] The control unit comprises, in particular, several modules, e.g., a control module for the braking system, a control module for a powertrain, a control module for the vehicle's steering system, and a control module for autonomous driving. Thus, the braking system, the powertrain, and / or the steering system can be controlled based on the information from the evaluation unit.

[0061] Precise environmental detection is particularly important for applications in the field of autonomous driving, whereby the present method or device can provide corresponding reliable data on which corresponding control signals can be based.

[0062] In a further aspect, the invention comprises a device for classifying targets, wherein the device comprises a sensor with a plurality of receiving elements and an evaluation unit with a neural network or a GMM, wherein the neural network is an FCN or CNN. The neural network or the GMM is configured to extract features from measurement data of a plurality of receiving elements of the sensor, wherein the measurement data of the receiving elements of the sensor is one or more sections of a respective photon histogram, wherein each photon histogram is configured as a one-dimensional array.

[0063] According to the invention, the device is designed to carry out the method described above. In particular, the evaluation unit comprises a computing unit, such as a hardware accelerator, which is designed based on the neural network or the GMM to carry out the steps of the method described above, in particular to extract features and / or classify targets, especially objects and environmental conditions, and / or to make a statement about an environmental condition. Furthermore, the evaluation unit can comprise a memory and an input / output interface.

[0064] In a further aspect not covered by the claimed invention, the disclosure relates to a vehicle with a device according to the invention and preferably a previously described control unit. Furthermore, the vehicle may include a user interface that may include an electronic display, for example, for displaying the determined information about an environmental condition.

[0065] Furthermore, the invention comprises a computer program product comprising a computer-readable storage medium on which a program is stored that enables a computer, after being loaded into the memory of the computer, to carry out a method described above in interaction with a device described above.

[0066] Furthermore, the invention relates to a computer-readable storage medium on which a program is stored which enables a computer, after it has been loaded into the memory of the computer, to carry out a method described above in interaction with a device described above. Short description of the characters

[0067] They show in a purely schematic representation: Figure 1: a process diagram of a method according to the invention; Figure 2: a device according to the invention; Figure 3: a block diagram regarding the configuration of a vehicle with a device according to the invention; Figure 4: a photon histogram as a one-dimensional array; and Figure 5: a photon histogram as a 2D image. Preferred embodiments of the invention

[0068] Figure 1shows a method 100 according to the invention for classifying targets. The method 100 preferably comprises performing 101 time-of-flight measurements using a sensor 11. A photon histogram 40 can be created 102 based on the time-of-flight measurements. For this purpose, the method can comprise time-correlated single-photon counting 103.

[0069] The method further comprises providing 106 the measurement data to a neural network 14a, specifically a CNN or an FCN, or a GMM 14b. Providing 106 may comprise converting 104 the photon histogram 40 from a one-dimensional array 41 into a 2D image 44. Furthermore, the photon histogram 40 is provided 106 as a one-dimensional array 41. Providing 106 comprises selecting 105 a section of the photon histogram 40. The section comprises a peak 45.

[0070] The method includes extracting 107 features based on the photon histogram 40 using the neural network 14a or the GMM 14b. The method further includes classifying 108 targets based on the previously extracted features. Based on the extracted features, the method 100 may include determining 109 a statement about an environmental condition. Furthermore, the method may include determining and assigning 110 a distance to each detected target based on the extracted features.

[0071] Figure 2shows a device 10 according to the invention, which comprises a sensor 11 with a plurality of receiving elements. The sensor 11 comprises a receiving unit 13 with the receiving elements and a transmitting unit 12 with transmitting elements. The device 10 further comprises a neural network 14a or a GMM 14b, which are located on an evaluation unit 14. The neural network 14a or the GMM 14b is designed to extract features from measurement data of the sensor 11, wherein the measurement data is at least a portion of a photon histogram 40. The evaluation unit 14 further comprises a classification unit 14c, which classifies objects and environmental conditions based on the extracted features.

[0072] Figure 3 shows a block diagram regarding the configuration of a vehicle 21 with a device 10 according to the invention. In other words, Figure 3 the integration of the device 10 into a vehicle 21.

[0073] The vehicle 21 includes a plurality of electronic components that are interconnected via a vehicle communication network 22. The vehicle communication network 22 may, for example, be a standard vehicle communication network installed in the vehicle, such as a CAN bus.

[0074] The vehicle 10 comprises a control unit 15 which comprises several modules, namely a control module 16 for the braking system, a control module 17 for a powertrain, and a control module 19 for the steering system of the vehicle 21 and a control module 18 for autonomous driving.

[0075] The vehicle comprises a device 10, namely a sensor and an evaluation unit that can perform an environment classification, which, for example, enables a control unit for autonomous driving to consider the classification result in its decision-making regarding the control of the vehicle 21. The evaluation unit 14 is configured to forward control signals to the control unit 15 based on the results of the classification unit 14c.

[0076] Furthermore, the vehicle 10 comprises a radio communication interface 23, here in particular a mobile radio communication interface designed according to the LTE / UMTS standard, which enables the evaluation unit 14 to communicate with external services such as a cloud service, in particular a navigation service, and here in particular with an external emergency call center. For this purpose, the mobile radio communication interface comprises, for example, a Subscriber Identity Module (SIM), by means of which the vehicle can register with a mobile radio network, thus enabling communication via the mobile radio network, in particular with an emergency call center (provided a mobile radio network does not already permit communication with the emergency number without SIM authentication).

[0077] The vehicle 10 further comprises a satellite navigation unit 24 for detecting the position of the vehicle 21.

[0078] The vehicle 10 further includes a user interface 20 that allows a vehicle occupant to interact with one or more vehicle systems. This user interface 20 may include an electronic display for outputting graphics, icons, and / or textual content, and an input interface for receiving input (e.g., manual input, voice input, and input via gestures, head movements, or eye movements). The input interface may include, for example, keyboards, switches, touchscreens, eye trackers, and the like.

[0079] Figure 4shows an exemplary photon histogram 40, which was obtained from a receiving element and which is designed as a one-dimensional array 41. It comprises bins 42, here 30, in which the detections 43 are entered, which were detected within a defined measuring interval 46. The measuring interval 46 can be assigned to a distance measuring range. The photon histogram was determined, for example, in a measuring interval 46 of one tenth of a second. The distance measuring range is divided into thirty bins (n ​​= 1 ... 30), which are shown in the histogram of the Figure 4 are plotted on the x-axis and numbered 1 to 30. Each bin corresponds to a predefined distance range. For example, assume that the distance measurement range covers the distance range from 0 to 90 m and the bins are equidistant. In this example, each bin corresponds to a distance range of 3 m.

[0080] The number of detections for the measurement interval 46 is entered as numbers in the respective bin. For example, in bin 21, which corresponds to a distance of 60 to 63 m, 4 detections occurred, in bin 22 6 detections, in bin 23 17 detections, in bin 24 30 detections, and in bin 25 6 detections. Bins 21 to 25 thus define a peak 45 with a maximum at bin 24, which here corresponds to a distance of 69 to 72 m. Furthermore, 20 detections occurred in bin 7, 21 detections in bin 8, and 24 detections in bin 9. This defines another peak 45. The neural network 14a or the GMM 14b is designed to detect peaks 45 and evaluate their shape.

[0081] The number of detections per bin represents the intensity of the reflected signal in the respective distance range. In the photon histogram of the Fig. 4it is evident that there is a first target in the field of view of the receiving element, which delivers measurement events in bins 21 to 25, with a maximum at bin 24, which indicates a distance of the detected target of approximately 71 m. Peak 45 shows a characteristic shape for an object, so it can be concluded that the target is an object. Furthermore, based on peak 45 in bins 7 to 9, it can be concluded that there is fog in the field of view of the receiving element. In other words, the target responsible for the peak can be classified as fog. The fog is located at a distance of approximately 21 to 27 m. This can be determined from the shape of peak 45. A statement can therefore be made about an environmental condition.

[0082] Figure 5 also shows the photon histogram 40 of the Figure 4, but in the form of a 2D image 44. Instead of the corresponding detections in the form of numbers as in Figure 4 The values ​​are displayed as columns in the Y direction at the positions of the corresponding bins. This results in a two-dimensional image 44. Again, the two peaks 45 are visible. Based on the shape of the peak, the present method can be used to determine that the rear peak, with a maximum in bin 24, is based on reflection from an object, while the front peak is due to reflection from fog. Thus, the detected targets can be classified based on the shapes of the individual peaks, even if they are environmental conditions. Examples of a neural network

[0083] As an example, the architecture of a CNN network is shown that processes one-dimensional input and evaluates the distance of a target: Convolution Layer 1: ∘ Input shape: 2016x1 ∘ Initialization: Xavier ∘ Kernelsize: 5x1 o Stride:0 ∘ Padding: zero padding ∘ Featuremap out: 2007x16 ∘ Activation: ReLU Max Pooling ∘ Kernelsize: 2x1 ∘ Stride:0 ∘ Output Shape: 1003x16 Convolution Layer 2: ∘ Input Shape: 1003x16 ∘ Initialization: Xavier ∘ Kernelsize: 5x1 ∘ Stride:0 ∘ Padding: zero padding ∘ Featuremap out: 999x32 ∘ Activation: ReLU Max Pooling ∘ Kernelsize: 2x1 ∘ Stride:0 ∘ Output Shape: 499x32 Convolution Layer 3: ∘ Initialization: Xavier ∘ Kernelsize: 5x1 ∘ Stride:0 ∘ Padding:zero padding ∘ Featuremap out: 497x64 ∘ Activation: ReLU Max Pooling ∘ Kernelsize: 2x1 ∘ Stride:0 ∘ Output Shape: 248x64 Fully connected layer: ∘ Input Shape: 15872x1 ∘ Initialization: Xavier ∘ Size: 50 ∘ Activation: ReLU Dropout ∘ Doropout rate: 0.3 Fully connected layer: ∘ Input Shape: 50x1 ∘ Size: 1 ∘ Activation: none Bezugszeichenliste

[0084] 100Procedures 101Performing time-of-flight measurements using a sensor 102Creating a photon histogram based on the time-of-flight measurements 103Time-correlated single-photon counting 104Converting the photon histogram into a 2D image 105Selecting a section of the photon histogram 106Providing the measurement data to a neural network or a GMM 107Extracting features from measurement data of one or more receiving elements of the sensor using the neural network or the GMM 108Classifying targets based on the previously extracted features 109Determining a statement about an environmental condition 110Determining and assigning a distance to each detected target 10Device 11Sensor 12Transmitter unit 13Receiver unit 14Evaluation unit 14Neural network 14bGMM 14cClassification unit 15Control unit 16Control module for braking system 17Control module for powertrain 18Control module for autonomous driving 19Control module for steering system 20User interface 21Vehicle 22Vehicle communication network 23Radio communication interface 24Satellite navigation unit 40Photon histogram 411D array 42Bin 43Detections 442D image 45Peak 46Measurement interval

Claims

1. Computer-implemented method (100) for classifying targets, wherein the method (100) comprises extracting (107) features from measurement data of a plurality of receiving elements of a sensor (11) by means of a neural network (14a) or by means of a Gaussian Mixture Model (GMM; 14b), wherein the measurement data of each receiving element of the sensor (11) is a section or several sections of a respective photon histogram (40), wherein each photon histogram (40) is formed as a one-dimensional array (41), wherein the neural network (14a) is a fully connected neural network (FCN) or a convolutional neural network (CNN), wherein the method (100) comprises providing (106) the measurement data to the neural network (14a) or the Gaussian Mixture Model (GMM; 14b), wherein the method comprises selecting the section or multiple sections of the respective photon histogram per measurement interval and per receiving element, wherein a section of the respective photon histogram is provided for each peak of the respective photon histogram, so that each section consists of a peak (45), and wherein the method (100) comprises classifying (108) the targets based on the previously extracted features, wherein the previously extracted features describe shapes of the peaks of the photon histograms.

2. The method (100) according to claim 1, wherein the sensor (11) is a solid state lidar sensor, wherein the method (100) comprises performing (101) time-of-flight measurements for at least one defined measurement interval (46) by means of the sensor (11), wherein the method (100) comprises generating (102) a respective photon histogram (40) per measurement interval (46) based on the time-of-flight measurements.

3. The method (100) according to one of claims 1 or 2, wherein the method (100) comprises converting (104) the respective photon histograms (40) from a one-dimensional array (41) into a 2D image (44).

4. The method (100) according to claim 1, wherein the classification (108) is carried out by means of another neural network, wherein the other neural network is formed as a fully connected neural network (FCN).

5. The method (100) according to one of the preceding claims, wherein the method (100) comprises determining (109) a statement of an environmental condition.

6. The method (100) according to claim 5, wherein the statement concerns the presence of the environmental condition and / or the nature of the environmental condition and / or the strength of the environmental condition.

7. The method (100) according to one of the preceding claims, wherein the method (100) comprises determining and assigning (110) a distance to each detected target, in particular each detected object and / or each detected environmental condition.

8. A device (10) for classifying targets, wherein the device (10) comprises a sensor (11) with several receiving elements and an evaluation unit with a neural network (14a) or a Gaussian Mixture Model (GMM; 14b), wherein the neural network (14a) is a fully connected neural network (FCN) or a convolutional neural network (CNN), wherein the neural network (14a) or the Gaussian Mixture Model (14b) is formed to extract features from measurement data of several receiving elements of the sensor (11), wherein the measurement data of the receiving elements of the sensor (11) is a section or several sections of a respective photon histogram (40), wherein each photon histogram (40) is formed as a one-dimensional array (41), wherein the device is formed to carry out a method according to one of claims 1 to 7.

9. A computer program product comprising a computer-readable storage medium on which is stored a program which, after having been loaded into the memory of the computer, enables a computer to perform a method (100) according to one of claims 1 to 7 in cooperation with a device (10) according to claim 8.

10. A computer-readable storage medium on which is stored a program which, after having been loaded into the memory of the computer, enables a computer to perform a method according to one of claims 1 to 7 in cooperation with a device (10) according to claim 8.