Method and device for classifying objects

Abstract

A method (100) for classifying an object is provided, the method (100) comprising providing (106) measurement data from a sensor to a feature extraction unit (13), and extracting (107) modality-independent features (24) from the measurement data by the feature extraction unit (13), wherein the modality-independent features (24) are independent of a sensor modality of the sensor, so that it is not possible to deduce the sensor modality of the sensor from the modality-independent features.

Description

Technical Field

[0001] The present invention relates to a method and a device for classifying objects. Background Art

[0002] For use in driverless navigation of a vehicle, it is necessary to know which types of road users are around the vehicle and which types of infrastructure are located in the surrounding area. In other words, it is crucial to classify the objects. It is known from the prior art to use measurement data of different sensor modalities for object classification or object detection. This is particularly advantageous since the measurement data of different sensor modalities can partially provide complementary information about the surrounding area. The data from the individual sensors are therefore usually processed separately and subsequently also classified separately. Only later can the data be merged again and further processed.

[0003] According to the prior art, the data extracted from the measurement data are also modality-dependent, so it is necessary to provide a separate classification module for each sensor modality and therefore each classification module must also be trained. This involves a huge amount of work, especially when no image data is involved, because the annotation of the lidar or radar datasets used to train the classification modules is very time-consuming and error-prone. The reason for this is that the measured data are difficult for humans to understand and the people who annotate the corresponding datasets have to be trained in a time-consuming manner before they can start to perform the task. In addition, with regard to lidar and radar, the available annotated datasets are much smaller than those for images. Summary of the invention

[0004] The present invention is based on the object of improving a method and a device for classifying objects in such a way that, compared to the prior art, a classification unit for classifying features does not need to be trained separately when measuring measurement data of different sensor modalities, but can be trained once based on the measurement data of a single sensor modality.

[0005] The above objects are achieved according to a method for classifying an object according to the present invention. The method comprises providing a feature extraction unit with measurement data from a sensor. The method further comprises, in particular by means of the feature extraction unit, extracting modality-independent features from the measurement data. The modality-independent features are independent of the sensor modality of the sensor, so that it is not possible to derive the sensor modality from the modality-independent features. The method may also comprise the extraction of modality-dependent features.

[0006] the term In particular, the German term for the English term "sensor modality". In other words, a sensor modality is a sensor class or sensor type. Sensors therefore differ in particular in terms of the sensor modality, i.e. the sensor type or sensor class. The sensor modality determines in particular the measuring method by which the measurement data are generated. The sensor modality is preferably a lidar, radar, image or ultrasonic. Preferably, this means that the corresponding sensor of the modality is a lidar sensor, a radar sensor, an image sensor, in particular a camera or ultrasonic sensor.

[0007] The term "feature" refers in particular to characteristics of the measurement data. They are in particular so-called "features", which usually depend on the sensor modality of the sensor which acquired the measurement data. However, modality-independent features are independent of the sensor modality of the sensor which acquired the measurement data. In other words, modality-independent features can be detected in the measurement data of at least two sensor modalities of the same object. Therefore, they can also be referred to as common features. In other words, they are roughly sensor-independent features. Based on the modality-independent features, it is not possible to deduce the sensor modality of the sensor. Modality-independent features are in particular independent of the type of measurement method that produced the measurement data.

[0008] In other words, the method is able to extract modality-independent features from measurement data that are independent of the sensor modality of the sensor. Therefore, it is no longer necessary to provide and train a classification unit for classification features for each sensor modality separately. It is sufficient to provide a classification unit for all measurement data of different sensor modalities.

[0009] Preferably, there are at least a first sensor modality and a second sensor modality, wherein the method is configured to extract modality-independent features from the measurement data of the sensor of the first sensor modality in such a way that the measurement data of the sensor of the second measurement modality can be reconstructed. The method is preferably configured to extract modality-independent features from the measurement data of the second sensor modality in such a way that the measurement data of the first modality can be reconstructed. In particular, there are more than two sensor modalities, in particular three or four, wherein the method extracts modality-independent features in such a way that the measurement data of each sensor modality can be reconstructed from these features.

[0010] Reconstruction means that when a decoder is provided for each sensor modality, i.e. a decoder for the first sensor modality and a decoder for the second measurement modality, they will be able to generate the measurement data of the respective modality from the modality-independent features. For example, the first sensor modality is LiDAR and the second modality is an image. The feature extraction unit can extract the modality-independent features from the LiDAR measurement data. Therefore, even if the mode-independent features only originate from the LiDAR measurement data, an image, i.e. the measurement data of the second sensor modality, can be reconstructed from these modality-independent features.

[0011] The sensor modality preferably determines the type of measurement data, wherein the type of measurement data is radar measurement data, lidar measurement data, image data or ultrasound measurement data.

[0012] The measurement data may in particular be point clouds and / or image data. The point cloud may in particular be an unclassified point cloud. The method is preferably configured to extract modality-independent features from the point cloud and / or image data in such a way that measurement data of other measurement modalities in the individual case can be reconstructed. The feature extraction unit is in particular configured to extract modality-independent features from the point cloud and / or image data in such a way that measurement data of other sensor modalities in the individual case, i.e. images and / or point clouds, can be reconstructed from the modality-independent features.

[0013] The sensor has in particular a first sensor modality, wherein this preferably means that the sensor corresponds to a certain sensor modality. The sensor may preferably have only a single sensor modality.

[0014] The method comprises in particular the generation of measurement data, in particular for classification purposes, which measurement data are subsequently transmitted to a feature extraction unit. The measurement data are in particular lidar measurement data, whereby the sensor of the first sensor modality is a lidar sensor. To generate the measurement data, the method comprises in particular the transmission of a plurality of measurement pulses, in particular for generating lidar measurement data. The measurement data of the different sensor modalities originate in particular from the same scene.

[0015] The measuring pulse is in particular an optical signal, in particular an electromagnetic signal. The measuring pulse preferably has a wavelength which does not originate from a wavelength range visible to the human eye. For safety reasons, invisible infrared radiation is preferably used. The measuring pulse preferably has a pulse width such that the measuring pulse can be understood as a time-limited portion of the electromagnetic radiation. Due to the fact that the measuring pulse is an electromagnetic signal, the speed of the measuring pulse is known, so that it is possible to draw conclusions from the speed of light from the flight time of the measuring pulse, i.e. how much distance the measuring pulse covers within the flight time.

[0016] The method comprises in particular the execution of a plurality of scans. This means that a measuring pulse or a series of measuring pulses are transmitted successively in different directions. The method is in particular a scanning lidar method. After the measurement pulses have been sent into the range to be measured, the respective scan is ended and a new scan can be started. After each ended scan, modality-independent features are extracted from the measurement data, in particular by a feature extraction unit, in order to classify the object.

[0017] The method in particular comprises generating measurement data from a sensor of a second sensor modality, wherein the method comprises providing the measurement data to a feature extraction unit. The second sensor modality and the first sensor modality are preferably not identical. In other words, the method preferably comprises generating measurement data from a sensor of the second sensor modality and generating measurement data from a sensor of the first sensor modality, wherein the method comprises providing the measurement data of the two sensor modalities to the feature extraction unit. The method may further comprise generating measurement data from a sensor of a third sensor modality, which measurement data is likewise provided to the feature extraction unit. The third sensor modality, the second sensor modality and the first sensor modality are preferably not identical. The method may further comprise generating measurement data from a sensor of a fourth sensor modality, which measurement data is likewise provided to the feature extraction unit. The first sensor modality and / or the second sensor modality and / or the third sensor modality and / or the fourth sensor modality may be a lidar, a radar, an image or an ultrasound. The sensor modalities are in particular not identical.

[0018] The method thus comprises in particular the generation of measurement data of different sensor modalities and the extraction of modality-independent features, making it possible to benefit from training the classification unit by relying on other sensor data.

[0019] The feature extraction unit preferably comprises at least one feature extractor for each sensor modality, wherein the feature extractor extracts features from the measurement data. The feature extraction unit in each case in particular comprises a feature extractor for each sensor modality of a sensor, which generates the measurement data. The feature extraction unit in particular comprises a feature extractor for a first sensor modality, which feature extractor is preferably used for lidar measurement data, wherein the feature extractor is configured to extract lidar features from the lidar measurement data. Therefore, the features extracted by the feature extractor include features that can only be seen in the measurement data of one sensor modality (in other words, modality-related features), as well as features that can be seen in multiple sensor modalities (in other words, modality-independent features). However, the feature extractor cannot distinguish these features from each other. In other words, the feature extractor does not know which features are modality-related and which features are modality-independent.

[0020] The feature extraction unit more preferably comprises separate corresponding feature extractors for the second sensor modality and / or the third sensor modality and / or the fourth sensor modality, which are configured to extract features from the corresponding measurement data, for example to extract corresponding radar features from the radar measurement data, or to extract corresponding image features from the image data.

[0021] The feature extractor is in particular an "encoder", which is preferably a lidar encoder, a radar encoder, an image encoder or an ultrasonic encoder, depending on the sensor modality. The encoder can preferably be a point cloud encoder, in particular when the measurement data are lidar measurement data.

[0022] In each case, the feature extractor comprises in particular a neural network. The feature extractor is in particular a neural network. Preferably, the neural network for each sensor modality preferably extracts features that can be detected therein.

[0023] Further preferably, the feature extractor for the second sensor modality, ie preferably for image data, comprises a convolutional neural network (CNN) comprising at least one convolutional layer. Each layer comprises a plurality of artificial neurons.

[0024] Each layer of CNN can be configured as a convolution layer. The convolution layer particularly performs convolution. CNN particularly comprises at least 5 layers, preferably at least 10 layers, and more preferably at least 15 layers.

[0025] In each case, all layers (except the last) are activated in particular by a non-linear function, in particular the so-called ReLU function. The ReLU function is thus a "rectified linear function". In other words, it is a function that is linear in the range of positive x values, but zero in the range of negative x values, where x represents the input value of the neuron. The last layer preferably has no activation function, in order to allow all values ​​as the last step.

[0026] CNN is able to process input in the form of images, especially in the form of matrices. In the last layer, CNN extracts simple features (such as straight lines), and the deeper the network, the more complex the extracted features (such as L features, circular features, star features, high-dimensional features). Therefore, the output of the network is a feature, especially in the form of a vector. The output is also called an image encoding. This image encoding is also referred to as "actual image encoding" below to distinguish it from the regenerated image encoding of the feature re-transformation unit.

[0027] The feature extractor for the first sensor modality, i.e. preferably for lidar measurement data, comprises in particular a "multilayer perceptron" (MLP) network, which comprises at least one "multilayer perceptron" as a layer (MLP layer). Each layer contains a plurality of artificial neurons. The MLP network comprises in particular a plurality of MLP layers. Preferably, all MLP layers are activated in each case by a nonlinear function, in particular a so-called ReLU function. The MLP preferably has at least 3 layers, in particular at least 5 layers, which may comprise a plurality of sublayers.

[0028] The MLP network may further comprise at least one fully connected layer, which may also be activated by a non-linear function, in particular a so-called ReLU function.

[0029] The MLP network is able to process input in the form of point clouds. Therefore, the output of the network is features, in particular in the form of vectors. The output is also called the point cloud encoding. This point cloud encoding will also be referred to as the "actual point cloud encoding" in the following to distinguish it from the regenerated point cloud encoding of the feature retransformation unit.

[0030] It is further preferred that the feature extraction unit comprises a feature transformation unit, which extracts modality-independent features from the features of at least two feature extractors. The term "transformation" should be understood in particular as the discovery of modality-independent features, wherein the modality-independent features are stored in a common feature space. In other words, the features are found and displayed in a common space. In other words, the feature transformation unit should therefore be understood as a unit for finding modality-independent features. All features extracted by the feature extractors are in particular provided to the feature transformation unit, wherein the feature transformation unit extracts modality-independent features therefrom.

[0031] This has the advantage that, based on the modality-independent features, it is no longer possible to conclude that the features originate from the sensor modality of the sensor. As a result of the extraction and transformation, the feature extraction unit in particular delivers a modality-independent feature vector. The feature extraction unit in particular supplies the feature vector to the classification unit for classification purposes.

[0032] The feature transformation unit comprises in particular a neural network for each sensor modality. The purpose of the network is to find modality-independent features and optionally modality-dependent features, and store them in a common feature space. The feature transformation unit is configured to process features (in particular feature vectors) originating from the feature extractor as input.

[0033] Each neural network of the feature transformation unit comprises in particular at least one fully connected layer. Preferably, the network comprises at least 2, preferably at least 3, fully connected layers. Preferably, in each case, the fully connected layers are activated by a non-linear function, in particular a so-called ReLU function. The network may further comprise at least one dropout layer. In particular, the network is arranged in such a way that a dropout layer is arranged between two fully connected layers. The network comprises in particular 5 layers. Therefore, the last layer may not be activated. In particular, all neural networks of different sensor modalities of the feature transformation unit are arranged identically.

[0034] The output of the network of the feature transformation unit is modality-independent features, in particular in the form of a vector. The feature transformation unit may also additionally publish modality-dependent features. In other words, the feature transformation unit looks for features that can be detected in all measurement data of the different sensor modalities, i.e. features that have commonality in all sensor modalities. These modality-independent features are published. However, features that only appear in one sensor modality, i.e. modality-dependent features, may also be additionally published.

[0035] The neural network of the first sensor modality specifically publishes modality-independent features (i.e., common features) and optionally lidar-specific features. The neural network of the second sensor modality specifically publishes modality-independent features (i.e., common features) and optionally image-specific features. Common features are features included in lidar measurement data and image data of the same scene.

[0036] The output is also called feature encoding. If the network also releases modality-dependent features, it can be specifically specified to the network, i.e. at which position in the network, in other words, at which index of the network output the modality-independent features and the modality-dependent features are arranged. Thus, the output can be segmented.

[0037] A specific example is a streaky wall, where both LiDAR measurement data and image data are available. The streaks cannot be detected in the LiDAR measurement data, while they can be detected in the image data. The opposite is true for the depth information of the wall. The latter is available from the LiDAR measurement data, but not from the image data. Thus, the streaks will be an image-specific feature, while the depth information is a LiDAR-specific feature. The outline of the wall, which can be detected in both the image data and the LiDAR measurement data, will be a common feature.

[0038] The extracted features of at least one feature extractor are preferably vectors, so-called feature vectors. The respective feature extractor is used in particular to extract features, in particular in the form of vectors, but in each case in different spaces depending on the sensor modality. In other words, features from different sensor modalities are located in different spaces.

[0039] The feature transformation unit may be configured to transfer features of different spaces to a common space, the so-called feature space. The feature space is in particular a vector space, preferably a metric space.

[0040] All features from different sensor modalities, in other words, the outputs of all feature extractors, are specifically provided to a feature transformation unit. The latter transforms the features in such a way that the first part of the features is present in a common feature space. This space shows features that have both common features. These features that have both common modalities are modality-independent features, in other words, common features.

[0041] Furthermore, other parts of the feature may exist in different feature spaces, in other words, in modality-dependent feature spaces or modality-specific feature spaces. The second part may, for example, exist in a feature space assigned to the first sensor modality, while the third part may exist in a feature space assigned to the second sensor modality. The same may apply to the third sensor modality and the fourth sensor modality. These special spaces contain features that are only visible in one specific modality.

[0042] Preferably, the method may comprise prior training of the feature extraction unit, in particular prior training of the feature transformation unit and / or the respective feature extractors. The term "prior" more preferably means that the training is performed before the generation of measurement data for classification purposes. For this purpose, at least one data set is in particular provided to the feature extraction unit for learning purposes. In particular, unsupervised learning is performed based on at least one unannotated data set. In order to learn the modality-independent features, the feature transformation unit, preferably the feature transformation unit and / or the respective feature extractors, is in particular trained by unsupervised learning.

[0043] The feature extraction unit, in particular the feature transformation unit and / or the respective feature extractor, can be specially trained. The feature transformation unit can be further trained by in-depth learning, so-called deep learning.

[0044] The feature transformation unit and / or the individual feature extractors are trained, in particular, by unsupervised learning. The corresponding trained neural network thus includes weights that are determined by learning.

[0045] Preferably, the feature extractor is trained first. The feature extractor, referred to as the encoder, is preferably trained together with the corresponding decoder. This is advantageous because they represent the outer layers of different units. Thus, the encoder and decoder of each sensor modality are learned separately from the other sensor modalities, so that they can be learned on different datasets. Thus, in particular, there is no reliance on a large number of generated datasets, which include measurement data of multiple sensor modalities.

[0046] The training of the feature extractor comprises the provision of an input. The input is in particular sensor modality specific. In other words, the input to the feature extractor for the first sensor modality is lidar measurement data, in other words a point cloud, while the input for the second sensor modality is image data.

[0047] The input is preferably plugged into a feature extractor, which publishes features as output. The feature extractor for the first sensor modality publishes a point cloud encoding, while the feature extractor for the second sensor modality publishes an image encoding. This "actual" encoding is directly plugged into the corresponding decoder, thereby generating an output related to the sensor modality from the decoder again. In other words, the decoder for the first sensor modality again generates a point cloud, while the decoder for the second sensor modality again generates an image, in particular a gray value image.

[0048] The sensor modality-specific output of the decoder is compared with the corresponding sensor modality-specific input inserted into the corresponding encoder. It is set as a condition that the output of the decoder is the same as the input to the encoder. Based on this condition, the weights of the corresponding neural network are assigned so that the similarity is as high as possible.

[0049] Preferably, the decoder for the second sensor modality issues gray values ​​of the "regenerated" image, which are compared with the gray values ​​of the original input image. In case of a color image as input, the gray value image of the input can be generated by simply adding the R, G, B values ​​of each pixel and then dividing by 3. For comparison purposes, a loss function is used in particular, which compares the "mean square error" of the gray values ​​of the generated gray value image of the decoder with the gray values ​​of the actual gray value image of the input. The loss should be equal to zero.

[0050] Preferably, the decoder for the first sensor modality issues a "regenerated" point cloud, which is compared with the point cloud provided to the feature extractor. For comparison purposes, preferably the chamfer distance between the point clouds is determined. Therefore, for each point of one point cloud, the distance to the nearest point of the other point cloud is preferably determined. The same operation is performed similarly for each point of the other point cloud. The distances are added. The sum of all distances is the chamfer distance. The more similar the point clouds are, the smaller the chamfer distance. In particular, a loss function is used, which compares the chamfer distance between the generated point cloud of the decoder and the original point cloud. The loss should be equal to zero.

[0051] The feature transformation unit can be trained as a second step. Preferably, the feature transformation unit is trained together with the feature re-transformation unit. The feature re-transformation unit preferably also has a neural network for each sensor modality. For training purposes, the corresponding encoders and decoders of the different sensor modalities are also used, but their weights are specified because the training of the corresponding encoders and decoders has been completed at this point in time.

[0052] The neural network of the feature retransformation unit has at least one fully connected layer. In particular, the neural network has two subunits, one of which processes the modality-dependent features and the other of which processes the modality-independent features.

[0053] Preferably, the network comprises at least 2, preferably at least 3 fully connected layers. Preferably, each fully connected layer is activated by a non-linear function, in particular a so-called ReLU function. The network may further comprise at least one dropout layer. In particular, the network is arranged in such a way that a dropout layer is arranged between two fully connected layers. The last layer is not activated.

[0054] To train the feature transformation unit, in each case an input related to the sensor modality, i.e. modality-specific for the same scene, is provided to the feature extractor. In other words, the input to the feature extractor for the first sensor modality is the lidar measurement data, in other words the point cloud, while the input for the second sensor modality is the image data.

[0055] The input is preferably plugged into a feature extractor which delivers features as output. The feature extractor for the first sensor modality delivers a point cloud encoding, while the feature extractor for the second sensor modality delivers an image encoding. This encoding is now plugged into a corresponding neural network of a feature transformation unit which extracts modality-independent features and optionally modality-dependent features therefrom.

[0056] The output of the neural network is inserted into the corresponding neural network of the feature re-transformation unit, which again generates a sensor modality-dependent code from the features. In other words, the neural network of the feature re-transformation unit for the first sensor modality again generates a point cloud code, while the neural network of the feature re-transformation unit for the second sensor modality again generates an image code.

[0057] The main condition is set to be that the common features of different networks are the same or that the common features of different modalities are the same. For this purpose, the common features that have been generated by the neural network of the feature transformation unit for the first sensor modality are compared with the common features that have been generated by the neural network for the second sensor modality. The cosine similarity of the same features can be detected in both modalities and should in particular be equal to zero. This is to achieve the purpose of finding common features.

[0058] The encoding of each neural network of the feature retransformation unit is further compared with the "actual" encoding of the corresponding feature extractor (i.e. the output of the feature extractor). In the first step, the second condition is set that the forward transformation and the reverse transformation will receive the same encoding, i.e. the same features. In other words, it is stipulated as a second condition that the mean square error should be equal to zero. This is in particular for the purpose of learning the initial weights of the neural network of the feature transformation unit.

[0059] Preferably, a loss function is used which first has the following conditions, preferably for at least 100 epochs, a maximum of 300 epochs, and most preferably 200 epochs:

[0060] Cosine similarity of common features of two modalities

[0061] The mean square error between the “actual image encoding” of the feature extractor and the re-transformed image encoding of the feature re-transformation unit

[0062] The mean square error between the “actual point cloud encoding” of the feature extractor and the re-transformed point cloud encoding of the feature re-transformation unit

[0063] Determine the common loss of the above conditions, which should be equal to zero, and optimize it by adjusting the weights.

[0064] In the second step, the output of the neural network of the feature retransformation unit is inserted into the corresponding decoder, and the decoder output is then compared with the input inserted into the feature extractor. The main terms then no longer refer to the characteristics of the features or the encoding, respectively, but to the characteristics of the original modality-dependent input to the feature extractor and the characteristics of the output of the decoder.

[0065] Preferably, the decoder for the second sensor modality issues grayscale values ​​of the "regenerated" image, which are compared with the grayscale values ​​of the original input image. For comparison purposes, the "mean square error" of the grayscale values ​​of the grayscale value image generated by the decoder and the grayscale values ​​of the input actual grayscale value image are compared.

[0066] Preferably, the decoder for the first sensor modality delivers a "regenerated" point cloud, which is compared with the point cloud. For the comparison, preferably the chamfer distance is determined.

[0067] Finally, the following conditions are set, preferably at least 100 epochs, a maximum of 300 epochs, and most preferably 200 epochs:

[0068] Cosine similarity of common features of two modalities

[0069] Mean square error between the “actual image data” and the generated image data

[0070] The chamfer distance between the “actual point cloud” and the generated point cloud

[0071] The common loss of the above conditions is determined and optimized by adjusting the weights.

[0072] The method comprises in particular classifying modality-independent features, in particular feature vectors received from the feature extraction unit, by means of a classification unit. The method comprises in particular using a single classification unit to classify all features of sensors of all sensor modalities, which sensors generate and provide measurement data. The method does not comprise in particular using a separate classification unit for each sensor modality.

[0073] The method may further comprise prior training of the classification unit by supervised learning. For this purpose, in particular at least one data set for learning is provided to the classification unit. In particular, the classification unit is trained based on annotated image data. The term "prior" more preferably means that the training is performed before the generation of measurement data for classification purposes. The classification unit is in particular trained by supervised learning in order to classify modality-independent features.

[0074] The training of a classification unit based on annotated image datasets is particularly advantageous, in particular since annotated image datasets currently account for the largest amount of annotated data. Due to the fact that the feature extraction unit is configured to extract modality-independent features independent of the sensor modality, a single classification unit trained on a dataset of a single sensor modality can be used. Even if an annotated image dataset is preferably used to train the classification unit, based on the modality-independent nature of the extracted features, the classification unit is also able to classify features from lidar measurement data, even if lidar measurement data have never been used to train the classification unit. This represents a significant advantage of the present method compared to the prior art.

[0075] The method comprises in particular the transfer of at least one feature vector from a feature extraction unit to a classification unit. The feature vector may comprise only modality-independent features or may also comprise modality-dependent features. The classification comprises a comparison of the received feature vector with a previously defined average feature vector for each class, wherein, if a previously defined deviation limit is exceeded, a corresponding class label is issued.

[0076] The training of the classification unit can include, in particular, the determination of classes. In particular, the classification unit determines classes, which are assigned so-called class labels after the training, based on at least one data set, which is provided to the classification unit for training purposes. One class can, for example, relate to cars, while another class can relate to pedestrians.

[0077] The at least one data set for training purposes preferably comprises a plurality of feature vectors, wherein a plurality of feature vectors can be assigned to each class. The training can further comprise determining an average feature vector for each class. The average feature vector of a class is in particular determined by averaging the feature vectors of the class. The method comprises storing the average feature vector.

[0078] The method may in particular comprise transmitting at least one feature vector from the feature extraction unit to a classification unit, the at least one feature vector being classified by the classification unit. First, the feature vector is processed, in particular by the classification unit. As a result, a processed feature vector, in particular a bottleneck vector, is received. The processing serves in particular the purpose of being able to detect clearer separation lines between different classes.

[0079] The classification unit comprises in particular a neural network, wherein the neural network preferably comprises at least 3 layers, preferably at least 4 layers, more preferably 5 layers. The neural network comprises in particular at least one fully connected layer. Preferably, the network comprises at least 2 fully connected layers, preferably at least 3 fully connected layers. Preferably, in each case, the fully connected layers are activated by non-linear functions, in particular so-called ReLU functions. The network comprises in particular 5 layers. Therefore, the last layer is not activated. The last layer is called the bottleneck vector.

[0080] The neural network of the classification unit is used for processing of the feature vector. As a result, a bottleneck vector is preferably received.

[0081] The classification unit may further comprise a decision module, wherein the decision module performs a comparison between the received feature vector and an average feature vector. The average feature vector is in particular also an average bottleneck vector. In particular, the corrected feature vector is compared with the average feature vector. Thus, the average feature vector represents some trained knowledge of the classification unit.

[0082] For comparison purposes, in each case in particular the cosine similarity between the received feature vector (preferably the processed feature vector) and the average feature vector is determined. Deviation limits, so-called thresholds, are specifically specified in advance. The purpose of this is to specify at which similarity a certain class is to be detected or from which deviation a class label is no longer assigned to the class. When the deviation limit to the average feature vector is fallen below, the corresponding class label is issued. In other words, in the case of sufficient similarity to the average feature vector of the class, the classification module detects an object of this class and issues the corresponding class label.

[0083] When all previously defined deviation limits are exceeded from the mean feature vector, the decision module can issue a result that an object of a class was not detected. In particular, no class label is issued. The comparison as well as the issuance is performed in particular by the decision module of the classification unit. The decision module thus decides whether a class label is to be issued in response to a deviation from certain knowledge of the classification unit.

[0084] The present invention also includes a method for optical distance measurement, which includes the above-mentioned method for classifying objects.

[0085] Optical distance measurement is characterized in particular in that the distance is determined by using an optical signal, here an optical measuring pulse. The term "distance" is to be understood as a range. The distance covered by the measuring pulse is to be understood as the path between the device for performing the method, which has sent the measuring pulse, and the object, which has reflected the measuring pulse, plus the path between the device and the object, which has received the corresponding reflected measuring pulse. The reflected measuring pulse in each case represents the backscattered signal of the sent measuring pulse. In other words, the measuring pulse reflected on the object is the measuring pulse previously sent by the sending unit. Optical distance measurement is preferably based on the time-of-flight principle.

[0086] The method for classifying objects and / or the method for measuring distances are used in particular for navigation of autonomous vehicles.

[0087] In another aspect, the present invention includes an apparatus for classifying an object, the apparatus being configured to perform the above-mentioned method for classifying an object.

[0088] The device comprises in particular a transmitting unit and a receiving unit. In particular, the receiving unit comprises in each case at least one sensor for each sensor modality, wherein the transmitting unit can in each case have a corresponding transmitting module for generating measurement data, which is used to transmit a signal for generating the measurement data of the respective sensor modality.

[0089] The device comprises in particular a sensor of a first sensor modality, preferably a sensor of a second sensor modality, and / or a sensor of a third sensor modality, and / or a sensor of a fourth sensor modality. Thus, the device is in particular configured to generate measurement data of at least two sensor modalities, preferably three or four sensor modalities.

[0090] The device is in particular a scanning LiDAR sensor, wherein the scanning LiDAR sensor may also preferably include sensors of other sensor modalities. The device may in particular be further configured to perform the above-mentioned method for distance measurement.

[0091] The device further comprises a feature extraction unit, which in turn has a feature extractor for each sensor modality of the used sensor. The feature extraction unit further comprises a feature transformation unit, which is used to extract modality-independent features and optionally modality-dependent features from the extracted features of the feature extractor. In particular, the feature transformation unit is trained by unsupervised learning, wherein the classification unit is more preferably trained by supervised learning.

[0092] The device is particularly useful for guiding unmanned vehicles.

[0093] The present invention also includes a computer program product, which includes a computer-readable storage medium on which a program is stored, which, after being loaded into the memory of a computer, enables the computer to execute the above-mentioned method for classifying objects and / or the method for distance measurement, optionally together with the above-mentioned device.

[0094] Furthermore, the invention relates to a computer-readable storage medium having stored thereon a program which, after being loaded into a memory of a computer, enables the computer to execute the above-described method for classifying objects and / or the method for distance measurement, optionally together with the above-described device. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] The following figures schematically show:

[0096] Figure 1 is a flow chart of a method according to the present invention;

[0097] Figure 2 is a device according to the present invention;

[0098] Figure 3 is a schematic diagram of a feature extraction unit; and

[0099] Figure 4 is a feature extractor and decoder for the first sensor modality and the second sensor modality for training the feature extractor; and

[0100] Figure 5 It is a component for training a feature transformation unit, which includes a feature extraction unit, a feature re-transformation unit and a decoder for training the feature transformation unit. DETAILED DESCRIPTION

[0101] Figure 1 A flow chart of a method 100 according to the present invention is shown.

[0102] The method 100 according to the present invention may preferably include previous training 112 of the feature extractor and previous training 101 of the feature transformation unit 17 of the feature extraction unit 13. Further preferably, the method 100 according to the present invention may include previous training 102 of the classification unit 18.

[0103] The training 102 of the classification unit 18 may in particular include the determination 102a of the categories, in particular based on at least one data set, which is provided to the classification unit 18 for training purposes. After the training is completed, the categories are determined, to which the classification unit 18 will assign a so-called class label. The training 102 may further include the identification and storage 102b of the average feature vector for each category.

[0104] The method comprises in particular the provision 106 of measurement data for the feature extraction unit 13. In particular, the method may comprise beforehand the generation 103 of measurement data from sensors of the first sensor modality, more preferably, likewise the generation 104 of measurement data from sensors of the second sensor modality and / or the generation 105 of measurement data from sensors of the third sensor modality. Steps 103 to 105 may preferably be performed simultaneously. The method comprises in particular the performance of a scan, wherein after the end of the scan the measurement data of all sensor modalities are available and can be provided to the feature extraction unit 13.

[0105] The method further comprises the extraction of modality-independent features 107. For this purpose, the extraction of features is preferably performed by corresponding feature extractors 108. Modality-independent features are extracted 109 from the features of at least two feature extractors by a feature transformation unit 17. The method may further comprise the classification 110 of the modality-independent features by a classification unit 18, so that object classification 111 may be performed.

[0106] The classification 110 may comprise forwarding 110a at least one feature vector from the feature extraction unit 13 to the classification unit 18, wherein the classification unit 18 preferably processes the feature vector 110b. Thus, preferably a bottleneck vector is received. In each case, the received or preferably corrected feature vector is compared 110c with a previously determined average feature vector. For comparison, the deviation limits (which may be multiple) of the average feature vector may further be specified 110d. The respective cosine similarities between the feature vector and the average feature vector may preferably be determined 110e. When one of the previously specified deviation limits is below, in particular, the publication of the corresponding class label occurs 110f, and when all previously specified deviation limits are exceeded, the publication that no class of objects is detected occurs 110g. This is carried out in particular by a decision module 18a of the classification unit 18.

[0107] Figure 2 A schematic diagram of an apparatus 10 according to the invention is shown. The apparatus 10 comprises a sending unit 11 and a receiving unit 12. In particular, the receiving unit 12 comprises in each case at least one sensor for each sensor modality, wherein the sending unit 11 can in each case have a corresponding sending module for generating measurement data, which is used to send a signal for generating the measurement data of the respective sensor modality. For example, the sending unit 11 can in each case have a source for sending lidar and radar measurement data.

[0108] The measurement data received in the receiving unit 12 are provided to a feature extraction unit 13. The feature extraction unit 13 comprises a feature extractor 14 for measurement data from a sensor of a first sensor modality, a feature extractor 15 for measurement data from a sensor of a second sensor modality, and a feature extractor 16 for measurement data from a sensor of a third sensor modality. The respective feature extractors 14, 15, 16 extract features, which are provided to a feature transformation unit 17, which uses these features to generate modality-independent features. The apparatus 10 further has a classification unit 18, which classifies the modality-independent features of the feature transformation unit 17. The classification unit 18 comprises a decision module 18a.

[0109] Figure 3 A schematic diagram of the feature extraction unit 13 is shown in . The feature extraction unit 13 comprises a feature extractor 14 for measurement data from a sensor of a first sensor modality, and a feature extractor 15 for measurement data from a sensor of a second sensor modality.

[0110] The feature extractor receives a modality-dependent input 19. The feature extractor 14 for the measurement data of the first sensor modality is configured to generate a point cloud encoding 23 from the measurement data of the first sensor modality, i.e. from the LiDAR point cloud 21. The feature extractor 15 for the measurement data of the second sensor modality is configured to generate an image encoding 22 from the measurement data of the second sensor modality, i.e. from the image data 20.

[0111] The feature extraction unit 13 further comprises a feature transformation unit 17. The feature transformation unit 17 comprises a neural network 17a for the measurement data of the first sensor modality and a neural network 17b for the measurement data of the second sensor modality. As input, they receive the respective encodings of the feature extractors.

[0112] The feature transformation unit 17 is configured to detect modality-independent features 24. These exist in a common feature space 26. The feature transformation unit 17 may also emit modality-dependent features 25, which exist in their own feature spaces, i.e., in a feature space 27 of modality-dependent features for the first sensor modality and in a feature space 28 of modality-dependent features for the second sensor modality.

[0113] exist Figure 4, a feature extractor 14 and a decoder 30 for a first sensor modality and a feature extractor 15 and a decoder 31 for a second sensor modality for training a feature extractor are shown. The decoder 30 for the first sensor modality emits an output 30a, i.e., a point cloud, which is compared with the modality-dependent input 19 of the feature extractor 14 to train the feature extractor 14. The decoder 31 for the second sensor modality emits an output 31a, i.e., image data, which is compared with the modality-dependent input 19 of the feature extractor 15 to train the feature extractor 15.

[0114] Figure 5 The components for training the feature transformation unit 17 are shown, which include Figure 3 The invention relates to a feature extraction unit 13, a feature re-transformation unit 29 and a decoder, i.e. a decoder 30 for the first sensor modality and a decoder 31 for the second sensor modality. The feature re-transformation unit 29 comprises a neural network 29a for the first sensor modality and a neural network 29b for the second sensor modality. As input, they receive the modality-independent features 24 and optionally the modality-dependent features 25 of the feature transformation unit 17.

[0115] The feature re-transformation unit 29 is configured to generate encodings from the inputs again, namely the image encoding 32 and the point cloud encoding 33. The respective decoders can again generate modality-related data from the corresponding encodings. The decoder 31 for the second sensor modality generates an output 31a corresponding to the regenerated image data. The decoder 30 for the first sensor modality generates an output 30a corresponding to the regenerated lidar point cloud 21.

[0116] For training the feature transformation unit 17, the similarity of the modality-independent features 24 is taken into account, as well as the similarity of the regenerated encoding and the regenerated modality-dependent data.

[0117] Examples of Neural Networks

[0118] The feature extractor for the second sensor modality can be set up as follows:

[0119] Input ← image (batch size × 192 × 256 × 3)

[0120] Layer 1: Convolution

[0121] Convolution kernel size: 3×3

[0122] stride: 1×1

[0123] Number of output feature maps: 16

[0124] Activation function: Rectified Linear Unit (ReLU)

[0125] Batch normalization

[0126] Padding: zero padding

[0127] Layer 2: Convolution

[0128] Convolution kernel size: 3×3

[0129] Step size: 1×1

[0130] Number of output feature maps: 16

[0131] Activation function: ReLU

[0132] Batch Normalization

[0133] Padding: Zero padding

[0134] Layer 3: Convolution

[0135] Convolution kernel size: 3×3

[0136] Step size: 2×2

[0137] Number of output feature maps: 32

[0138] Activation function: ReLU

[0139] Batch Normalization

[0140] Padding: Zero padding

[0141] Layer 4: Convolution

[0142] Convolution kernel size: 3×3

[0143] Step size: 1×1

[0144] Number of output feature maps: 32

[0145] Activation function: ReLU

[0146] Batch Normalization

[0147] Padding: Zero padding

[0148] Layer 5: Convolution

[0149] Convolution kernel size: 3×3

[0150] Step size: 1×1

[0151] Number of output feature maps: 32

[0152] Activation function: ReLU

[0153] Batch Normalization

[0154] Padding: Zero padding

[0155] Layer 6: Convolution

[0156] Convolution kernel size: 3×3

[0157] Step size: 2×2

[0158] Number of output feature maps: 64

[0159] Activation function: ReLU

[0160] Batch Normalization

[0161] Padding: Zero padding

[0162] Layer 7: Convolution

[0163] Convolution kernel size: 3×3

[0164] Step size: 1×1

[0165] Number of output feature maps: 64

[0166] Activation function: ReLU

[0167] Batch Normalization

[0168] Padding: Zero padding

[0169] Layer 8: Convolution

[0170] Convolution kernel size: 3×3

[0171] Step size: 1×1

[0172] Number of output feature maps: 64

[0173] Activation function: ReLU

[0174] Batch Normalization

[0175] Padding: Zero padding

[0176] Layer 9: Convolution

[0177] Convolution kernel size: 3×3

[0178] Step size: 2×2

[0179] Number of output feature maps: 128

[0180] Activation function: ReLU

[0181] Batch Normalization

[0182] Padding: Zero padding

[0183] Layer 10: Convolution

[0184] Convolution kernel size: 3×3

[0185] Step size: 1×1

[0186] Number of output feature maps: 128

[0187] Activation function: ReLU

[0188] Batch Normalization

[0189] Padding: Zero padding

[0190] Layer 11: Convolution

[0191] Convolution kernel size: 3×3

[0192] Step size: 1×1

[0193] Number of output feature maps: 128

[0194] Activation function: ReLU

[0195] Batch Normalization

[0196] Padding: Zero padding

[0197] Layer 12: Convolution

[0198] Convolution kernel size: 5×5

[0199] Step size: 2×2

[0200] Number of output feature maps: 256

[0201] Activation function: ReLU

[0202] Batch Normalization

[0203] Padding: Zero padding

[0204] Layer 13: Convolution

[0205] Convolution kernel size: 3×3

[0206] Step size: 1×1

[0207] Number of output feature maps: 256

[0208] Activation function: ReLU

[0209] Batch Normalization

[0210] Padding: Zero padding

[0211] Layer 14: Convolution

[0212] Convolution kernel size: 3×3

[0213] Step size: 1×1

[0214] Number of output feature maps: 256

[0215] Activation function: ReLU

[0216] Batch Normalization

[0217] Padding: Zero padding

[0218] Layer 15: Convolution

[0219] Convolution kernel size: 5×5

[0220] Step size: 2×2

[0221] Number of output feature maps: 512

[0222] Activation function: ReLU

[0223] Batch Normalization

[0224] Padding: Zero padding

[0225] Layer 16: Convolution

[0226] Convolution kernel size: 3×3

[0227] Step size: 1×1

[0228] Number of output feature maps: 512

[0229] Activation function: ReLU

[0230] Batch Normalization

[0231] Padding: Zero padding

[0232] Layer 17: Convolution

[0233] Convolution kernel size: 3×3

[0234] Step size: 1×1

[0235] Number of output feature maps: 256

[0236] Activation function: ReLU

[0237] Batch Normalization

[0238] Padding: Zero padding

[0239] Layer 18: Convolution

[0240] Convolution kernel size: 3×3

[0241] Step size: 1×1

[0242] Number of output feature maps: 256

[0243] Activation function: ReLU

[0244] Batch Normalization

[0245] Padding: Zero padding

[0246] Layer 19: Convolution

[0247] Convolution kernel size: 5×5

[0248] Step size: 2×2

[0249] Number of output feature maps: 128

[0250] Activation function: None

[0251] Batch Normalization

[0252] Padding: Zero padding

[0253] Output: batch size × encoding (batch size × 1536)

[0254] The corresponding decoder for training the feature extractor can be set as follows:

[0255] Input←batch size×encoding

[0256] Layer 1: reshaped to batch size × 3 × 4 × 128

[0257] Layer 2: ‘Transposed convolution’ = learned upsampling

[0258] Convolution kernel size: 5×5

[0259] Padding: Zero padding

[0260] Step size: 2×2

[0261] Feature maps: 256

[0262] Activation function: ReLU

[0263] Layer 3: Convolution

[0264] Convolution kernel size: 3×3

[0265] Padding: Zero padding

[0266] Step size: 1×1

[0267] Feature maps: 256

[0268] Activation function: ReLU

[0269] Layer 4: Convolution

[0270] Convolution kernel size: 3×3

[0271] Padding: Zero padding

[0272] Step size: 1×1

[0273] Feature maps: 512

[0274] Activation function: ReLU

[0275] Layer 5: ‘Transposed Convolution’

[0276] Convolution kernel size: 5×5

[0277] Padding: Zero padding

[0278] Step size: 2×2

[0279] Feature maps: 512

[0280] Activation function: ReLU

[0281] Layer 6: Convolution

[0282] Convolution kernel size: 3×3

[0283] Padding: Zero padding

[0284] Step size: 1×1

[0285] Feature maps: 256

[0286] Activation function: ReLU

[0287] Layer 7: Convolution

[0288] Convolution kernel size: 3×3

[0289] Padding: Zero padding

[0290] Step size: 1×1

[0291] Feature maps: 256

[0292] Activation function: ReLU

[0293] Layer 8: ‘Transposed Convolution’

[0294] Convolution kernel size: 5×5

[0295] Padding: Zero padding

[0296] Step size: 2×2

[0297] Feature maps: 256

[0298] Activation function: ReLU

[0299] Layer 9: Convolution

[0300] Convolution kernel size: 3×3

[0301] Padding: Zero padding

[0302] Step size: 1×1

[0303] Feature maps: 128

[0304] Activation function: ReLU

[0305] Layer 10: ‘Transposed Convolution’

[0306] Convolution kernel size: 5×5

[0307] Padding: Zero padding

[0308] Step size: 2×2

[0309] Feature maps: 128

[0310] Activation function: ReLU

[0311] Layer 11: Convolution

[0312] Convolution kernel size: 3×3

[0313] Padding: Zero padding

[0314] Step size: 1×1

[0315] Feature maps: 64

[0316] Activation function: ReLU

[0317] Layer 12: ‘Transposed Convolution’

[0318] Convolution kernel size: 5×5

[0319] Padding: Zero padding

[0320] Step size: 2×2

[0321] Feature maps: 64

[0322] Activation function: ReLU

[0323] Layer 13: Convolution

[0324] Convolution kernel size: 3×3

[0325] Padding: Zero padding

[0326] Step size: 1×1

[0327] Feature maps: 32

[0328] Activation function: ReLU

[0329] Layer 14: ‘Transposed Convolution’

[0330] Convolution kernel size: 3×3

[0331] Padding: Zero padding

[0332] Step size: 2×2

[0333] Feature maps: 32

[0334] Activation function: ReLU

[0335] Layer 15: Convolution

[0336] Convolution kernel size: 3×3

[0337] Padding: Zero padding

[0338] Step size: 1×1

[0339] Feature map: 1

[0340] Activation function: None

[0341] Output: batch size × 192 × 256 × 1 (= grayscale value image of input image)

[0342] Therefore, the term "transposed convolution" should not be understood in the mathematical sense as the transposition of the matrix, but rather as the upsampling being trained. In other words, the "scaling" to the original image size is being trained.

[0343] The training of feature extractor and decoder can be determined by the following parameters:

[0344] Optimizer: Adaptive Moment Estimation Optimizer (AdamOptimizer)

[0345] Learning rate: 0.0001

[0346] Exponential Decay: Staircase

[0347] The learning rate decreases like a step function. After 200,000 steps, it is reduced by half until it reaches a minimum of 0.00001, after which it remains constant.

[0348] Batch size: 32

[0349] Initialization of all variables: Xavier initialization

[0350] Loss function: mean square error

[0351] Total epochs learned: 550

[0352] The feature extractor for the first sensor modality may be set up as follows:

[0353] Input ← batch size × 2048 × 3

[0354] Layer 0:

[0355] Layer 0.1: Multilayer Perceptron (MLP)

[0356] Number of functions: 64

[0357] Convolution kernel: 1×3

[0358] Batch Normalization

[0359] Activation function: ReLU

[0360] Layer 0.2: MLPs

[0361] Number of functions: 128

[0362] Convolution kernel: 1×1

[0363] Batch Normalization

[0364] Activation function: ReLU

[0365] Layer 0.3: MLPs

[0366] Number of functions: 1024

[0367] Convolution kernel: 1×1

[0368] Batch Normalization

[0369] Activation function: ReLU

[0370] Layer 0.4: Fully connected

[0371] Output: 512

[0372] Batch Normalization

[0373] Activation function: ReLU

[0374] Layer 0.5: Fully connected

[0375] Output: 256

[0376] Batch Normalization

[0377] Activation function: ReLU

[0378] Layer 0.6: Fully connected

[0379] Output: 9

[0380] No activation function

[0381] Initialize variables with 0

[0382] Layer 0.7: add the identity matrix [1 0 0 0 1 0 0 0 1]

[0383] Output: batch size × 3 × 3

[0384] Layer 1: Input * Output of Layer 0

[0385] Layer 2: MLPs

[0386] Number of functions: 64

[0387] Convolution kernel: 1×3

[0388] Batch Normalization

[0389] Activation function: ReLU

[0390] Layer 3: MLPs

[0391] Number of functions: 64

[0392] Convolution kernel: 1×1

[0393] Batch Normalization

[0394] Activation function: ReLU

[0395] Layer 4: Learning matrix multiplication

[0396] Layer 4.1: MLP

[0397] Number of functions: 64

[0398] Convolution kernel: 1×1

[0399] Batch Normalization

[0400] Activation function: ReLU

[0401] Layer 4.2: MLPs

[0402] Number of functions: 128

[0403] Convolution kernel: 1×1

[0404] Batch Normalization

[0405] Activation function: ReLU

[0406] Layer 4.3: MLPs

[0407] Number of functions: 1024

[0408] Convolution kernel: 1×1

[0409] Batch Normalization

[0410] Activation function: ReLU

[0411] Layer 4.4: Fully connected

[0412] Output: 512

[0413] Batch Normalization

[0414] Activation function: ReLU

[0415] Layer 4.5: Fully connected

[0416] Output: 256

[0417] Batch Normalization

[0418] Activation function: ReLU

[0419] Layer 4.6: Fully connected

[0420] Output: 4096

[0421] No activation function

[0422] Initialize variables with 0

[0423] Layer 4.7: Add the identity matrix (size 64)

[0424] Output: batch size × 64 × 64

[0425] Layer 5: Output of layer 3*layer 4

[0426] Layer 6: MLPs

[0427] Number of functions: 64

[0428] Convolution kernel: 1×1

[0429] Batch Normalization

[0430] Activation function: ReLU

[0431] Layer 7: MLPs

[0432] Number of functions: 128

[0433] Convolution kernel: 1×1

[0434] Batch Normalization

[0435] Activation function: ReLU

[0436] Layer 8: MLPs

[0437] Number of functions: 1536

[0438] Convolution kernel: 1×1

[0439] Batch Normalization

[0440] Activation function: ReLU

[0441] Output: batch size × encoding (batch size × 1536)

[0442] The decoder for the first sensor modality may be set up as follows:

[0443] Input←batch size×encoding

[0444] Layer 1: Fully connected

[0445] Output: 2048

[0446] Activation function: ReLU

[0447] Layer 2: dropout

[0448] Dropout rate: 0.2

[0449] Layer 3: Fully connected

[0450] Output: 2048

[0451] Activation function: ReLU

[0452] Layer 4: Discard

[0453] Discard rate: 0.2

[0454] Layer 5: Fully connected

[0455] Output: 6144 (= 2048 * 3)

[0456] Activation function: None

[0457] Output: batch size × 2048 × 3 (= the same size of the input point cloud)

[0458] The training of the feature extractor and decoder for the first sensor modality may be determined by the following parameters:

[0459] Optimizer: Adaptive moment estimation optimizer

[0460] Learning rate: 0.0001

[0461] Exponential Decay: Staircase

[0462] The learning rate decreases like a step function. After 200,000 steps, it is reduced by half until it reaches a minimum of 0.00001, after which it remains constant.

[0463] Batch size: 32

[0464] Initialization of all variables (except for initialization with 0): Xavier initialization

[0465] Loss function: The chamfer distance between the decoder’s input point cloud and the generated output point cloud.

[0466] Total epochs learned: 550

[0467] The network of all feature transformation units can be set up as follows:

[0468] Input ← batch size × encoding (= image encoding or point cloud encoding) = batch size × 1536

[0469] Layer 1: Fully connected

[0470] Output: 2048

[0471] Activation function: ReLU

[0472] Layer 2: dropout

[0473] Dropout rate: 0.2

[0474] Layer 3: Fully connected

[0475] Output: 1400

[0476] Activation function: ReLU

[0477] Layer 4: Discard

[0478] Discard rate: 0.2

[0479] Layer 5: Fully connected

[0480] Output: 1400

[0481] Activation function: None

[0482] Output: encoding batch size × 1400

[0483] Encoding of modality-specific features: batch size × [0:700]

[0484] Encoding of common features: batch size × [700:1400] The network of all feature retransformation units can be set as follows:

[0485] Optional: I_spec = input encoding of modality specific features

[0486] I_common = encoding of common features

[0487] Optional: I_spec:

[0488] Layer 1: Fully connected (input specific modality)

[0489] Output: 512

[0490] Activation function: ReLU

[0491] Layer 2: dropout

[0492] Dropout rate: 0.2

[0493] Layer 3: Fully connected

[0494] Output: 1024

[0495] Activation function: ReLU

[0496] Layer 4: Discard

[0497] Discard rate: 0.2

[0498] Layer 5: Fully connected

[0499] Output: 1536

[0500] Activation function: None

[0501] Output modality-specific features: batch size × 1536

[0502] I_common:

[0503] Layer 1: Fully connected (input common features)

[0504] Output: 512

[0505] Activation function: ReLU

[0506] Layer 2: dropout

[0507] Dropout rate: 0.2

[0508] Layer 3: Fully connected

[0509] Output: 1024

[0510] Activation function: ReLU

[0511] Layer 4: Discard

[0512] Discard rate: 0.2

[0513] Layer 5: Fully connected

[0514] Output: 1536

[0515] Activation function: None

[0516] Output common features: batch size × 1536

[0517] Output = common features of output + optional specific features of output

[0518] The training of the feature transformation unit can be determined by the following parameters:

[0519] Optimizer: Adaptive moment estimation optimizer

[0520] Learning rate: 0.001

[0521] Exponential Decay: Staircase

[0522] The learning rate decreases like a step function. After 200,000 steps, it decreases by a factor of 0.3 until it reaches a minimum of 0.00001, after which it remains constant.

[0523] Batch size: 32

[0524] Initialization of all variables: Xavier initialization

[0525] Total epochs of study: 400

[0526] Loss function:

[0527] Loss for the first 200 epochs

[0528] 1) Cosine similarity of common features of two modalities

[0529] 2) Mean square error between the “actual image encoding” and the reconverted image encoding

[0530] 3) Mean square error between the “actual point cloud code” and the reconverted point cloud code

[0531] Loss = 1) + 2) + 3)

[0532] Loss after 200 epochs

[0533] 1) Cosine similarity of common features of two modalities

[0534] 2) Mean square error between “actual image data” and generated image data

[0535] 3) Chamfer distance between “actual point cloud” and generated point cloud

[0536] Loss = 1) + 2) + 3)

[0537] The neural network for the classification unit can be set up as follows:

[0538] Layer 1: Fully connected

[0539] Output: 1024

[0540] Activation function: ReLU

[0541] Layer 2: Fully connected

[0542] Output: 1024

[0543] Activation function: ReLU

[0544] Layer 3: Fully connected

[0545] Output: 2048

[0546] Activation function: ReLU

[0547] Layer 4: Fully connected = bottleneck vector

[0548] Output: The number of categories to be classified

[0549] Activation function: None

[0550] This vector stores the mean values ​​of each category and is used for comparison.

[0551] The training of the classification unit can be determined by the following parameters:

[0552] Loss function: Softmax Cross Entropy

[0553] Optimizer: Adaptive moment estimation optimizer

[0554] Learning rate: 0.01

[0555] Exponential decay: The learning rate decreases like a step function. After 200,000 steps, it decreases by a factor of 0.7 until it reaches a minimum of 0.00001 and then remains constant.

[0556] Batch size: 32

[0557] Initialization of all variables: Xavier initialization

[0558] Total epochs of study: 50.

[0559] DRAWINGS REFERENCE NUMBERS

[0560] 101 Training Feature Transformation Unit

[0561] 102 training classification units

[0562] 102a Determination of category

[0563] 102b Identification and storage of the average feature vector for each class

[0564] 103 Generation of measurement data from a sensor of a first sensor modality

[0565] 104 Generation of measurement data from a sensor of a second sensor modality

[0566] 105 Generation of measurement data from a sensor of a third sensor modality

[0567] 106 Provision of measurement data for feature extraction unit

[0568] 107 Extraction of modality-independent features

[0569] 108 Extraction of features by corresponding feature extractors

[0570] 109 Extraction of modality-independent features via feature transformation units

[0571] 110 Classification by non-morphological characteristics of taxa

[0572] 110a forwarding at least one feature vector from the feature extraction unit to the classification unit

[0573] 110b Processing feature vectors through classification units

[0574] 110c Comparison between the feature vector and the previously determined mean feature vector for each class

[0575] 110d Provision of deviation limits

[0576] 110e Determination of individual cosine similarities between eigenvectors and average eigenvectors

[0577] 110f When the deviation limit specified previously is exceeded, the corresponding category label is issued

[0578] 110g When all previously specified deviation limits are exceeded, no object of the class is detected.

[0579] 111 Object Classification

[0580] 112 Training Feature Extractor

[0581] 10. Installation

[0582] 11 Sending unit

[0583] 12 Receiving Unit

[0584] 13 Feature extraction unit

[0585] 14. Feature extractor for measurement data from a sensor of a first sensor modality

[0586] 15. Feature extractor for measurement data from a sensor of a second sensor modality

[0587] 16. Feature extractor for measurement data from a sensor of a third sensor modality

[0588] 17 Feature Transformation Unit

[0589] 17a Neural network for the first sensor modality

[0590] 17b Neural Network for Second Sensor Modality

[0591] 18 Taxa

[0592] 18a Decision Module

[0593] 19 Modality-related input

[0594] 20 images

[0595] 21 LiDAR Point Cloud

[0596] 22 Image Encoding for Feature Extractor

[0597] 23 Point Cloud Encoding for Feature Extractor

[0598] 24. Features independent of mode

[0599] 25 Modality-related features

[0600] 26 Common feature space

[0601] 27 Feature space for modality-dependent features of the first sensor modality

[0602] 28 Feature space for modality-dependent features of the second sensor modality

[0603] 29 Feature Retransformation Unit

[0604] 29a Neural network for the first sensor modality

[0605] 29b Neural Network for Second Sensor Modality

[0606] 30 Decoder for first sensor modality

[0607] 30a Output of decoder for first sensor modality

[0608] 31 Decoder for the second sensor modality

[0609] 31a Output of decoder for second sensor modality

[0610] 32 Image Coding with Feature Retransformation Unit

[0611] 33 Point cloud encoding of feature retransformation unit

Claims

1. A method (100) for classifying an object, in, The method (100) comprises providing (106) measurement data from a sensor to a feature extraction unit (13), The method (100) comprises extracting (107) modality-independent features (24) from the measurement data, The modality-independent feature (24) is independent of the sensor modality of the sensor, so that it is not possible to deduce the sensor modality of the sensor from the modality-independent feature, The method is characterized in that: There is at least one first sensor modality and one second sensor modality, and The method (100) is configured to extract modality-independent features (24) from measurement data from sensors of the first sensor modality in such a way that measurement data from sensors of the second sensor modality can be reconstructed.

2. The method (100) according to claim 1, characterized in that: The feature extraction unit (13) comprises at least one feature extractor (14, 15, 16) for each sensor modality, Therein, the feature extractor (14, 15, 16) extracts (108) features from the measurement data.

3. The method (100) according to claim 2, characterized in that: The feature extractor (14, 15, 16) is a neural network.

4. The method (100) according to claim 3, characterized in that: The first sensor modality is LiDAR, Wherein, the feature extractor (14) of the first sensor modality is a multi-layer perceptron (MLP) network.

5. The method (100) according to claim 2, characterized in that: The second sensor modality is image, Wherein, the feature extractor (15) of the second sensor modality is a convolutional neural network.

6. The method (100) according to claim 2, characterized in that: The feature extraction unit (13) comprises a feature transformation unit (17), The feature conversion unit (17) comprises a neural network for each sensor modality, The feature conversion unit (17) extracts (109) modality-independent features (24) from the features of at least two feature extractors.

7. The method (100) according to claim 2, characterized in that: The method comprises prior training (101) of the feature extraction unit (13) by unsupervised learning.

8. The method (100) according to claim 1, characterized in that: The method (100) includes classifying (110) modality-independent features (24) by a classification unit (18).

9. The method (100) according to claim 8, characterized in that: The method (100) comprises prior training (102) of the classification unit (18) by supervised learning.

10. The method (100) according to claim 8, characterized in that: The method (100) comprises forwarding (110a) at least one feature vector from the feature extraction unit (13) to the classification unit (18), wherein the classification (110) comprises comparing (110c) the received feature vector with a respective previously determined average feature vector for each class, and When the deviation is below a previously specified limit, a corresponding category label is issued (110f).

11. A method for optical distance measurement, characterized in that: The method for optical distance measurement comprises a method for classifying an object (100) according to any one of claims 1 to 10.

12. A device (10) for classifying objects, in, The apparatus (10) is configured to perform the method (100) according to any one of claims 1 to 11.

13. A computer-readable storage medium having a program stored thereon, which, after being loaded into the memory of the computer, enables the computer to execute the method (100) according to any one of claims 1 to 11, together with the device (10) according to claim 12.

Images