Encoder-based object detection using data from an environmental sensor

Abstract

A system for detecting objects using data from at least one environmental sensor (10); an encoder (20) is configured to process encoder input data specifying the data (12) of the at least one environmental sensor (10), whereby a representation (22) is generated from the encoder input data, representing information about located objects contained in the encoder input data. An object detection model (50) is configured to process the generated representation (22) as model input data and to generate object hypotheses (60) from the representation (22) generated by the encoder (20). The object hypotheses (60) each comprise an object position and / or object features, wherein the object features include at least one bounding box and a classification.

Description

[0001] The invention relates to a system for detecting objects based on data from at least one environmental sensor.

[0002] Advanced Driver Assistance Systems (ADAS) and Autonomous Driving (AD) require a precise representation of the vehicle's surroundings. In addition to cameras, point-based sensors (location sensors) such as lidar or radar are used for this purpose. These sensors provide measurements, for example, in the form of point clouds. A lidar sensor, for instance, characterizes each point by a Cartesian coordinate (x, y, z) and the reflection intensity, while a radar sensor provides a polar coordinate (distance, azimuth angle) and other properties such as signal strength, radar cross-section (RCS), elevation angle, etc. The position, orientation, class, and potentially other properties of relevant objects (e.g., cars, trucks, or pedestrians) can then be determined.Traditionally, perception algorithms include an object tracking step (typically based on a Kalman filter), followed by object type classification. Object tracking involves a model that describes how objects appear in the sensor data (e.g., a reflection model for radar sensors or an L-shaped vehicle model for lidar sensors). With the advent of deep learning, traditional perception algorithms are increasingly being replaced by neural networks, and in particular, object detection networks. Typically, an object detection neural network outputs oriented bounding boxes (OBBs) containing the probability of existence, size, orientation, and class of the objects. Only then are the detected OBBs tracked over time.

[0003] One advantage of neural networks is that they enable data-driven system development. However, they are generally not interpretable, and it is often difficult to correct errors in the network observed in the test dataset (the portion of the dataset not used for training) or in the real world. In fact, retraining the network for these specific cases may not resolve the errors and may even degrade performance in instances where the network previously performed well. An error is defined as the incorrect or missed detection of an object.

[0004] On the other hand, classical approaches require considerable development effort to achieve satisfactory performance. However, their behavior is more predictable, and errors can be corrected because the system's behavior is more interpretable than that of neural networks. System improvements can be implemented in a way that demonstrably or with a high degree of certainty (compared to neural networks) addresses the targeted error cases without affecting existing, previously well-functioning scenarios. This is particularly crucial for justifying safety.

[0005] One object of the invention is to provide a novel system for object detection based on data, in particular location data, from at least one environmental sensor, which avoids or mitigates the respective disadvantages of neural networks and classical approaches. In particular, it is an object of the invention to provide a system for object detection based on data from at least one environmental sensor that combines advantageous properties of neural networks and classical approaches while avoiding or mitigating their respective disadvantages. It is desirable that the system, on the one hand, is interpretable or enables the resolution of errors with reasonable effort, while, on the other hand, allowing the exploitation of advantages of data-driven approaches.

[0006] One or more of the problems are solved by the subject matter of the independent patent claims. Advantageous embodiments or further developments of the invention are specified below and in the dependent claims.

[0007] The core of the invention is a hybrid approach combining a neural network and a conventional or classical approach (in particular, an object detection model). According to this approach, a neural network is used to encode, for example, radar data into a hidden representation, and a conventional (or classical) approach performs object detection (object recognition) based on this hidden representation. Ideally, the conventional approach is interpretable and allows for the meaningful handling of error cases. As a result, the hybrid approach is also more interpretable and allows for simpler error handling than a method based solely on a neural network. Here and subsequently, "neural network" always refers to an artificial neural network.

[0008] According to one aspect of the invention, a system for detecting objects using data from at least one environmental sensor is described. The system includes an encoder configured to process encoder input data, which specifies the data from the at least one environmental sensor. Processing the encoder input data includes generating a representation from the encoder input data, wherein the representation represents information about located objects (of the environmental sensor) contained in the encoder input data. The system further includes an object detection model configured to process a representation generated by the encoder as model input data. Processing the model input data includes generating object hypotheses from the representation generated by the encoder, wherein the object hypotheses each comprise or specify an object position and / or object features.The object characteristics include at least one boundary and one classification. The object characteristics can also be referred to as object properties.

[0009] The system can, in particular, be an environmental sensing system for motor vehicles. The environmental sensing system can, for example, be a system for capturing a traffic environment, specifically an environmental sensing system for motor vehicles or a (e.g., stationary) traffic monitoring system. In an environmental sensing system for motor vehicles, the at least one sensor can be a sensor for providing location data about objects in the vicinity of the motor vehicle.

[0010] The system can include a control and evaluation unit for at least one environmental sensor. The control and evaluation unit can include the encoder and the object detection model. The system can include at least one environmental sensor.

[0011] The data from at least one environmental sensor can be location data, for example, location data from an environmental sensor in the form of a radar or LiDAR sensor. Alternatively, the data can also include an image, for example, an image from an image sensor, such as a camera image. The data can be obtained directly from the environmental sensor or can be pre-processed.

[0012] The data is input into the encoder, which converts the data into a representation (encodes the data into a representation). This representation is then passed as input to the object detection model (which can be structured according to a conventional or classical method, for example). Depending on the task of detecting (recognizing) objects, the model then outputs, for example, OBBs along with the probability of existence, size, orientation, position, and class of the objects. Alternatively, the model can also be configured to generate classifications as object hypotheses. The classifications can include or be classification probabilities.

[0013] The combination of the encoder and the object detection model allows for improved performance compared to strictly classical methods. This combination also enables better interpretability compared to approaches based solely on neural networks. Furthermore, the object detection model (e.g., a classical part of the system) facilitates a more robust justification of the system's safety. A particularly advantageous feature is the ability to handle error cases (i.e., to improve the system's handling of these cases) with minimal or no impact on scenarios for which the system already performs well. Another benefit is the ability to design a simpler (less complex) system compared to approaches using an end-to-end neural network (which generates object hypotheses from the data / location data).The system therefore facilitates implementation on integrated (embedded) hardware.

[0014] The system can be adapted to different tasks. For example, the actual object detection can be one task. However, the representation can also be used for one or more tasks, including semantic segmentation, classification, etc. Semantic classification here refers to the classification of individual units of the representation (as opposed to the classification of detected / complete objects).

[0015] In embodiments, the data (from at least one environmental sensor) includes location data. The location data includes at least one point cloud comprising individual points and a grid comprising grid cells.

[0016] Grid cells can contain location data or be empty. Grid coordinates or indices can correspond to spatial coordinates (e.g., relative to the sensor position). Spatial coordinates can correspond to a forward direction and a lateral or transverse direction. For example, in the case of a 3D grid, a grid coordinate can also correspond to a vertical direction.

[0017] The location data can be in the form of a point cloud, specifically a point cloud including features, i.e., features of the points. The location data can be entered into the encoder as a list of (unordered) points as encoder input data. The order or sequence of the points is irrelevant.

[0018] The positioning data can include measured positioning data. This data can be obtained from the environmental sensor or can be pre-processed. Points in a point cloud or "filled" grid cells can correspond to individual positions or reflections, such as reflections of radar signals or LiDAR signals from an environmental sensor. Each point or grid cell can have specific properties (also called features), such as a radar cross-section (RCS) and / or a radial velocity (compensated for the environmental sensor's own motion).

[0019] The encoder can comprise or be at least one artificial neural network. The encoder can be part of a neural network architecture. The encoder can also be referred to as an encoder network. In particular, the encoder can comprise at least one recurrent neural network or at least one neural network with a transformer architecture. Recurrent neural networks are a class of artificial neural networks that have connections between neurons or nodes of one layer and neurons or nodes of the same or preceding layer. A neural network with a transformer architecture is also called a transformer and is a deep learning architecture.

[0020] The encoder can be structured like a decoder in an encoder-decoder model. The encoder can also be structured like an encoder in an autoencoder.

[0021] Preferably, the encoder comprises a first encoder layer and at least one second encoder layer. The first encoder layer is preferably configured to process the encoder input data as input. The second (or a final second) encoder layer is preferably configured to generate the representation.

[0022] The encoder is specifically configured to convert the encoder input data into the representation. In particular, the representation may be a different representation or encoding of the data or location data than the encoder input data. The representation may have a lower dimensionality (dimension or size) than the encoder input data. The encoder may be configured to capture relevant features or patterns (relating to objects) in the encoder input data and reproduce them in the representation.

[0023] In some embodiments, the representation comprises multiple units, or is subdivided into multiple units. Each unit can comprise a feature vector (features). The unit can be identical to the feature vector. A position within the representation is assigned to the unit (e.g., by means of indices or coordinates that designate the unit). Coordinates, indices, or the position of a unit in the grid can correspond to spatial coordinates, in particular Cartesian spatial coordinates (e.g., relative to the position of an environmental sensor). Spatial coordinates can be 2D or 3D.

[0024] In embodiments, the representation comprises at least a point cloud containing individual points, or at least a grid containing grid cells. The points can be units of the representation. The grid cells can be units of the representation. The grid can correspond to the environment of at least one environmental sensor.

[0025] The representation can also be called a hidden or latent representation. It can be a (hidden) representation of the encoder input data. For example, the representation can be a point cloud with feature vectors (features) or a grid with feature vectors (features). These features can also be called hidden features. Units of the representation can be, for example, grid cells (of a grid) or points (of a point cloud). For example, each unit of the representation can contain a feature vector. The feature vector can be associated with the respective unit (e.g., grid cell). The representation can, for example, contain or be a 2D grid or a 3D grid. The encoder output can be the input of the object detection model.

[0026] In embodiments, the representation can comprise multiple grids, each containing grid cells. Each grid can correspond to the environment of at least one environmental sensor. Each grid can be a representation that embodies information about located objects contained in the encoder input data. In particular, information about a located object can be represented in multiple or each of the multiple grids.

[0027] The object detection model is a model or algorithm designed to process the representation generated by the encoder as model input. The object detection model transforms the received representation into object hypotheses. A generated object hypothesis can correspond to a detected object. The generated object hypotheses can be output by the system. The output of the object detection model or system can be a list of object hypotheses or a set of object hypotheses.

[0028] The object hypotheses each comprise an object position and / or object characteristics. The object characteristics include at least one boundary frame and a classification. The object hypotheses represent predictions of objects or their characteristics. In particular, the object hypotheses correspond to probable objects that were located according to the data or positioning data from the at least one environmental sensor.

[0029] In one embodiment, the object hypotheses each comprise a classification of an object type (type of object), an object position (position of the object), dimensions of the object or of a bounding frame corresponding to the object, and / or an orientation of the object or of the bounding frame. The dimensions may, in particular, include length and width. The dimensions may also include height. In one example, the object hypotheses each comprise a classification of an object type, wherein the classification includes a respective classification probability for at least one object class.

[0030] The object detection model can include a (pre-trained) machine learning algorithm. The object detection model can be a machine learning algorithm.

[0031] In a simple case, the object detection model can include a nearest-neighbor algorithm, for example, a nearest-neighbor search algorithm. The object detection model can be a nearest-neighbor(s) algorithm, i.e., a nearest-neighbor(s)-based search. The object detection model can include a database.

[0032] In another case, the object detection model may include a decision tree algorithm, in particular a gradient-boosted algorithm or gradient-boosted tree algorithm.

[0033] In some embodiments, the object detection model comprises a database containing feature vectors along with their associated object data. The feature vectors can be stored individually with their associated object data. Alternatively, the feature vectors can be stored as feature vectors of units within a representation, along with the object data associated with each feature vector or unit. That is, representations can be stored together with the object data associated with each of their units or feature vectors. The database can contain pairs (associations) of representations or feature vectors and the object data associated with each feature vector. The object detection model can be configured to generate object hypotheses from the representation, which includes feature vectors, generated by the encoder, based on the database.The object detection model can be configured, in particular, to generate object hypotheses from the representation generated by the encoder, based on the database, using a nearest-neighbor algorithm. The nearest-neighbor algorithm can be configured to search the database for one or more nearest neighbors to a feature vector of a respective unit in the representation generated by the encoder. The algorithm can be configured to evaluate the associated object data, and in particular, to generate object hypotheses based on this associated object data. The object detection model can be configured to generate object hypotheses by interpolating between the associated object data of multiple nearest neighbors. The nearest neighbors can be searched for or determined unit by unit.The nearest neighbors of a feature vector of a unit in the representation can be searched across unit positions (across representations), meaning specifically among all units of stored representations. For example, the nearest neighbor of a unit / grid cell at position (x1, y1) can be a unit of a stored representation at a different position (x2, y2). The nearest neighbor algorithm is part of the object detection model. Alternatively, the nearest neighbor algorithm can be configured to search the database for one or more nearest neighbors of the representation generated by the encoder. That is, nearest neighbors are searched for in the form of representations as a whole.

[0034] If the algorithm searches for one or more nearest neighbors to the respective feature vectors of units in the representation generated by the encoder, the object hypothesis can be constructed from the respective evaluations of one or more nearest neighbors. For each object position in the object hypothesis, a base position corresponding to a position (or coordinates) of the respective unit within the representation can be considered. The position of a unit in the representation contains information about spatial coordinates, e.g., a (rough) object position. The respective feature vector can contain information about a relative object position, where the relative object position specifies the object position relative to a position (or spatial coordinate) corresponding to the unit.

[0035] By storing feature vectors and their associated object data in the database, resolving specific errors is particularly easy and reliable. For example, to correct a data scenario that is not detected or not detected correctly, the relevant feature vector (or its corresponding representation) can be added to the database along with the expected object data as its associated object data. This feature vector (or representation) then becomes the nearest neighbor (with a distance of zero). Existing scenarios that are already well-processed are hardly affected, also due to the properties of the nearest neighbor search. This is particularly advantageous when specific error scenarios need to be resolved.

[0036] The associated object data can describe, for example, one or more objects and their characteristics (object attributes). The associated object data can specify an object position and / or object attributes. The object attributes include at least one boundary frame and a classification. In particular, the associated object data can describe attributes that correspond to the object attributes of the object hypotheses generated by the object detection model. Thus, they can include, for example, a classification of an object type (type of object), an object position (position of the object), dimensions of the object or a boundary frame corresponding to the object, and / or an orientation of the object or the boundary frame. The dimensions can include, in particular, length and width. The dimensions can also include height.The object position can be a relative object position, specifically in relation to the position (the spatial coordinates) of a unit of the representation. Alternatively, the object position can be an overall object position, specifically in relation to an entire representation.

[0037] The object hypotheses generated by the object detection model can each include an object position determined based on spatial coordinates corresponding to a relevant unit of the representation generated by the encoder, and on a relative object position of stored or interpolated stored object data.

[0038] In some embodiments, the system includes a non-maximum suppression (NMS) algorithm configured to filter the object hypotheses generated by the object detection model. The algorithm may be part of the object detection model.

[0039] For example, if an object can extend across multiple grid cells (units), it can lead to spatially (within the grid) overlapping object hypotheses. These can be filtered using the non-maximum suppression algorithm to identify a likely correct object hypothesis (without overlap with other object hypotheses). For each object, the NMS algorithm can, for example, select the object hypothesis with the highest probability from among the spatially (spatially) overlapping object hypotheses. The NMS algorithm can therefore be configured to select the one with the highest probability of being correct from among the overlapping object hypotheses as the object hypothesis.

[0040] In embodiments, the representation comprises multiple units, each containing a classification vector, where the components of the classification vector are multivalued numbers. In particular, a feature vector of the unit can contain or be the classification vector. The encoder can, for example, comprise a label embedding network or a classification part of a label embedding network, or the encoder can correspond to (be structured accordingly with) a classification part of a label embedding network. The components can, for example, be floating-point numbers. Predefined classes can be assigned their respective classification vectors.

[0041] In a conventional classification vector of one-hot coding, different object types are assigned to different indices or components of the vector, and a component set to 1 corresponds to a classification of the assigned object type. In contrast, label embedding or classification vectors with multi-valued components allows for a better mapping of real-world object types to the classification vectors. In particular, it can be achieved that similar objects are coded with similar classification vectors. A further advantage is that class hierarchies or class relationships can be mapped or represented. Classification vectors can be defined that form a hierarchy in the vector space. Another advantage is that better alignment of the feature space can be achieved, and that the vectors become suitable for the use of foundation models.In particular, the feature space can be better aligned with the semantic meanings of the classes than with one-hot coding. Foundation models can also be used to define the class vectors.

[0042] The encoder can be configured to represent relatively similar objects using relatively similar classification vectors in the representation, and to represent relatively dissimilar objects using relatively dissimilar classification vectors.

[0043] The object detection model can be configured to convert a given classification vector of a unit of representation into a classification of an associated object hypothesis.

[0044] In a nearest-neighbor algorithm, the distance between the classification vectors can be calculated according to the Euclidean distance or according to another distance function, for example, according to least cosine similarity (where the cosine of the angle between the vectors is evaluated).

[0045] In embodiments, the processing of the model input data by the object detection model includes determining a classification of a respective object hypothesis from at least one respective classification vector of the representation. This can be done, in particular, using a nearest-neighbor algorithm, as explained above for feature vectors.

[0046] In some embodiments, the object detection model is configured to determine at least one classification probability for a given object hypothesis based on the distance between a classification vector of the representation and at least one predefined class vector. The classification can correspond to a predefined class vector. The classification probability can also represent a measure of uncertainty for the classification. This provides the advantage of determining an interpretable measure of uncertainty based on the distance between the feature vectors or classification vectors and the class vectors. For example, a classification probability can be determined for each of several classifications of an object hypothesis (e.g., the following classification probabilities can be determined for an object: 20% background, 60% passenger car, 20% truck).

[0047] In embodiments, the encoder comprises at least the first-mentioned, the first two, or all of the following components: - at least one layer set up to convert the encoder input data into a first representation in the form of a grid; - a backbone network set up to convert the first representation into a second representation in the form of multiple grids; - multiple detection heads configured to convert the second representation into a third representation in the form of multiple grids, wherein the multiple detection heads are each configured to convert at least one of the multiple grids of the second representation into a respective grid of the third representation.

[0048] The third representation can be a representation in the form of multiple grids with classification probabilities or with feature vectors describing classifications. A respective detection head can be configured to convert one of its assigned grids from the second representations into a grid from the third representation. A respective detection head can also be configured to convert the multiple grids from the second representations into a grid from the third representation. Grid cells of the first and second grids can each contain feature vectors.

[0049] In embodiments where the object detection model includes the database, the representations stored in the database can correspond to the first representation, the second representation, or the third representation, or the feature vectors stored in the database can correspond to units of the first representation, units of the second representation, or units of the third representation.

[0050] The other components mentioned (i.e., the multiple detection heads, or the backbone network and the multiple detection heads) can also be components of the object detection model, for example.

[0051] The encoder can be created using different methods. It can be created through autoencoder training. For example, the encoder can correspond to a part of a trained autoencoder. The autoencoder can comprise the encoder and an associated decoder. The encoder generates a (hidden) representation from the encoder input data. The decoder uses this representation to reconstruct the encoder input data. The autoencoder can, for example, be trained to produce outputs (from the decoder of the autoencoder) that are as similar as possible to similar inputs (encoder input data). The encoder can be used within the system; that is, the encoder is employed within the system for inference or reasoning.

[0052] Alternatively, the encoder can also be trained without a decoder. For example, the encoder can be trained using contrastive learning. In this approach, the encoder is trained with a set of encoder input data so that similar encoder input data produces similar representations, and dissimilar encoder input data produces dissimilar representations. The advantage is that similar input data can be generated from an existing training dataset through data augmentation in a way that does not alter the desired semantics. Thus, the training dataset can be augmented in a data-driven manner. For example, additive noise on the features and rotations can be used to augment point clouds.

[0053] The encoder training can be self-monitored or unmonitored.

[0054] Alternatively, the encoder can be trained using backpropagation (also known as error feedback) together with the object detection model. In this case, the system is trained end-to-end. This requires that the object detection model is differentiable or that a differentiable approximation for the object detection model can be used.

[0055] The object detection model distinguishes between the phases of training and inference (conclusion). A given dataset is typically divided into a training dataset and a test dataset.

[0056] In a nearest-neighbor algorithm, the object detection model can be trained by having the encoder generate a corresponding representation for each pattern (sample). For each grid cell (each unit) of the representation, the corresponding feature vector, along with the associated object data, can be stored according to the expected output (e.g., an oriented bounding box, OBC). In particular, the feature vector can be associated with the cell's occupancy, the object's position, and object dimensions such as length, width, height, and orientation. Thus, representations with feature vectors, and the associated object data for one or more objects, can be stored in the database.

[0057] During inference, the encoder converts the input data into its corresponding latent representation. Then, for each grid cell (or unit), a nearest-neighbor search finds the feature vector that is most similar / nearest to that of the grid cell in question. Finally, the output of the object detection model is the occupancy, object position, and dimensions associated with the found feature vector. Alternatively, k nearest neighbors can be found, k ≥ 1, and any type of interpolation and / or aggregation can be performed to obtain a nearest neighbor result.

[0058] The following are examples of implementation explained in more detail with reference to the drawing. The drawing shows: Fig. 1 a schematic diagram of a system for detecting objects according to embodiments of the invention; Fig. 2 a schematic diagram of a system for detecting objects according to embodiments using a nearest-neighbor algorithm; Fig. 3 a schematic diagram of a system for detecting objects according to further embodiments of the invention; and Fig. 4 schematic examples of classification vectors.

[0059] In the figures, identical or corresponding features are marked with the same reference numerals.

[0060] The in Fig. 1 The system shown for detecting objects using data from at least one environmental sensor 10 comprises the environmental sensor 10, an encoder 20 and an object detection model 50.

[0061] The encoder 20 is configured to process encoder input data specifying data 12, namely location data from at least one environmental sensor 10 in the form of a point cloud. Processing the encoder input data includes generating a latent representation 22 from the encoder input data. The representation 22 represents information contained in the encoder input data about objects located by the environmental sensor 10.

[0062] The object detection model 50 is configured to process the representation 22 generated by the encoder 20 as model input data. Processing the model input data includes generating object hypotheses 60 from the representation 22 generated by the encoder 20. Each object hypothesis 60 comprises an object position and object features, where the object features include a bounding box and a classification.

[0063] Representation 22 is a grid that comprises units 24 of representation 22 in the form of grid cells. For example, a unit 24 comprises a representation of an object as a feature vector.

[0064] The system optionally includes a non-maximum suppression algorithm 58, which is set up to filter the object hypotheses 60 generated by the object detection model 50.

[0065] The trained encoder 20 can be created using an associated decoder 26. The encoder 20 and the decoder 26 are trained together, with the encoder 20 converting input data 12 into a latent representation 22 and the decoder 26 generating reconstructed data 28 from the latent representation 22, which represent a reconstruction of the input data 12.

[0066] Fig. Figure 2 shows a system for object detection, where the object detection model 50 comprises a database 52 and a nearest-neighbor algorithm 54. Above the dashed horizontal line, the training of the object detection model 50 is shown; below the line, the inference, in particular its practical application, is shown. The application can be described according to the system. Fig. 1 corresponds.

[0067] During the training phase, the system stores associations 55 between latent representations 22 and ground truth data in the form of object data 56. The object data 56 are oriented bounding boxes with positions and classifications. The object data 56 can be stored, for example, as a vector (dx, dy, dz, I, w, h, yaw), where dx, dy, dz are relative positions along the x-, y-, z-axes within the associated grid cell, I, w, h are length, width, and height, and yaw is a yaw angle.

[0068] During the inference phase, the system finds the nearest neighbor of the latent representation 22; more precisely, for each unit 24 of representation 22, the nearest neighbor algorithm 54 finds the nearest neighbor among the units of the representations 22 stored in the database 52. The associated ground truth object data 56 (or interpolated results) are output as object hypotheses 60 in the form of predicted oriented bounding frames.

[0069] Fig. Figure 3 schematically shows a system for object detection that receives data 12 in the form of a point cloud as input and outputs detected boundary frames as object hypotheses 60. The system comprises a radar object detection network as encoder 20. In particular, a classification part of the network can represent the encoder 20.

[0070] The system comprises at least one layer 30 configured to convert the encoder input data into a first representation 32 in the form of a grid. A backbone network 34 is configured to convert the first representation 32 into a second representation 36 in the form of multiple grids with feature vectors. The grids can have different scales and / or sizes and each represents the entire field of view of the sensor 10. Multiple detection heads 38 are configured to convert the second representation 36 into a third representation 40 in the form of multiple grids. The multiple detection heads 38 can each process one grid of the second representation 36, or the entire second representation 36. The representations are converted by the respective detection heads 38 into individual grids of a third representation 40.

[0071] The radar object detection network includes a label embedding network. The latent representations 40 are grids with class probabilities predicted by the detection heads 38.

[0072] Fig. Figure 4 a) schematically shows a classification vector 70 according to a conventional one-hot coding. For k classes, a vector of length k is generated, which has a value of 1 for the corresponding class and a value of 0 for all other classes. Fig. Figure 4 b) shows a classification vector 80, which is defined according to a label embedding procedure with specific values ​​for each class. The individual components are, for example, floating-point numbers.

[0073] A classification procedure then predicts a vector of the same size. This is then transformed (by the object detection model 60) into a corresponding class. According to the description above, the class of the nearest neighbor vector can be chosen (according to the Euclidean distance). However, other distance functions can also be used, e.g., the least cosine similarity.

[0074] Experiments have shown that label embedding can significantly increase detection performance.

[0075] Alternatively, one of the intermediate results of the network can also be used. Fig. 3 can be used as latent representation 20. The label embedding example is a special case where latent representation 40 corresponds to the final output of the neural network.

Claims

[1] System for detecting objects using data from at least one environmental sensor (10), wherein the system comprises: an encoder (20), wherein the encoder (20) is configured to process encoder input data specifying the data (12) of the at least one environmental sensor (10), wherein the processing of the encoder input data includes generating a representation (22) from the encoder input data, wherein the representation (22) represents information about located objects contained in the encoder input data; and an object detection model (50), wherein the object detection model (50) is configured to process a representation (22) generated by the encoder (20) as model input data, wherein the processing of the model input data includes generating object hypotheses (60) from the representation (22) generated by the encoder (20), wherein the object hypotheses (60) each comprise an object position and / or object features, wherein the object features comprise at least one of a bounding frame and a classification. [2] System according to claim 1, wherein the data (12) comprise location data comprising at least one of a point cloud comprising individual points and a grid comprising grid cells. [3] System according to claim 1 or 2, wherein the representation (22) comprises at least a point cloud comprising individual points or at least a grid comprising grid cells. [4] System according to claim 3, wherein the representation (22) comprises several grids, each comprising grid cells. [5] System according to one of the preceding claims, wherein the object detection model (50) comprises a database (52) in which feature vectors are stored together with the object data (56) associated therewith, wherein the object detection model (50) is configured to generate the object hypotheses (60) from the representation (22) generated by the encoder (20), which comprises feature vectors, based on the database (52) by means of a nearest neighbor algorithm (54) or a nearest neighbor algorithm. [6] System according to claim 5, wherein the nearest neighbor algorithm (54) or nearest neighbor algorithm is configured to search in the database (52) for one or more nearest neighbors to a feature vector of a respective unit (24) of the representation (22) generated by the encoder (20). [7] System according to one of the preceding claims, wherein the system comprises a non-maximum suppression algorithm (58) configured to filter the object hypotheses (60) generated by the object detection model (50). [8] System according to one of the preceding claims, wherein the representation (22) comprises several units (24) each comprising a classification vector (80), wherein the components of the classification vector (80) are multivalued numbers, wherein [9] System according to claim 7 or 8, wherein the object detection model (50) is configured to provide at least a classification probability for classifying an object hypothesis (60) to certain based on a distance between a classification vector (80) of the representation (22) and at least one predetermined class vector. [10] System according to any of the preceding claims, wherein the encoder (20) comprises at least the first, the first two or all of the following components: - at least one layer (30) configured to convert the encoder input data into a first representation (32) in the form of a grid; - a backbone network (34) set up to convert the first representation (32) into a second representation (36) in the form of several grids; - several detection heads (38) configured to convert the second representation (36) into a third representation (40) in the form of several grids, wherein the several detection heads (38) are each configured to convert at least one of the several grids of the second representation (36) into a respective grid of the third representation (40).

Images