Classifying out-of-distribution objects

Abstract

Embodiments of the present disclosure relate to a vehicle, an apparatus, a data carrier, a computer program, a method for classifying an out-of-distribution (OOD) object, and a method for providing object class embeddings for a query model for classifying OOD objects. The method comprises detecting (step a) ) an OOD object in image data, processing (step b) ) image data of the OOD object using a classification model (also referred to herein as "query model"), thereby obtaining an embedding for the OOD object, determining (step c) ) a distance of the embedding for the OOD object to one or more predetermined object classes in an embedding space, and obtaining (step d) ) an object type for the OOD object based on the distances to the object classes in the embedding space.

Description

[0001] Description

[0002] Classifying out-of-distribution objects

[0003] Embodiments of the present disclosure relate to a vehicle, an apparatus, a data carrier, a computer program, a method for classifying an out-of-distribution (OOD) object, and a method for providing object class embeddings for a query model for classifying OOD objects. In particular, embodiments relate to an approach for distinguishing between vulnerable and non-vulnerable OOD objects.

[0004] Object detection (OD) becomes more and more important in connection with autonomous driving and driver assistance systems. It involves identifying and categorizing objects in the vehicle's environment, such as pedestrians, vehicles, traffic signs, and obstacles. This process is essential for making informed decisions and ensuring the safety of both the vehicle and its occupants.

[0005] Object Detection for (H)AD Automotive systems for automated and highly automated driving (AD, HAD) rely on techniques for environment perception. Some technical solutions apply deep neural networks (DNN) for object detection. OD means to process a temporal sequence of sensor inputs, such as images from a video sequence, as input, and from these predict as outputs

[0006] (1 ) localization information, such as bounding boxes or pixel masks, and

[0007] (2) class information of objects in the image. Typical DNN-based OD functions provide for an object candidate a score for how certain this object belongs to a class from a limited set of pre-defined classes (the class logits). High scores are often associated with high “confidence” of the model regarding the class prediction.

[0008] The Need for Out-of-distribution Detection for (H)AD:

[0009] A key method for assuring the safe and efficient operation of autonomous driving systems in conditions outside of their training distribution is out-of-distribution (OOD) detection (as will be detailed in the following). Autonomous driving (AD) and highly automated driving (HAD) systems rely on accurate perception of the environment to ensure safety. The state-of-the-art approach to implementing this perception is using DNNs for object detection (OD), which showed best state-of-the-art (SOTA) performance so far. However, DNNs may not work reliably when presented with inputs that are too different from the training data, referred to as out-of-distribution (OOD), resulting in mediocre performance and safety concerns. For example, unreliable DNN models can lead to dangerous erroneous outputs such as false negatives (=overlooked objects) and false class information (=wrong assumptions on object behavior). A common form of OOD is concept drift, which occurs when novel object classes, such as a shopping cart, have not or rarely occurred in the training data.

[0010] OOD detection typically computes an anomaly score for input images or regions therein, like the regions of detected objects. Current state of the art methods for out-of-distribution detection rely on only using a single frame to compute the anomaly score of unknown objects. Such solutions rely on using transformer object queries which produce instance segmentation mask for detecting OOD object: In these methods each object query acts as a one-vs-all classifier. Using these object queries it becomes feasible to segment OOD objects as pixels that get high anomaly scores from each of the object queries.

[0011] To ensure safety, it is crucial to identify instances of OOD and take appropriate countermeasures: If such unreliable situations are identified, OOD objects can be identified and flagged, and the system can take safety measures. In practice, e.g., the countermeasure includes alerting the driver and / or braking or stopping the vehicle. However, the countermeasure for handling OOD objects may be unsuitable for some OOD objects and traffic situations.

[0012] Hence, there may be a demand for an improved concept of handling OOD objects.

[0013] This demand may be satisfied by the subject-matter of the appended independent claims. Optional embodiments are disclosed by the appended dependent claims. Embodiments provide a method for classifying an out-of-distribution, OOD, object for a computer vision task. The method comprises detecting (step a) ) an OOD object in image data, processing (step b) ) image data of the OOD object using a classification model (also referred to herein as “query model”), thereby obtaining an embedding for the OOD object, determining (step c) ) a distance of the embedding for the OOD object to one or more predetermined object classes in an embedding space, and obtaining (step d) ) an object type for the OOD object based on the distances to the object classes in the embedding space. In this way, the proposed approach allows for a classification or categorization of OOD objects. This, e.g., allows dedicated actions for the vehicle for different OOD objects for more appropriate vehicle maneuvers and encountering OOD objects. So, the proposed approach may improve the handling of OOD objects.

[0014] In some embodiments the object classes include or correspond to object classes for in-distribution objects. In this way, findings of the training phase may be reused for a suitable categorization, i.e. , for obtaining a (more) suitable object type for the OOD object.

[0015] In some embodiments, the distance of the embedding for the OOD object to one or more predetermined object classes is indicative of a distance of the embedding for the OOD object to respective representations of the predetermined object classes in the embedding space.

[0016] A skilled person having benefit from the present disclosure will appreciate that proposed approach may be applied for various computer vision tasks. In automotive applications, examples of the computer vision task may include object recognition, traffic sign recognition, collision avoidance, and / or driver monitoring. As the skilled person may further appreciate, a machine learning model may be applied for the computer vision task. In practice, such models may require a lot of resources in terms of computing power and / or time.

[0017] Some embodiments of the present disclosure therefore suggest that the classification model is a lightweight (machine learning) model in addition to a task-specific model (also referred to herein as “gallery model” or “foundation model”) for the computer vision task.

[0018] So, in some embodiments, the method further comprises, providing a task-specific model for the computer vision task, wherein the classification model is more lightweight than the task-specific model, and wherein the representations of the object classes are obtained by mapping them from the embeddings space of the task-specific model to the embedding space of the classification model. In this way, less resources (computing power and / or time) may be required for the classification of the OOD object then when using the task-specific model for it.

[0019] In practice, the object type may be selected from predefined object types. The predefined object types may be associated with different actions of the vehicle for handling a respective OOD object. In particular, the object types may differentiate between objects for which a collision with the vehicle is acceptable and objects for which the collision with the vehicle is not acceptable.

[0020] For example, the object type may be one of a plurality of different object types including at least one object type for a vulnerable object and at least one object type for a non-vulnerable object. For a vulnerable object, a collision with the vehicle may be considered not acceptable while a collision with a non-vulnerable object may be considered acceptable. Alternatively or additionally, different object types may include at least one object type for an object posing an unacceptable risk to the vehicle and / or passengers in the vehicle and at least one object type for an object posing an acceptable risk to the vehicle and / or passengers in the vehicle.

[0021] In practice, a collision with persons, animals (equally big or bigger than a bunny or cat), and / or other objects of (approximately) of the same or higher weight and / or size may be considered as unacceptable while collisions with smaller, more lightweight, and / or unlively objects may be considered acceptable.

[0022] In such embodiments, obtaining the object type may comprise assigning the object type for a vulnerable object to the OOD object if the distance of the embedding for the OOD object to one or more predetermined object classes in the embedding space is smaller for one or more vulnerable object classes than for one or more non-vulnerable object classes or assigning the object type for a non-vulnerable object to the OOD object if the distance of the embedding for the OOD object to one or more predetermined object classes in the embedding space is smaller for one or more non-vulnerable object classes than for one or more vulnerable object classes.

[0023] In some embodiments, the method further comprises providing the object type of the OOD object to an autonomous driving system for controlling a vehicle based on the object type. In this way, the autonomous driving system may react appropriately when encountering an OOD object.

[0024] For example, the vehicle may be controlled such that a collision with the OOD object is accepted if such collision is acceptable with respect to the object type or such that the vehicle tries to avoid a collision with the OOD object (e.g., by braking or stopping the vehicle) if a collision is unacceptable with respect to the object type.

[0025] In some embodiments, controlling the vehicle comprises controlling the vehicle such that it does not collide with the OOD object if the OOD object is a vulnerable object according to the obtained object type and / or such that a collision with the OOD object is accepted if the OOD object is a non-vulnerable object according to the obtained object type. The same may be applied analogously for other object types for which the collision is acceptable or unacceptable, respectively. In this way, undesired damages to objects, the vehicle, and / or passengers in the vehicle may be avoided and / or at least reduced.

[0026] In some embodiments, the method further comprises checking if the detected OOD object is detected in at least a predefined number of consecutive frames in the image data and ignoring the detection of the OOD object if the OOD object is not detected in at least the predefined number of consecutive frames or executing steps b) to d) if the OOD object is detected in at least the predefined number of consecutive frames. In this way, an impact of false-positives (within less than the predefined number of consecutive frames) on the vehicle’s driving behavior may be avoided or at least reduced.

[0027] Further embodiments provide a method for providing object class embeddings for a query model for classifying OOD objects. The method comprises applying a gallery model or foundation model for classifying objects to image data for obtaining embeddings for predetermined object classes, and mapping, using an appropriately trained bridging model, the embeddings for the predetermined object classes to an embedding space of a query model for classifying objects, wherein the query model is more lightweight than the gallery model and configured to run on an edge device for object classification in real-time. In this way, classification abilities from training the gallery model can be reused for the classification of the OOD objects. In practice, this may be done prior to runtime such that the object class embeddings are available during runtime. In this way, the OOD object classification may be performed in real-time in automotive applications.

[0028] In some embodiments, the method further comprises providing the mapped embeddings for obtaining an object type for an OOD object based on distances between embeddings for the OOD object and the object classes in the embedding space of the query model. This way, the method embeddings are available to the query model during runtime for real-time classification of OOD objects.

[0029] Further embodiments provide a computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out an embodiment of any one of the proposed methods.

[0030] Further embodiments provide a computer-readable data carrier having stored thereon the proposed computer program.

[0031] Further embodiments provide an apparatus comprising one or more interfaces for communication and a data processing circuit configured to execute an embodiment of any one of the proposed methods. Further embodiments provide a vehicle comprising an embodiment of the proposed apparatus.

[0032] Further, embodiments are now described with reference to the attached drawings. It should be noted that the embodiments illustrated by the referenced drawings show merely optional embodiments as an example and that the scope of the present disclosure is by no means limited to the embodiments presented:

[0033] Brief description of the drawings

[0034] Fig. 1 shows a flow chart schematically illustrating an embodiment of a method for classifying an OOD object for a computer vision task;

[0035] Fig. 2 shows a flow chart schematically illustrating another embodiment of the proposed method;

[0036] Fig. 3 shows a flow chart schematically illustrating an embodiment of a method for providing object class embeddings for a query model for classifying OOD objects; and

[0037] Fig. 4 shows a block diagram schematically illustrating an embodiment of an apparatus according to the proposed approach.

[0038] In open world scenarios, is infeasible to train a neural network or object detector to detect all possible classes. So, applications are usually limited to only a few selected classes that are most common in connection with the desired use case - in automotive, e.g., objects frequently occurring in typical traffic situations such as road users classes (e.g., cars, trucks, motorcycles, pedestrians, cyclists, and / or the like). The advantage of this approach is that it provides us the capability to handle many cases that, e.g., situations a car might face while on the road, without explicitly training on all the objects. As mentioned above, therefore, out-of-distribution object detection may be applied to identify unknown or rarely trained objects. However, OOD object detection approaches are only able to localize and detect unknown objects, but they do not offer any information how to handle or treat OOD objects or traffic situations involving OOD objects.

[0039] Present disclosure provides an approach to classify such objects in order to obtain an appropriate response that the AD system should take when encountering such objects. More aspects and details are described below with reference to Fig. 1 .

[0040] Fig. 1 shows a flow chart schematically illustrating an embodiment of a method 100 for classifying an OOD object for a computer vision task.

[0041] In the context of the present disclosure, OOD objects refer to data points that differ significantly from the data the model was trained on. These objects fall outside the distribution of the training dataset, meaning the model has not seen (enough) similar examples during its training phase.

[0042] In contrast, in-distribution objects are data points that fall within the range of the training dataset. These objects share similar characteristics and patterns with the data the model has been trained on. For example, if a model is trained on images of cats and dogs, any new image of a cat or dog would be considered an in-distribution object. The model is expected to perform well on these objects because it has learned to recognize their features during training. This familiarity allows the model to make accurate predictions or classifications for in-distribution objects.

[0043] For instance, if a model is trained to recognize traffic objects using images of these objects, an image of a shopping cart would be considered an out-of-distribution object. The model might struggle to accurately classify or make predictions about such data because it lacks prior exposure to similar examples.

[0044] A skilled person having benefit from the present disclosure will appreciate that the proposed approach may be applied in various applications for out-of-distribution object detection. So, the proposed approach may be not only applied in automotive applications but also in other computer vision tasks, e.g., in the medical sector, healthcare, computer vision, robotics and / or production quality control.

[0045] Method 100 comprises detecting 110 an OOD object in image data. In practice, the image data, e.g., is obtained from one or more cameras. The skilled person will appreciate that the proposed approach may be applied for different kinds of image data (e.g., single or consecutive frames) and various types of cameras or imaging centers. In automotive applications, the image data may be obtained from one or more environmental cameras or imaging sensors attached to a vehicle for capturing the vehicle’s surrounding.

[0046] Detecting 110 and OOD object in the image data may involve identifying images that differ significantly from the training data. Generally, this process may include training a model on a specific dataset and then using one of various techniques to detect anomalies or unfamiliar patterns in new images.

[0047] One common approach is to use a neural network trained on in-distribution data. During inference, the model's confidence scores or output probabilities are analyzed. If the model is highly uncertain or assigns low confidence to all classes (e.g., lower than a predefined threshold), the image is likely OOD. The same may be applied to objects within the image data. Another method involves using autoencoders, which are neural networks trained to reconstruct input images. If an image cannot be accurately reconstructed, it may be OOD.

[0048] A specific example is the use of a neural network (e.g., a convolutional neural network) trained on a training data that containing images of different task-specific object classes. As mentioned above in connection with automotive applications, such task-specific object classes may include different classes of traffic objects such the classes “car”, “truck”, “motorcycle”, “pedestrian”, “cyclist”, “infrastructure object”, and / or the like using images of these objects. When the trained neural network encounters an image of a shopping cart (not included in the training data), it might produce low confidence scores across all classes. Alternatively, an autoencoder trained on the training data set would struggle to reconstruct the horse image accurately, indicating it as OOD.

[0049] Further, method 100 comprises processing 120 image data of the OOD object using a classification model, thereby obtaining an embedding for the OOD object.

[0050] In doing so, the image data may be transformed into a high-dimensional representation known as an “embedding in the latent / embedding space”. This process may involve several layers of the neural network (also referred to herein as machine learning model or short: model) extracting features from the image data, such as edges, textures, and shapes, and combining them into a compact vector that captures the essential characteristics of the image data and the OOD object.

[0051] For an OOD object like a shopping cart in an automotive application, the neural network processes the image through its layers, generating an embedding that represents the shopping cart's features. Since the shopping cart is not part of the training data (e.g., the model was trained on images of cars, pedestrians, and traffic signs), its embedding will likely differ from those of in-distribution objects (see object classes above).

[0052] The latent space may be a multi-dimensional space where each point corresponds to an embedding. In this space, similar objects are located close to each other, while dissimilar objects are farther apart. For example, embeddings of different car models might cluster together, while the embedding of a shopping cart would be farther away from these clusters (than cars), indicating its OOD nature.

[0053] In other words, such embeddings and the latent space help the model generalize and understand the relationships between different objects.

[0054] One idea of the proposed approach is to compare the embeddings of the OOD object with embeddings for one or more predetermined object classes to get an idea of a general nature or behavior of the detected OOD object in order to determine an appropriate action to treat / handle the OOD object. For this, method 100 includes determining 130 a distance of the embedding for the OOD object to one or more predetermined object classes in an embedding space.

[0055] Distances between embeddings of an OOD object and embeddings of object classes may be determined using distance metrics like Euclidean distance or cosine similarity. These metrics measure how far apart the embeddings or their clusters are in the latent space, e.g., with respect to one or more samples or an average for multiple samples. For example, if the neural network processes an image of a shopping cart (OOD) and generates its embedding, it can compare this embedding to those of known object classes (e.g., cars, pedestrians). In context of the proposed approach, the distance between the embedding’s may be seen as a measure of similarity or dissimilarity, respectively. The smaller the distance between embeddings of the OOD object and any predetermined object class, the more similar the OOD object may be to the object class and vice versa.

[0056] In the mentioned example, the shopping cart may be, e.g., more similar to other moving or mobile objects on wheels, e.g., cars or bicycles, and less similar to static objects like traffic signs.

[0057] In practice, the distance of the embedding for the OOD object to one or more predetermined object classes may be indicative of a distance of the embedding for the OOD object to respective representations of the predetermined object classes in the embedding space.

[0058] Method 100 further includes obtaining 140 an object type for the OOD object based on the distances to the object classes in the embedding space.

[0059] As the skilled person will appreciate, the object type may be selected from a plurality of predefined object types. The plurality of object types may be specifically adapted to a desired use case. In practice, the object types may be associated with different actions in response to it. In some embodiments, the object type may correspond to a subordinate object type for predetermined object classes which are handled or treated similarly.

[0060] In some embodiments for automotive applications, the method may further comprise providing the object type of the OOD object to an autonomous driving system for controlling a vehicle based on the object type.

[0061] In automotive applications, the object types may include “vulnerable” and “non-vulnerable”.

[0062] Obtaining the object type may comprise assigning the object type for a vulnerable object to the OOD object if the distance of the embedding for the OOD object to one or more predetermined object classes in the embedding space is smaller for one or more vulnerable object classes than for one or more non-vulnerable object classes. Vulnerable object classes, e.g., may include objects involving animate beings (e.g., persons and / or animals). In other words, the object type “vulnerable” is determined for the OOD object if the distance of the embedding for the OOD object to one or more predetermined object classes in the embedding space is smaller for one or more vulnerable object classes than for one or more non-vulnerable object classes.

[0063] Alternatively or additionally, obtaining the object type may comprise assigning the object type for a non-vulnerable object to the OOD object if the distance of the embedding for the OOD object to one or more predetermined object classes in the embedding space is smaller for one or more non-vulnerable object classes than for one or more vulnerable object classes. In other words, the object type “non-vulnerable” is determined for the OOD object if the distance of the embedding for the OOD object to one or more predetermined object classes in the embedding space is larger for one or more vulnerable object classes than for one or more non-vulnerable object classes.

[0064] The skilled person will appreciate that, alternatively or additionally, further or other object types may be applied. Further object type examples include be “living”, “not living”, “person involved”, “no person involved”, “(gentle) collision with OOD object acceptable” or “(gentle) collision with OOD object unacceptable”, and / or the like.

[0065] The object type for a plastic bag, for example, may be “collision with OOD object acceptable” or “not living” or “no person involved” such that it is just ignored and does not affect the behavior of the vehicle. In contrast, for a three wheeler or tricyclist the object type may be “living”, “person involved”, “collision unacceptable”, or “vulnerable” due to its similarity to bicycles. Accordingly, a collision with the three wheeler or tricyclist may be less acceptable or unacceptable to avoid injuries of persons (e.g., a driver of the three wheeler). So, the vehicle may rather perform an evasive maneuver than except a collision with the three wheeler or tricyclist.

[0066] In other words, controlling the vehicle may comprise controlling the vehicle such that it does not collide with the OOD object if the OOD object is a vulnerable object according to the obtained object type and / or such that a collision with the OOD object is accepted if the OOD object is a non-vulnerable object according to the obtained object type.

[0067] The skilled person will appreciate that, in this way, various object types of OOD objects may be arbitrarily associated with appropriate actions. In this way, the proposed approach may be specifically tailored or adapted to the envisioned computer vision tasks or use case. In practice, medical applications or robotic applications (as well as other applications mentioned herein) may provide other / different object types and / or other actions associated with the respective object types.

[0068] As can be seen, the proposed approach allows for different and, particularly, for more appropriate responses or actions when encountering OOD objects in connection with the computer vision tasks.

[0069] Even though examples explained herein may refer to automotive applications, it is noted that the proposed approach may be analogously applied other applications where the proposed approach may lead to equally improved results. In practice, the object classes may include or correspond to object classes for in-distribution objects. So, in automotive applications, the object classes may include object classes frequently occurring in traffic situations (see above).

[0070] In practice, the computer vision tasks may require plenty of resources, e.g., computing time, computing power etc. According to some embodiments, the classification model may be a dedicated separate model applied in addition to a machine learning model for solving the envisioned computer vision tasks. This, e.g., allows to use a classification model which is more lightweight than a machine learning model used for solving the envisioned computer vision task.

[0071] Accordingly, in some embodiments, the method further comprises, providing a task-specific model (also referred to herein as “gallery model” or “foundation model”) for the computer vision task. In such embodiments, the classification model is preferably more lightweight than the task-specific model.

[0072] As a result, the proposed OOD object classification may be executed in real-time, even on edge devices, e.g., vehicles, with limited computing resources.

[0073] Optionally, the representations of the object classes are obtained by mapping them from the embeddings space of the task-specific model to the embedding space of the classification model. In this way, training results of the task-specific model may be efficiently reused and consistency with the task-specific model may be ensured. For example, in this way inconsistencies during interference time between the task-specific model and classification model may be avoided. That is, e.g., objects considered similar by the task-specific model may be also considered similar by the classification model and vice versa.

[0074] Further aspects and (optional) features are now described with reference to Fig. 2 schematically illustrating another embodiment 200 of the proposed method. In a first step 210, the OOD object (here: “OoD object”) is detected using an OOD detector.

[0075] As the skilled person may appreciate, various OOD (object) detectors may be applied. An exemplary OOD detector, for example, combines a standard object detection model with an OOD detection mechanism. The standard model detects and classifies objects within known categories, while the OOD mechanism evaluates confidence scores or feature representations to flag objects that deviate significantly from the training data. This can involve techniques like thresholding on confidence scores, using auxiliary models trained to recognize anomalies, or leveraging uncertainty estimation methods. The goal is to ensure the detector can reliably identify and handle novel or unexpected objects, enhancing the robustness and safety of applications like autonomous driving or surveillance.

[0076] In another step 220, the OOD object is then tracked through consecutive frames. For this, e.g., and intersection-over-unity (loll) tracker (or any other equivalent tracker) and / or a Kalman filter may be used. In doing so, it may be ensured that detections are consistent and not a false-positive.

[0077] An exemplary loU tracker may follow (OOD) objects across consecutive frames by comparing the overlap between detected bounding boxes. For example, the tracker detects objects in a first (video) frame in the image data and assigns unique IDs to each. In one or more second, subsequent frames, it detects objects again and calculates the loU between each new detection and the existing / detected tracked objects. The loU measures a ratio of the intersection area to the union area of two bounding boxes. If the loU exceeds a predefined threshold, the tracker associates the new detection with the existing or detected (OOD) object, maintaining its ID. This process continues frame by frame, allowing the tracker to follow objects even as they move.

[0078] In this way, it is determined if the detected OOD object can be tracked in at least a predefined number of consecutive frames of the image data. In the present example, the predefined number of consecutive frames, for example, is 10. In other embodiments, any other arbitrary number of consecutive frames can be applied. In doing so, false-positive OOD object detections may be reduced or even avoided.

[0079] Then, if the OOD object is tracked in less than the predefined number of frames, the OOD detection may be ignored or discarded (see step 230). Otherwise, if the OOD object is tracked in at least the predefined number of frames, it is assumed that the OOD object is not a false-positive detection (and steps b) to d) of the proposed method are executed).

[0080] In doing so, an undesired impact of false-positive may be reduced or even avoided. Depending on the application, fewer or more consecutive frames may be observed in order of a desired trade-off of few false-positive detections being assumed to be real OOD objects and few discarded true OOD detections.

[0081] In another step 240, the OOD detection is passed through a light-weight so-called “query model” for producing an embedding for the OOD object.

[0082] In practice, machine learning models for computer vision tasks may be complex and may require a lot of computing resources. For example, a so-called (complex) “gallery model” (trained on a large training data set) and / or a foundation model may be used for computer vision task. According to embodiments of the present disclosure, the “query model” is a more light-weight or less complex machine learning model (trained for the same computer vision task) which has less parameters. In context of the present disclosure and machine learning models, "parameters" can be understood as internal variables that the model learns from the training data. They define the model's structure and behavior, such as weights in neural networks, which adjust the strength of connections between neurons.

[0083] Parameters are optimized during training to minimize the error between the model's predictions and the actual outcomes.

[0084] As it has less parameters, the query model may require less computing resources such as time, computing power, and / or computing costs. Accordingly, the query model may process the OOD object in real-time on an edge device with limited computing power such as a mobile device or a vehicle.

[0085] In step 250, the OOD object or its embedding resulting from processing the OOD objection by the query model is classified according to a predefined number of so-called “neighbors”. In context of the present disclosure, the neighbors may be embeddings and / or embedding clusters for in-distribution objects in the same latent space. For the classification, e.g., a predefined number of the nearest neighbors may be considered, e.g., the k-nearest neighbors. In practice, e.g., k is preferably equal to 5. The skilled person will appreciate that in other applications, also other values for k may be more suitable for a desired application.

[0086] In doing so, the OOD object may be classified in accordance with in-distribution classes. That is, an object type corresponding to a superordinate class or category for the in-distribution classes may be assigned to the OOD object. For example, if the neighbors relate to a vulnerable object (e.g., a person or an animal), the object type for the OOD object may be “vulnerable object”. In other scenarios, other object types may be assigned to the OOD object.

[0087] Once the OOD object has been classified, in another step 260, appropriate measures for the determined object type may be signaled.

[0088] A skilled person will appreciate, that the measures may be different for different applications. In some applications, the measures may include safety measures for handling OOD objects. In robotic applications, a robot may be stopped or brought in a safe operation state. In autonomous driving (AD), a vehicle may brake, stop, and / or perform an evasive maneuver.

[0089] Optionally, if the open parent average) distance to the neighbors are greater than a predefined threshold, the classification is rejected. For obtaining object class embeddings for the proposed OOD object classification, embeddings of the gallery model or foundation model for the computer vision task may be transferred to the latent or embedding space of the query model.

[0090] Accordingly, embodiments of such method (see Fig. 3) comprise applying 310 a gallery model or foundation model for classifying objects to image data for obtaining embeddings for predetermined object classes and mapping 320 the embeddings for the predetermined object classes to an embedding space of a query model for classifying objects.

[0091] For this, an appropriately trained bridging model may be used. For training such bridging model, areas machine learning techniques may be used and different in-distribution objects may be used as samples while respective embeddings obtained from processing images of those in-distribution objects by the query model may be used as ground truth. The trained bridging model, then may be used to transfer or map embeddings obtained from processing in-distribution objects by the gallery or foundation model to the embedding or latent space of the query model. This may be done before shipping and during the production so that the embeddings are available on the edge device after shipping. A skilled person will appreciate that the use of the embedding for the gallery or foundation model and / or the bridging model may lead to more reliable (reference) embeddings for the OOD object classification in the embedding or latent space of the query model. As a result, reliable reference embeddings are quickly available after shipping and during operation. Accordingly, embodiments of the method may further comprise providing the mapped embeddings for obtaining an object type for an OOD object based on distances between embeddings for the OOD object and the object classes in the embedding space of the query model.

[0092] The query model may be trained as follows:

[0093] The main idea here is to train a lightweight query model that can run on the vehicle, and to train a much larger “gallery” model that knows everything and is trained on millions of different images. (Alternatively, a vision foundational model like CLIP can be used). We then train a “bridge” network to learn a transformation from the gallery model to the query model. In an offline setting all our gallery images are processed by the gallery model and transformed using the bridge network into the query model embedding space. The gallery images can be categorized into what the appropriate action is (is it a vulnerable road use -animal-, and therefore the car must stop, or is it a non-critical object that the vehicle can pass over-shopping bag-), such that we can signal the appropriate action.

[0094] The steps can be formalized as follows:

[0095] Step 1 : train a gallery model or use large foundational model for the computer vision task.

[0096] Step 2: train a light-weight query model that can run on edge (for the same computer vision task as the gallery model).

[0097] Step 3: train a bridge network that transforms the embeddings from the gallery embedding space to the query model embedding space.

[0098] -Gallery Indexing-

[0099] Step 4: pass all images that contain any road obstacle which may occur in traffic scenes using the gallery model.

[0100] Step 5: transform all the embeddings from the gallery embedding space (embedding space of the gallery or foundation model) to the query embedding space (embedding space of the query model) using the bridge network.

[0101] Step 6: save all the embeddings in the query embedding space to use during inference.

[0102] Further aspects and details are now explained with reference to an exemplary use case:

[0103] A self driving vehicle of the vehicle is on the highway and a deer appears is on the road. For the neural network (object detector) the deer is an unknown object since it was not included in the training set of the network, and therefore, a self-driving system of the vehicle does not know whether it should stop, maneuver around, or just ignore this unknown object. The proposed approach first localizes such OOD object and tracks it for X (e.g., 10) frames to obtain a time-series of cut-outs of this unknown object. Then using a light-weight querying model the system is able to classify what the appropriate action should be depending on which set of objects / object type it belongs to. In this case the deer belongs to a class / object type of vulnerable road users, so and so the vehicle is stopped until the animal leaves the road, but if the unknown object is a plastic bag, then the autonomous vehicle may simply ignore or move around while traditional approaches may only provide one single default solution for all OOD objects. Thus, the proposed approach may lead to a better handling of OOD objects which may result in a higher level of safety and / or comfort (e.g., because vehicles may only brake if appropriate)

[0104] Further embodiments pertain to a computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out any one of the proposed methods.

[0105] The computer program may be stored on a computer-readable data carrier. Hence, embodiments may provide such data carrier.

[0106] Also, the proposed approach may be implemented in an apparatus as laid out in more detail below with reference to Fig. 4.

[0107] Fig. 4 shows a block diagram schematically illustrating an embodiment of such an apparatus 400. The apparatus comprises one or more interfaces 410 for communication and a data processing circuit 420 configured to execute the proposed method.

[0108] In embodiments, the one or more interfaces 410 may comprise wired and / or wireless interfaces for transmitting and / or receiving communication signals in connection with the execution of the proposed concept. In practice, the interfaces, e.g., comprise pins, wires, antennas, and / or the like. As well, the interfaces may comprise means for (analog and / or digital) signal or data processing in connection with the communication, e.g., filters, samples, analog-to-digital converters, signal acquisition and / or reconstruction means as well as signal amplifiers, compressors and / or any encryption / decryption means. The data processing circuit 420 may correspond to or comprise any type of programable hardware. So, examples of the data processing circuit 420, e.g., comprise a memory, microcontroller, field programable gate arrays, one or more central, and / or graphical processing units. To execute the proposed method, the data processing circuit 420 may be configured to access or retrieve an appropriate computer program for the execution of the proposed method from a memory of the data processing circuit 420 or a separate memory which is communicatively coupled to the data processing circuit 420.

[0109] In practice, the proposed apparatus may be installed on a vehicle. So, embodiments may also provide a vehicle comprising the proposed apparatus. In implementations, the apparatus, e.g., is part or a component of the ADS.

[0110] It should be understood that various modifications and variations may be made without departing from the scope of the present disclosure. Accordingly, the detailed description and specific examples provided herein are intended to be illustrative rather than limiting. Where appropriate, features of different embodiments may be combined or substituted to form further embodiments. The figures and examples are not necessarily drawn to scale, and like numerals in different figures may refer to similar elements. It is to be noted that the present disclosure can be practiced in the absence of any element not specifically disclosed herein, unless otherwise indicated.

Claims

Patent claims1 . A method (100) for classifying an out-of-distribution, OOD, object for a computer vision task, the method (100) comprising: a) detecting (110) an OOD object in image data; b) processing (120) image data of the OOD object using a classification model, thereby obtaining an embedding for the OOD object; c) determining (130) a distance of the embedding for the OOD object to one or more predetermined object classes in an embedding space; and d) obtaining (140) an object type for the OOD object based on the distances to the object classes in the embedding space.

2. The method (100) of claim 1 , wherein the object classes include or correspond to object classes for in-distribution objects.

3. The method (100) of claim 1 or 2, wherein the distance of the embedding for the OOD object to one or more predetermined object classes is indicative of a distance of the embedding for the OOD object to respective representations of the predetermined object classes in the embedding space.

4. The method (100) of any one of the preceding claims, wherein the method (100) further comprises, providing a task-specific model for the computer vision task, wherein the classification model is more lightweight than the task-specific model, and wherein the representations of the object classes are obtained by mapping them from the embeddings space of the task-specific model to the embedding space of the classification model.

5. The method (100) of any one of the preceding claims, wherein the object type is one of a plurality of different object types including at least one object typefor a vulnerable object and at least one object type for a non-vulnerable object.

6. The method (100) of claim 5, wherein the obtaining the object type comprises: assigning the object type for a vulnerable object to the OOD object if the distance of the embedding for the OOD object to one or more predetermined object classes in the embedding space is smaller for one or more vulnerable object classes than for one or more non-vulnerable object classes; or assigning the object type for a non-vulnerable object to the OOD object if the distance of the embedding for the OOD object to one or more predetermined object classes in the embedding space is smaller for one or more non-vulnerable object classes than for one or more vulnerable object classes.

7. The method (100) of any one of the preceding claims, wherein the method (100) further comprises providing the object type of the OOD object to an autonomous driving system for controlling a vehicle based on the object type.

8. The method (100) of claim 7 and any one of the claims 5 and 6, wherein controlling the vehicle comprises controlling the vehicle such that it does not collide with the OOD object if the OOD object is a vulnerable object according to the obtained object type and / or such that a collision with the OOD object is accepted if the OOD object is a non-vulnerable object according to the obtained object type.

9. The method (100) of any one of the preceding claims, wherein the method (100) further comprises checking if the detected OOD object is detected in at least a predefined number of consecutive frames in the image data and:a1 ) ignoring the detection of the OOD object if the OOD object is not detected in at least the predefined number of consecutive frames; or a2) executing steps b) to d) if the OOD object is detected in at least the predefined number of consecutive frames.

10. A method (300) for providing object class embeddings for a query model for classifying OOD objects, the method comprising: applying (310) a gallery model or foundation model for classifying objects to image data for obtaining embeddings for predetermined object classes; and mapping (320), using an appropriately trained bridging model, the embeddings for the predetermined object classes to an embedding space of a query model for classifying objects, wherein the query model is more lightweight than the gallery model and configured to run on an edge device for object classification in real-time.11 . The method (300) of claim 10, wherein the method further comprises providing the mapped embeddings for obtaining an object type for an OOD object based on distances between embeddings for the OOD object and the object classes in the embedding space of the query model.

12. A computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to carry out any one of the methods (100, 300) of any one of the claims 1 to 11 .

13. A computer-readable data carrier having stored thereon the computer program of claim 12.

14. An apparatus (400) comprising:one or more interfaces (410) for communication; and a data processing circuit (420) configured to execute any one of the methods (100, 300) of any one of the claims 1 to 11 .

15. A vehicle comprising the apparatus (400) of claim 14.

Images