Classification with improved focus on task at hand
Patent Information
- Application Number
- JP2024148617
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-08-31
- Filing Date
- 2024-08-30
- Publication Date
- 2025-03-14
Smart Images
Figure 2025036368000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to the classification of measured data, such as images, for specific tasks, such as classifying instances of objects, in particular for automated locomotion of vehicles and robots. [Background technology]
[0002] Many automation tasks require classification of measurement data for a task. For example, when a vehicle or robot moves in an at least partially automated manner, objects in the environment of the vehicle or robot need to be classified as to whether they are relevant for planning the future behavior of the vehicle or robot. In quality inspection, objects under test need to be classified as to their quality grade.
[0003] When using trainable classifiers such as neural networks, there is a choice between using a generically trained model or a model specifically trained for the task at hand. A generically trained model may be readily available, but the burden of training for the specific task may be on the user. On the other hand, a specifically trained model will only deliver outputs that are relevant to the task at hand, while a generic model may also deliver outputs that are quite irrelevant to this specific task. Summary of the Invention [Means for solving the problem]
[0004] The present invention provides a method for classifying input measurement data for a given task using a given classifier. The measurement data may include, for example, still or video images, thermal images, ultrasound data, radar or lidar data in the form of point clouds, or time series of any other measurable quantity. The classification maps each record of the measurement data (images, video sequences, points, time series, etc.) to a classification score for one or more classes available in the context of the given task. For example, in any task involving classification of object instances, only a specific set of potential object types is required. An autonomous vehicle, for example, needs to reliably identify pedestrians, but what type of clothing they are wearing is not directly relevant.
[0005] In the course of the method, based on a given task, a relevant subset of input measurement data is identified that has a higher relevance for the given task than the remaining part of the input measurement data. This analysis can be performed at any level of detail and based on any available information source. For example, if objects from a given set of types are to be detected, the relevant subset of input measurement data can be selected to contain information about one object of the given type. However, the analysis is not limited to a semantic interpretation of the content of the input measurement data. Other available prior knowledge can be used as well. For example, in the case of classifying traffic-related objects in an image, areas that may not contain traffic-related objects, such as the inner decoration of a shop window, can be masked from the image.
[0006] Based on the input measurement data and the identified subsets, an enhanced input of the given classifier is determined, in which parts of the input measurement data that correspond to the identified subsets have a higher weight than other content of the input measurement data that does not correspond to the identified subsets. In an extreme example, this may involve restricting the enhanced input to content from the identified relevant subsets. In a less extreme example, the enhanced input may include a mixture of content from the identified relevant subsets and other content. To further emphasize the different weights, in the example of an image as input measurement data, parts that do not belong to the identified relevant subsets can be rendered with lower opacity in the enhanced input.
[0007] The enhanced input is provided to a given classifier. In this way, an output is obtained from the classifier. From this output, a final classification result is determined. In a simple example, the output may be used directly as the final classification result. However, as will be described later, more processing can be applied to the output.
[0008] It has been found that by creating an enhanced input for a given classifier that primarily contains the identified relevant subset of input measurement data, at least some of the advantages of the specially trained classifier can be brought into a setting where a generically trained classifier is used, such that the advantages of this generically trained classifier remain active, i.e., the advantages of the two types of classifiers can be combined, the classifier being easily available and having a high ability to generalize, but at the same time, the output is limited to that which is relevant to the task at hand.
[0009] In a simple example, consider the task of classifying images of people on personality attributes such as intelligence, persistence, aggression, or criminality. A general classifier for images of people may be well trained on images of a large number of people, without then being tailored to the specific task at hand. Rather, it may output classification scores for attributes that are unrelated to the task at hand, such as a person's clothing. Essentially, training a general classifier tries to encode nearly everything visible in the image so that it has the material for any specific classification task that may arise.
[0010] In a particularly advantageous embodiment, determining the enriched input includes cutting out from the input measurement data a portion that includes the identified relevant subset. In this way, a given classifier is essentially limited to processing information relevant to the task at hand without being derailed by other information. This does not only prevent outputs that are irrelevant to the task at hand. Rather, it also prevents the classifier from basing its decision for one of the classes relevant to the task at hand on input image information that is irrelevant to this task. In the example of an image of a person, for a personality attribute sought, the person's face is the relevant part of the image and the decision of the classifier. If the person is wearing a T-shirt with a decorative print or has tucked-in shoulder pads, this is not relevant to the personality attribute. However, if this information is made available to a given classifier, it is still possible that this information will affect the decision in some way. Rather than having to control this behavior after the fact by saliency analysis, it is better to prevent it from the beginning.
[0011] In a further advantageous embodiment, the size of the cropped portion is scaled by a predefined scaling factor α with respect to the size of the identified relevant subset. In this way, in addition to the identified relevant subset, some context may also be included in the enriched input, which can improve the accuracy of the classification. In a simple example, if an image only shows a car and nothing else, it may be difficult to distinguish whether the car is a toy car or a real car. However, if the image additionally shows some context that allows to measure the size of the object, this distinction becomes much easier.
[0012] As mentioned above, in a particularly advantageous embodiment, the input measurement data comprises images and the given task comprises classifying the type of objects shown in these images from a given set of types. Here, the advantages of focusing the enhanced input on the task at hand are very noticeable. If a given classifier is trained generically, this means that it is trained on a set of classes that may be a very large superset of the set of classes relevant to the task at hand. Cropping facilitates the restriction to these relevant classes without the need to apply any modifications to the given classifier itself. If, for example, a classifier provided by another entity in the cloud is used, it may not be possible to apply such modifications at all.
[0013] In a further particularly advantageous embodiment, identifying the relevant subset includes detecting, by a given object detector, a bounding box that encloses an instance of an object of a type from a given set of types. In this way, the enriched input can be tailored to a particular class without already predicting the decision of the classifier. For example, if a bounding box for a car is used to crop the image, the classifier is still free to determine that anything seen within this bounding box is part of a truck.
[0014] Thus, in a further particularly advantageous embodiment, in a setting where a classifier trained on a set of general classes is selected, regardless of the specific type of measurement data, identifying the relevant subset comprises extracting from the input measurement data relevant information for a given subset of the set of general classes, What can be done with bounding boxes in the case of images can also be adapted, for example, to point clouds of radar, lidar or ultrasound data, or to time series of measurements.
[0015] In a further particularly advantageous embodiment, a classifier is selected that accepts the measurement data and the text prompt as input. The classifier then determines a classification score for the class corresponding to the text prompt by evaluating the similarity between the measurement data and the text prompt. Such a classifier can be trained in a most general way for a wide variety of classes. For example, for each image in a set of training images, everything visible in the image can be annotated. The classifier can then be trained in a contrastive way to attribute high similarity scores to matching measurement data and text prompts, but low similarity scores to non-matching measurement data and text prompts.
[0016] In one exemplary embodiment, the measured data is mapped by a data encoder to a data representation in a latent space Z by a classifier. The text prompt is mapped by a text encoder to a text representation in the same latent space Z. The similarity between the measured data and the text prompt can then be evaluated according to the distance between the data representation and the text representation in the latent space Z. This gives full freedom to annotate the same feature seen in the image with multiple attributes, and also encodes the relationship between these attributes. For example, because a data representation is close to a first text representation that is very far from a second text representation, this data representation will also be far from the second text representation.
[0017] Once the classifier is trained in this way, the training is kept as general as possible so that the classifier can be used with many different prompts. This allows the classifier to be used for many different tasks. It also increases the usefulness of the classifier since there are multiple ways to encode the same or similar things in a text prompt. This means encoding as much as possible into the data representation so that the data encoder, who does not know which task it will be used for, is prepared for as many different tasks as possible. In the image example, the encoder records every feature in the input image. The drawback of this is that given any one specific task, the data representation also contains a lot of information that is not relevant to this one task. This is mitigated by selecting a relevant subset from the input measurement data and preparing an enhanced input for the classifier accordingly.
[0018] Apart from focusing the attention of a given classifier on relevant features in the input measurement data, the selection of a relevant subset of the input data can also alleviate the bandwidth shortage for transmitting the measurement data. In particular, if a lot of measurement data is collected on the vehicle, the existing vehicle bus network (such as CAN bus) may not have the capacity to transfer all this measurement data. The situation is similar if the classifier is not mounted on the vehicle itself, but rather is implemented in the cloud and needs to be accessed by a mobile network. Thus, in a further particularly advantageous embodiment, the input measurement data is obtained using at least one sensor mounted on the vehicle. The relevant subset and / or the enriched input, rather than the original input measurement data, is transmitted for further processing via the vehicle bus network of the vehicle and / or via the public land mobile network. In many situations, the relevant subset and / or the enriched input may only have a small fraction of the volume of the original input measurement data from one or more sensors.
[0019] In a further particularly advantageous embodiment, a number of enriched inputs are fed to a given classifier. Determining a final classification result then involves aggregating the output from the classifier for these enriched inputs. For example, the output may be averaged. All these enriched inputs contain identified relevant subsets of the input measurement data, but each of them contains different context information. Thus, in the aggregation result, the identified relevant subsets of the measurement data have a higher weight than the context information, but at the same time, the full diversity of this context information is taken into account.
[0020] This may be further emphasized in a further advantageous embodiment where, in the aggregation towards a final classification result, the output is weighted according to how much of the enhanced input belongs to the relevant subset of the input measurement data, e.g., in the case of an image as input measurement data, it may be possible to measure how many pixels of the enhanced input belong to the identified relevant subset.
[0021] In a further particularly advantageous embodiment, the input measurement data are obtained from at least one sensor. From the final classification result, an action signal is obtained. The vehicle, the driver assistance system, the robot, the quality inspection system, the monitoring system and / or the medical imaging system are operated by the action signal. In this way, the probability that the action performed by the respective action system in response to the action signal is appropriate in the situation characterized by the input measurement data is improved, since a given classifier is not tracked laterally to information that is no longer relevant for the task at hand.
[0022] The method may be wholly or partly computer-implemented and embodied in software. The invention therefore also relates to a computer program having machine-readable instructions, which when executed by one or more computers and / or computing instances, cause the one or more computers and / or computing instances to carry out the above-mentioned method. In this specification, control units for vehicles or robots and other embedded systems capable of executing machine-readable instructions should also be considered as computers. Computing instances include virtual machines, containers or other execution environments that allow the execution of machine-readable instructions in the cloud.
[0023] The non-transitory machine-readable data carrier and / or the download product may include a computer program. The download product is an electronic product that can be sold online and transferred over a network for immediate fulfillment. One or more computers and / or computing instances may be equipped with the computer program and / or the non-transitory storage medium and / or the download product.
[0024] In the following, the invention will be described with the aid of drawings, without intending to limit the scope of the invention. [Brief description of the drawings]
[0025] [Figure 1] FIG. 1 illustrates an exemplary embodiment of a method 100 for classifying input measurement data 3. [Figure 2a] FIG. 2 illustrates the effect of generating an enhanced input 3b for a classifier 1 on the performance of this classifier 1. [Figure 2b] FIG. 2 illustrates the effect of generating an enhanced input 3b for a classifier 1 on the performance of this classifier 1. [Diagram 3] FIG. 2 illustrates an exemplary method for selecting a relevant subset 3a of measurement data 3 and generating an enhanced input 3b. [Figure 4a]FIG. 2 illustrates an exemplary method for generating multiple enhanced inputs 3b from the same measurement data 3. [Figure 4b] FIG. 2 illustrates an exemplary method for generating multiple enhanced inputs 3b from the same measurement data 3. [Figure 5a] FIG. 1 shows how the final achieved classification accuracy on the publicly available ImageNetS919 dataset depends on the object size condition. [Figure 5b] FIG. 1 shows how the final achieved classification accuracy on the publicly available ImageNetS919 dataset depends on the object size condition. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0026] FIG. 1 is a schematic flow chart of an exemplary embodiment of a method 100 for classifying input measurement data 3 for a given task 2 using a given classifier 1 .
[0027] In particular, according to block 105, a classifier 1 trained on a set of common classes can be selected.
[0028] Alternatively or in combination, according to block 106: Accepts measurement data 3 and a text prompt 8 as input; Determine a classification score as output 4 for the class corresponding to the text prompt 8 by evaluating the similarity between the measurement data 3 and the text prompt 8. A classifier 1 configured as follows may be selected.
[0029] According to block 106a, classifier 1: mapping the measured data 3 to a data representation 3* in the latent space Z by a data encoder 1a; Mapping the text prompt 8 to a text representation 8* in the same latent space Z by a text encoder 1b; Evaluate the similarity between the measured data 3 and the text prompt 8 according to the distance between the data representation 3* and the text representation 8* in the latent space Z. The present invention may be configured as follows.
[0030] In particular, the input measurement data 3 may be obtained from at least one sensor.
[0031] In step 110, based on a given task 2, a relevant subset 3a of the input measurement data 3 is identified that has a higher relevance to the given task 2 than the remaining input measurement data 3.
[0032] According to block 111, this identification may involve detecting, by a given object detector 6, a bounding box 7 surrounding an instance of an object of a type from a given set of types. Such bounding boxes can be defined primarily for images as input measurement data 3, but also for other data types such as point clouds.
[0033] According to block 112, relevant information can be extracted for a given subset of the set of general classes on which classifier 1 was trained.
[0034] In step 120, based on the input measurement data 3 and the identified subset 3a, an enhanced input 3b of the given classifier 1 can be determined, in which the part of the input measurement data 3 that corresponds to the identified subset 3a has a higher weight than other contents of the input measurement data 3 that do not correspond to this identified subset 3a.
[0035] According to block 121, determining the enhanced input 3b may include cutting out from the input measurement data 3 a portion that includes the identified relevant subset 3a.
[0036] According to block 121a, the size of the cropped portion may be scaled by a predefined scaling factor α relative to the size of the identified relevant subset 3a.
[0037] In step 130, the enhanced input is provided to a given classifier 1, so that an output 4 is obtained from the classifier 1.
[0038] From this output 4, the final classification result 5 is obtained in step 140.
[0039] According to block 131, multiple reinforced inputs 3b can be fed to a given classifier 1. According to block 141, determining a final classification result 5 can then include aggregating the outputs 4 obtained from the classifier 1 for these multiple reinforced inputs 3b.
[0040] In particular, according to block 141a, in aggregating towards a final classification result 5, the output 4 may be weighted depending on how many of the enriched inputs 3b belong to the relevant subset 3a of the input measurement data 3.
[0041] 1 , in step 150, an action signal 9 is obtained from the final classification result 5. In step 160, the vehicle 50, the driver assistance system 51, the robot 60, the quality inspection system 70, the monitoring system 80 and / or the medical imaging system 90 are operated by the action signal 9.
[0042] Figure 2 shows the effect of generating an enhanced input 3b for this classifier 1 on the performance of this classifier 1, in particular on its accuracy. In the exemplary application analyzed in Figure 2, images as input measurement data 3 are classified with respect to what type of objects they contain. In this simple example, the five available classes are: Sandbar (C1) Canoe (C2) Paddle (C3) Snorkel (C4) Speedboat (C5)
[0043] Fig. 2a shows the classification scores produced by a conventional classifier 1. This classifier 1 includes an image encoder 1a that encodes an image 3 into a data representation 3* in a latent space Z, and a text encoder that encodes text prompts 8 representing classes C1-C5 into representations 8* in the same latent space Z. The classification score 4 for classes C1-C5 depends on the distance between the data representation 3* on the one hand and the text representations 8* corresponding to classes C1-C5 on the other hand. The image 3 shows a water surface 35 with a sandbar 36, a canoe 37 with a paddle 38, and a snorkeler 39.
[0044] Conventional classifier 1 produces classification scores that are very close to each other. There is only a difference of 1.09 between the lowest score of class C5 and the highest score of class C1. The class with the highest score of 25.30 is C1 "Sandbar", which is different from the ground truth class C2 "Canoe" labeled in image 3.
[0045] The reason is that image 3 contains a lot of information about other classes than the desired class C2 "canoe". In particular, image 3 contains a lot of water and a large sandbar, as well as a person with several paddles and a snorkel. According to the ground truth labels, the canoe, which is the main object in image 3, constitutes only a small part of this image 3. The generally trained classifier 1 knows non-zero scores for all classes and their attributes every time it finds each object in image 3. Small differences between the scores simply reflect that each object was recognized differently and well.
[0046] FIG. 2b shows what changes when image 3 is processed by steps 110 and 120 of method 100 into enhanced input 3b. Enhanced input 3b is a crop of original image 3 that emphasizes and enlarges the canoe of class C2 as the main object. The canoe fills the enhanced input 3b much more. As a result, the classification score of class C2 “canoe” is the highest, beating the runner-up score of 24.77 of class C3 “paddle” by 1.26 due to the presence of many paddles in enhanced input 3b. From this score, there is a drop of 1.7 to the next score of class C5 “speed boat”. The difference between the highest score of class C2 “canoe” and the lowest score of class C1 “sandbar” is 6.75 compared to 1.09 obtained for original image 3.
[0047] That is, by keeping the classifier 1 as it is, and simply cropping the image to mainly contain the target object, its classification accuracy can be improved in a zero-shot manner without further training.
[0048] 3 shows the preparation of an enhanced input 3b in another example, where the image 3 includes an instant camera 31, a package of instant camera film 32, a writing pad 33, and a photo frame .
[0049] The images 3 are analyzed using a generically trained classifier 1 with an architecture similar to that shown in Fig. 2a. This classifier 1 may also be identical to the one used later for further processing of the enriched input 3b. The text prompts 8 correspond to the set of generic classes on which the classifier was trained. For each such text prompt 8, a classification score 4 is obtained.
[0050] In the example shown in FIG. 3, text prompts 8# that partially overlap with the general text prompt 8 are relevant to the classification task at hand. For example, the text prompts 8# may relate to different types of cameras, such as instant cameras, compact cameras and reflex cameras. Of the general text prompts 8 that are also in the specialized set 8#, only those with the best classification scores 4 are considered to represent the relevant subset 3a of images as input measurement data 3. For these prompts (and thus classes), the object detector 6 determines bounding boxes 7 of object instances in the images 3. Among several bounding box candidates 7, the bounding box 7* with the highest score is selected. This is then enlarged by a factor α, so that the final bounding box 7′ also includes a bit of context around the respective object. With this final bounding box 7′, in step 120 of the method 100, the original image 3 is cropped and resized to form the enhanced input 3b.
[0051] 3, the only instance of a camera in image 3 is an instant camera 31. As a result, the final bounding box 7' includes this instant camera 31 and a little bit of the context around it.
[0052] FIG. 4 shows how multiple enhanced inputs 3b can be created from the same identified relative subsets 3a of input measurement data 3.
[0053] Figure 4a shows an example of one enhanced input 3b generated from an identified relative subset 3a, which is a crop from the original input measurement data 3, here an image. Compared to the original image 3, the identified relative subset has lost a strip with width Δw around it. The enhanced input 3b is formed by adding to the identified relative subset 3a a strip with width α Δw around it.
[0054] According to FIG. 4b, this is done by using different scaling factors α to form multiple enhanced inputs 3b.k About α k = 1. In each such reinforced input 3b, the weight of any one individual feature decreases with increasing size of the reinforced input 3b.
[0055] Figure 5 shows how the final achieved classification accuracy on the publicly available ImageNetS919 dataset depends on the object size condition. Accuracy A is plotted over the relative object size s. Curve a is the accuracy achieved by applying a prior art classifier 1. Curve b is the accuracy achieved by generating a single reinforced input 3b and providing this to the same classifier 1. Curve c is the accuracy achieved by generating multiple reinforced inputs 3b, providing them all to the classifier 1 and aggregating the results. In Figure 5a, each class corresponds to exactly one text prompt 8, while in Figure 5b, each class corresponds to multiple possible text prompts 8.
[0056] The relative object size, s, is measured as the ratio of the size of the object's bounding box to the total size of the image.
[0057] As can be seen in Figure 5, the use of a single enhanced input 3b focused on the specific classification task at hand significantly improves accuracy A when the relative object size s is small. For larger object sizes s, the benefit decreases and eventually vanishes. This can be understood by summarizing Figure 2. The smaller the size that the object of interest occupies in the image, the more information the image may contain that is not relevant to the classification task at hand. If the object of interest already completely fills the image, little or nothing can be gained by cropping.
[0058] Using multiple enhanced inputs 3b provides only a very small additional advantage for smaller object sizes s. However, for larger object sizes s, the advantage increases.
Claims
1. A method (100) for classifying input measurement data (3) for a given task (2) using a given classifier (1), comprising: - identifying (110) a relevant subset (3a) of said input measurement data (3) based on said given task (2), said relevant subset (3a) having a higher relevance with respect to said given task (2) than the remainder of said input measurement data (3); - determining (120) an enhanced input (3b) for the given classifier (1) based on the input measurement data (3) and the identified subset (3a), such that in the enhanced input (3b) a part of the input measurement data (3) that corresponds to the identified subset (3a) has a higher weight than other contents of the input measurement data (3) that do not correspond to the identified subset (3a); - providing (130) said enhanced input (3b) to said given classifier (1), thereby obtaining an output (4) from said classifier (1); - determining (140) a final classification result (5) from said output (4); The method (100) comprising:
2. 2. The method (100) of claim 1, wherein the step (120) of determining the enhanced input (3b) comprises the step of cutting out (121) a portion from the input measurement data (3) that includes the identified relevant subset (3a).
3. 3. The method (100) of claim 2, wherein the size of the cropped portion is scaled (121a) by a predetermined scaling factor α relative to the size of the identified relevant subset (3a).
4. 4. The method (100) of any one of claims 1 to 3, wherein the input measurement data (3) comprises images and the given task (2) comprises classifying the type of objects shown in these images from a given set of types.
5. The method (100) of any one of claims 2 to 3 and 4, wherein identifying the relevant subset (3a) comprises detecting (111) by a given object detector (6) a bounding box (7) surrounding an instance of an object of a type from the given set of types.
6. A classifier (1) trained on a set of common classes is selected (105); said step of identifying relevant subsets (3a) comprises a step (112) of extracting from said input measurement data (3) relevant information for a given subset of said set of general classes, The method (100) of any one of claims 1 to 5.
7. A classifier (1) is selected (106), said classifier (1) comprising: Accepts measurement data (3) and text prompts (8) as input; determining a classification score for a class corresponding to said text prompt (8) by assessing the similarity between said measurement data (3) and said text prompt (8); The method (100) of any one of claims 1 to 6, further comprising:
8. The classifier (1) mapping said measured data (3) by a data encoder (1 a) into a data representation (3*) in a latent space Z; mapping said text prompts (8) to text representations (8*) in the same latent space Z by a text encoder (1 b); assessing the similarity between the measured data (3) and the text prompt (8) according to the distance between the data representation (3*) and the text representation (8*) in the latent space Z; The method (100) of claim 7, further comprising:
9. the input measurement data (3) are acquired using at least one sensor installed on the vehicle, the relevant subset (3a) and / or the enriched inputs (3b), rather than the original input measurement data (3), are transmitted via a vehicle bus network of the vehicle and / or via a public land mobile network for further processing. The method (100) of any one of claims 1 to 8.
10. A plurality of reinforced inputs (3b) are fed (131) to said given classifier (1); the step of determining the final classification result (5) comprises a step (141) of aggregating the outputs (4) obtained from the classifier (1) for these multiple enriched inputs (3b); The method (100) of any one of claims 1 to 9.
11. The method (100) of claim 10, wherein in the step of aggregating towards the final classification result (5), the outputs (4) are weighted (141 a) depending on how much of the enriched inputs (3 b) belong to the relevant subset (3 a) of the input measurement data (3).
12. said input measurement data (3) being obtained from at least one sensor; From said final classification result (5), an action signal (9) is obtained (150), a vehicle (50), a driver assistance system (51), a robot (60), a quality inspection system (70), a surveillance system (80) and / or a medical imaging system (90) are operated (160) by said operating signal (9), The method (100) of any one of claims 1 to 11.
13. A computer program comprising machine-readable instructions which, when executed by one or more computers and / or computing instances, cause the one or more computers and / or computing instances to perform the method (100) of any one of claims 1 to 12.
14. A non-transitory machine-readable data carrier and / or download product comprising a computer program according to claim 13.
15. One or more computers comprising a computer program according to claim 13 and / or comprising a non-transitory machine-readable data carrier and / or a download product according to claim 14.