Object classification with content and location sensitive classifiers

Abstract

Object classification with content and location sensitive classifiers. A system and method for classifying objects in spatial data using a machine learning model, and a system and method for training a machine learning model are provided. The machine learning model can include a content sensitive classifier, a location sensitive classifier, and at least one outlier detector. The two classifiers can jointly distinguish between whether an object in the spatial data is in-distribution or out-of-distribution. The outlier detection portion can be trained on inlier examples from the training data, while the presence of actual outliers in the input data of the machine learnable model can be simulated in the feature space of the machine learnable model during training. The combination of these portions can provide more robust object classification in the spatial data with respect to outliers without having to increase the size of the training data.

Description

Technical Field

[0001] This invention relates to a system and a computer-implemented method for training a machine-learnable model for object classification in spatial data, such as image data. The invention further relates to a system and a computer-implemented method for classifying objects in spatial data using a machine learning model. The invention further relates to a computer-readable medium comprising data representing instructions for a processor system to perform one or both of the computer-implemented methods. Background Technology

[0002] Object classification is a key enabler in various technological applications such as autonomous driving, where it is used to detect and identify objects in sensor data acquired by one or more sensors integrated into a vehicle. Such classification can also be called "recognition" because it involves identifying the class of objects. For example, other road users (cars, pedestrians, etc.) or traffic signs can be identified in image data acquired by camera sensors. In some cases, the output of object classification can be used to control actuators, such as those involved in vehicle steering and / or braking, for example, to autonomously avoid collisions with other road users. Various other uses exist for object classification; for example, in robotics, it enables robots to recognize objects in their environment and act accordingly, but it can also be used to enable character recognition, pose recognition, image search, and more.

[0003] Generally, objects can be classified in image data acquired by a camera, but also in other types of spatial data acquired by other types of sensors. A specific example is the classification of cars and pedestrians in LiDAR data. Spatial data can generally be two-dimensional, three-dimensional, or even higher-dimensional.

[0004] It is known to use machine learning techniques for object classification. This typically involves training machine-learnable models, such as neural networks, on large training datasets similar to the type of sensor data to which the machine-learnable model is to be applied. So-called deep neural networks (DNNs) represent the current state of techniques in image classification [1]-[4]. They learn to classify images by learning to extract discriminative features that are highly optimized for the classification task given the training data at hand. However, these feature extractors are known to fail to unravel the factors that are independent of each other and can jointly define the object class. Instead, they often take a “shortcut” by focusing only on individual characteristics that do not uniquely define the object class. This can, for example, lead a trained DNN to identify objects by texture rather than shape. Another example is that a trained DNN may focus only on content information such as texture, color, and edges, while ignoring positional information such as the absolute and / or relative positions of object parts.

[0005] This can lead to trained DNNs generating highly confident but incorrect predictions about instances of spatial data, where so-called bias factors (e.g., the aforementioned content information) are absent or where those factors (e.g., the aforementioned location information) are predominantly present, and the DNN has effectively learned to ignore them. This is particularly likely to occur if the trained DNN is applied to instances of spatial data located outside the training data distribution—for example, representing previously unseen outliers. However, in many real-world applications, such outliers may be unavoidable.

[0006] It is known that significantly increasing the amount of training data allows for encountering more possible instances of objects in the spatial data during training, thereby reducing the chance of encountering outliers after deploying the trained DNN. However, there are limitations to such an increase in training data—both practical and intrinsic—because in many real-world applications, the possibility of encountering outliers will still exist. References [1]“ EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks Mingxing Tan et al. [2] Identity Mappings in Deep Residual Networks "Kaiming He et al." [3] “ Billion-scale semi-supervised learning for image Classification “, I. Zeki Yalniz et al., [4] Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution Yunpeng Chen et al. . Summary of the Invention

[0007] The desired outcome is the ability to train machine-learnable models and use machine learning models to classify objects in spatial data, which are more robust to outliers and can be trained without a large amount of training data.

[0008] According to a first aspect of the invention, as defined in claims 1 and 11, a computer-implemented method and corresponding system are provided for training a machine-learnable model for classifying objects in spatial data. According to another aspect of the invention, as defined in claims 8 and 12, a computer-implemented method and corresponding system are provided for classifying objects in spatial data using a machine learning model. According to another aspect of the invention, as defined in claim 10, a computer-readable medium is provided, comprising instructions for causing a processor system to perform the method of any one of claims 1 to 9.

[0009] The above measures are based on the understanding that two fundamental factors may define an object class in spatial data: the presence of the parts that make up the object and their spatial arrangement. The presence of the parts that make up the object can be understood as a "content" factor, and is represented in the spatial data itself by the presence of certain textures, colors, edges, etc. The spatial arrangement of the parts can be called a "position" factor, and can be represented in the spatial data by object parts with a specific relative spatial arrangement, and in some cases, the parts have a specific absolute spatial position. The inventors have recognized that it can be problematic if a machine-learnable model focuses only on one of the two factors, for example, only on content or only on position, or if one factor significantly outweighs the other in the machine learning model's decision-making. That is, this can make the machine learning model susceptible to outliers, where only one of the two factors exists in the spatial data, or where one of the two factors has characteristics that match the objects in the training data ("inner dimension"), while the other factor has completely different characteristics (another "inner dimension" class or "outlier"), as will also be referenced elsewhere. Figure 3 Please provide an explanation.

[0010] This can lead machine learning models to provide highly confident but incorrect object classifications, which could be highly problematic in real-world applications. For example, an autonomous driving system might recognize a yield sign within a triangular architectural feature, potentially causing the vehicle to unintentionally slow down or brake suddenly.

[0011] The above measures address these issues by providing two separate classification components: one component focuses on classifying objects based on content information, and the other focuses on classifying objects based on location information. For this purpose, two separate classification components can be provided in the machine-learnable model, which can then be trained specifically for either content information or location information.

[0012] More specifically, machine-learnable models may include convolutional parts that generate one or more feature maps from instances of spatial data. Such convolutional parts are known in themselves and may include one or more convolutional layers, but may also include other types of layers, such as pooling layers. One or more feature maps may represent the “activations” of the convolutional parts, which, like the spatial data from which they are generated, may contain both content and location information. For example, feature maps may have a spatial dimension that represents the spatial dimension of the spatial data, and activations in a feature map at a particular location may represent the presence of a feature (e.g., specific content) at that particular location.

[0013] The content classification component can be trained on a version of the feature maps from which the location information has been removed, or a version with one or more feature maps removed. This version can also be called a content-specific feature map, and any location information can be omitted or at least significantly reduced in importance. The content classification component can therefore be trained to be content-sensitive and invariant to location information, and can also be referred to elsewhere as a "content-sensitive classifier" (CSC). Similarly, the location classification component can be trained on a version of the feature maps from which the content information has been removed. This version can also be called a location-specific feature map, and any content information can be omitted or at least significantly reduced in importance. The location classification component can therefore be trained to be location-sensitive and invariant to content information, and can also be referred to as a "location-sensitive classifier" (LSC). This separation of classification based on two factors avoids the overall classifier unintentionally biasing towards only one of the two factors and providing highly confident predictions, such as indicating the presence of an object class even if the other factor is missing or indicates a different object class.

[0014] However, classification based on these two factors may not completely reduce the sensitivity of machine learning models to outliers. Therefore, an outlier detection component can be provided as part of the machine-learnable model, which can be trained during training to detect outliers. However, to avoid having to provide explicit outliers in the training data and thus increasing the size of the training data, the outlier detection component can be trained on inline examples, and the presence of actual outliers in the input data of the machine-learnable model can be simulated during training. For this purpose, one or more previously generated feature maps—e.g., generated by a convolutional part, or for a content classification part or a location classification part—can be modified to simulate the presence of outliers. Therefore, it may not be necessary to modify the spatial data itself, in which case it may be difficult to create real outliers, but rather such outliers may be generated in the feature maps during training; that is, it may be easier to generate spatial outliers in the machine-learnable model itself through feature manipulation than to modify the actual input data.

[0015] In practice, machine-learnable models can provide a content-sensitive classifier, a location-sensitive classifier, and at least one outlier detector. The combination of these three components can provide a more robust object classification in spatial data relative to outliers without increasing the size of the training data. That is, object classification can now be separated from content and location, allowing for more informed decisions about object classification. Specifically, if the content classification part indicates the presence of an object in class A, while the location classification part indicates the presence of an object in class B, it can be determined that the actual object may represent an outlier relative to the training data, and therefore the overall classification may be unreliable. Previous image classification could simply indicate the presence of class A if the classifier unintentionally biased towards content information, or class B if the classifier biased towards location information. Separate classification based on content and location can therefore contribute to the interpretability and decipherability of the classification, which may become increasingly important as machine-learnable models grow more sophisticated and are used in more security-conscious applications.

[0016] More generally, two classifiers can be combined to distinguish whether an object in spatial data is within a distribution or outside a marginal distribution. That is, in the former case, the classification outputs of the two classifiers may be consistent, while in the latter case, the classification outputs of the two classifiers may be inconsistent.

[0017] To better detect and distinguish outliers, such as to detect and differentiate more types of outliers and / or detect outliers with greater certainty, an outlier detector can explicitly indicate the presence of an outlier. Accordingly, a decision regarding the classification of an object class can only be made if both classification components classify the input instance into the same class, and the outlier detector indicates that the input instance does not represent an outlier. Therefore, object classification can depend on consensus between the individual classifier and at least one outlier detector. If such consensus is not reached, potential objects may not be classified, which in some cases may involve the potential object being simply ignored, or the object being classified as an unknown object, or other types of actions being taken. This can provide a machine learning model for image classification that allows for interpretable and interpretable decisions. The corresponding outputs of the classifier and (one or more) outlier detectors can together provide such an interpretation, as will also be illustrated with reference to various tables in specific embodiments of this specification.

[0018] Therefore, the above measures enable classification networks to learn feature representations that are particularly sensitive to content and location factors, allowing the network to detect out-of-distribution samples and provide interpretations of their properties. Thus, outliers that share common content and location factors with in-distribution samples can still be identified, even if those factors originate from a mixture of different classes. Crucially, the survey network may be trained only on in-distribution samples, yet it provides explicit clues for interpretable decisions regarding previously unseen outliers.

[0019] Optionally, the machine-learnable model includes a location and content outlier detection component, and the method includes generating pseudo-outlier feature maps for the location and content outlier detection component by modifying feature information contained in one or more previously generated feature maps and associated with both location and content information. As one or the sole outlier detector, the machine-learnable model may include a location and content outlier detection component, which may also be referred to hereinafter as joint marginal distribution outlier detection (“ODJ”), meaning that the outlier detector is trained to detect samples that are inline values ​​in both the location marginal distribution and the content marginal distribution of the training data, but whose association between the marginal distributions is inaccurate because the association between spatial and content features is incorrect. In other words, the outlier detector can detect samples of outlier values ​​as a joint marginal distribution of location and content relative to the classes present in the training data. For this purpose, pseudo-outlier feature maps can be generated by modifying feature information in one or more previously generated feature maps containing location and content information. For example, one or more feature maps generated by the convolutional component can be modified to simulate the presence of actual outliers in the input data of the machine-learnable model. A contour detector can be trained on one or more pseudo-outlier feature maps thus generated to classify objects represented by one or more of the generated pseudo-outlier feature maps as outliers.

[0020] Optionally, the machine-learnable model includes a location outlier detection component, and the method includes generating pseudo-outlier feature maps for the location outlier detection component by modifying feature information contained in one or more previously generated feature maps and associated with location information. The machine-learnable model may include a location outlier detection component, which may also be referred to hereinafter as location edge outlier detection (“ODL”), referring to an outlier detector trained to detect samples that are outliers in the location edge distribution of the training data. For this purpose, pseudo-outlier feature maps can be generated by modifying feature information in one or more previously generated feature maps that contain location information. In some examples, feature maps(one or more) that primarily contain or only contain location information can be modified. For example, location-specific feature maps for a location-sensitive classifier can be modified to simulate the presence of actual outliers in the input data of the machine-learnable model.

[0021] Alternatively, the location outlier detection part can be implemented by the location classification part by providing pseudo-outlier feature maps to the location classification part as part of a separate outlier object class to be learned. Instead of implementing the outlier detector at the edge location as a separate detector, such as a separate head of a deep neural network, the detector can also be implemented by the location-sensitive classifier itself, i.e., configured to classify N object classes by having the location-sensitive classifier learn to classify outliers from the pseudo-outlier feature maps and classify such outliers into a separate class, such as class N+1 for a machine-learnable model.

[0022] Optionally, the machine-learnable model includes a content outlier detection component, and the method includes generating pseudo-outlier feature maps for the content outlier detection component by modifying feature information contained in one or more previously generated feature maps and associated with content information. As one of, or as the sole, outlier detectors, the machine-learnable model may include a content outlier detection component, which may also be referred to below as content edge outlier detection (“ODC”), referring to an outlier detector trained to detect samples that are outliers in the content edge distribution of the training data. For this purpose, pseudo-outlier feature maps can be generated by modifying feature information from one or more previously generated feature maps containing content information. In some examples, one or more feature maps from the convolutional part can be modified, after which positional information can be removed from the modified feature maps(s), for example, by spatial aggregation, to obtain pseudo-outlier feature maps that omit positional information and contain primarily or solely content information.

[0023] Optionally, each of the one or more feature maps generated by the convolutional part has at least two spatial dimensions associated with location information, and the feature values ​​of the one or more feature maps at each spatial coordinate together form a feature vector representing the content information at the corresponding spatial coordinate, wherein: - Removing location information from one or more feature maps involves aggregating one or more feature maps above the spatial dimension to form a content-specific feature map that includes a feature vector; Removing content information from one or more feature maps involves aggregating feature values ​​for each spatial coordinate on one or more feature maps to form a location-specific feature map with at least two spatial dimensions and one feature value channel.

[0024] The above illustrates advantageous methods for generating content-specific and location-specific feature maps. In a specific example, if the convolutional part generates output activations of size HxWxC, where H and W represent height and width, and thus spatial dimensions, and C represents multiple channels, each representing a separate feature (and thus the output activations correspond to a C feature map for each HxWx1), then content-specific feature maps can be aggregated to generate 1x1xC feature maps, while location-specific feature maps can be aggregated to generate HxWx1 feature maps.

[0025] Optionally, modifying one or more previously generated feature maps includes at least one of the following: - Remove feature information from the feature map; - The positions of feature information in the feature map are pseudo-randomly shuffled; - Mix feature information between feature maps of different object classes; - Exchange feature information at different locations in the feature map.

[0026] The above illustrates a favorable approach to generating pseudo-outlier feature maps from feature maps generated by a classifier for a machine-learnable model.

[0027] Optionally, the machine-learnable model is a deep neural network, wherein the convolutional part is the convolutional part of the deep neural network, and wherein the content classification part and the location classification part are corresponding classification heads of the deep neural network. In some embodiments, the corresponding classifiers can be implemented as different heads of the deep neural network. In some embodiments, at least some outlier detectors can also be implemented as corresponding heads of the deep neural network. In other embodiments, individual classifiers and / or outlier detectors can be implemented as separate neural networks, which can be jointly trained on training data. In this regard, it is noted that the advantageous effects of using content-sensitive classifiers, location-sensitive classifiers, and at least one outlier detector can also be obtained using machine-learnable models other than deep neural networks. For example, instead of using deep neural networks for classification and / or outlier detection, the machine-learnable model can be based on Bayesian networks, hidden Markov models, etc.

[0028] Optionally, the input interface of the system configured to classify objects in spatial data is a sensor interface to a sensor, wherein the sensor is configured to acquire spatial data. The system can therefore classify objects in the sensor data, for example, in real-time or pseudo-real-time. For example, the sensor data from which the objects to be classified can be obtained from an image sensor, a LiDAR sensor, or a combination of such sensors.

[0029] Optionally, the system is a control system configured to adjust control parameters based on the object's classification. For example, the control parameters may be linked to actuators, and changing the control parameters may alter the actuator's operation. Thus, the control system can directly or indirectly control the actuators based on the classification results. For example, in a (semi-)autonomous vehicle, the control system may control braking and / or steering to avoid detected pedestrians.

[0030] Those skilled in the art will appreciate that two or more of the above-mentioned embodiments, implementations and / or alternative aspects of the invention can be combined in any manner deemed useful.

[0031] Modifications and variations of any system, any computer-implemented method, or any computer-readable medium may be made by those skilled in the art based on this description, corresponding to the modifications and variations described for another entity in the entities described. Attached Figure Description

[0032] These and other aspects of the invention will become clear from the embodiments and drawings, and will be further illustrated with reference to the embodiments and the accompanying drawings, which are described below as examples, in which... Figure 1 A schematic representation of a system for training a machine-learnable model to classify objects in spatial data is shown. Figure 2 A schematic representation of a method for training a machine-learnable model to classify objects in spatial data is shown. Figure 3 The diagram illustrates the relationship between content and location information in spatial data and object classification within the spatial data. Figure 4 The diagram illustrates the joint space of location (x-axis) and content (y-axis) factors in object classification in spatial data, with the edges of the two factors indicated at the corresponding axes; Figure 5 A schematic representation of an exemplary classification framework is shown, in which input instances are mapped to two factor-sensitive classification branches and three outlier branches, and in which information about factors is removed in the selection branches to generate invariance about the factor in that branch; Figure 6 It shows in Figure 5 Examples of pseudo-outliers generated in the three outlier branches of the example, where labels a, b, c, d indicate the locations in the intermediate network states where information about the elements is encoded, and where pseudo-outliers are obtained by perturbing the locations in four different ways; Figure 7 A schematic representation of a system for classifying objects in spatial data using a machine learning model is shown. Figure 8 A schematic representation of a method for classifying objects in spatial data using a machine learning model is shown. Figure 9 The system is shown as part of a (semi-)autonomous vehicle; and Figure 10 A computer-readable medium containing data is shown.

[0033] It should be noted that the figures are purely illustrative and not drawn to scale. In each figure, elements corresponding to those already described may have the same reference numerals.

[0034] Reference number list The following list of reference numerals is provided to facilitate the interpretation of the drawings and should not be construed as limiting the claims. 20 sensors 22 cameras 40 Actuators 42 Electric motor 60 Physical Environment 80 (semi) autonomous vehicles 100 Systems for Training Machine-Learning Models 160 Processor Subsystem 180 Data Storage Interface 190 Data storage devices 192 training data 194 Data representations of machine-learnable models 196 Data representation of machine learning models 200 Methods for Training Machine-Learning Models 210 Access the training data used for training 220 provides machine-learnable models 225 provides outlier detection functionality. 230 Training machine-learnable models 240 Generate content-specific feature mappings 245 Training Content Classification Section 250 Generate location-specific feature maps 255 Training Location Classification Section 260 Generate pseudo-outlier feature maps 265 Training Outlier Detection Section 212 Accessing the input data used for inference 220 Training machine-learnable models 222 Using machine learning models for inference 230 provides state memory 240 Extract previous internal state information 250 Update the state memory with the current internal state. 300 Objects (personnel) that can be categorized by content and location information. 310 Violation of content and location information 320 Fruits arranged to resemble people's faces The element of the face that was replaced at position 330 340 randomly mixed facial elements 400 Position (Spatial Layout) Edge 410 Content Edge 420 samples within the distribution 430 out-of-distribution samples 440 samples outside the marginal distribution 450 out-of-distribution samples 500 Input Examples 510 Position-Sensitive Classifier 512 Feature Clustering 514 HxWx1 feature maps 516 N+1 Classification 520 Content Sensitive Classifier 522 Space Aggregation 524 1x1xC Feature Mapping 526 N-class classification 528 Outlier Detection Based on Content Edge Distribution 530 Location and Content Outlier Detection Section 532 Flattening 534 HxWxC Feature Mapping 536 Outlier Detection Based on Joint Marginal Distribution 540 Intermediate Feature Mapping Intermediate feature mapping of 600 inner range values Class A, 610 Class B, 620 Intermediate feature mapping of 650 pseudo-outliers 660 Feature Mapping Generated by Removing Information 670 Feature maps generated by shuffling at random positions 680 Feature Mapping Generated Through Content Mixing Between Classes 690 Feature Maps Generated Through Position Swapping 700 Systems for control or monitoring using machine learning models 720 Sensor Data Interface 722 Sensor Data 740 Actuator Interface 742 Control Data 760 Processor Subsystem 780 Data Storage Interface 790 Data storage device 800 Methods for classifying objects in spatial data 810 Accessing Machine Learning Models 820 Access Input Data 830 Generate one or more feature maps 840 Generate content-specific feature mappings 850 Generate location-specific feature maps 860 Generate content-based object classification results 870 Generate location-based object classification results 880 Generate outlier detection results 890 Classifying objects in spatial data 900 Computer-readable media 910 Non-temporary data. Detailed Implementation

[0035] The following is for reference. Figure 1 and Figure 2 Provides a schematic overview of systems and methods for training machine-learnable models to classify objects in spatial data, and references Figure 7 and Figure 9 This provides a schematic overview of systems and methods for classifying objects in spatial data using the resulting machine learning models. Machine-learnable / learning models and their training and use will be discussed in reference. Figures 3 to 6 To explain further, Figure 8 This involves using machine learning models to control or monitor a physical system, which in this example is a (semi-)autonomous vehicle. Figure 10 This refers to computer-readable media that include computer programs.

[0036] Figure 1A system 100 for training a machine-learnable model to classify objects in spatial data is illustrated. Given such spatial data, for example, objects can be classified into different object classes by a person by combining content information and location information contained in the spatial data. System 100 may include an input interface for accessing training data 192 for the machine-learnable model. Training data 192 may include instances of spatial data, wherein the instances of spatial data include objects belonging to different object classes. For example, the spatial data may be image data, and each instance may be an image, which may include one or more objects. In a particular example, the image may be a road image, and the objects contained in the image may be cars, pedestrians, cyclists, road signs, obstacles, the road itself, etc.

[0037] like Figure 1 As shown, the input interface can be composed of a data storage interface 180, which can access the training data 192 from the data storage device 190. For example, the data storage interface 180 can be a memory interface or a permanent storage interface, such as a hard disk or SSD interface; but it can also be a LAN, local area network, or wide area network interface, such as a Bluetooth, Wi-Fi, or Ethernet or fiber optic interface. The data storage device 190 can be an internal data storage device of the system 100—such as a hard disk drive or SSD—but it can also be an external data storage device—such as a data storage device accessible via a network.

[0038] In some embodiments, data storage device 190 may further include a data representation 194 of an untrained version of the machine-learnable model, which can be accessed by system 100 from data storage device 190. However, it will be appreciated that the training data 192 and data representation 194 of the machine-learnable model may also be accessed from different data storage devices, for example, via different subsystems of data storage interface 180. Each subsystem may have the type as described above for data storage interface 180. In other embodiments, the data representation 194 of the untrained version of the machine-learnable model may be generated internally by system 100, for example, based on design and / or architectural parameters for the machine-learnable model, and therefore may not be explicitly stored in data storage device 190.

[0039] System 100 may further include a processor subsystem 160, which can be configured to: during operation of system 100 and as part of training a machine-learnable model, generate content-information-specific feature maps by removing location information from one or more feature maps and training a content classification part on content-information-specific feature maps; generate location-information-specific feature maps by removing content information from the one or more feature maps and training a location classification part on location-information-specific feature maps; as part of the machine-learnable model, provide at least one outlier detection part for detecting outliers in the input data of the machine-learnable model that do not conform to the distribution of the training data; and as part of training the machine-learnable model, generate pseudo-outlier feature maps by modifying one or more previously generated feature maps generated as instances of spatial data to simulate the presence of actual outliers in the input data of the machine-learnable model, and train the outlier detection part on the pseudo-outlier feature maps. It will be appreciated that these aspects of the operation of system 100 will be referred to... Figure 3-6 Further explanation.

[0040] System 100 may further include an output interface for outputting a data representation 196 of a trained machine-learnable model, also referred to as a machine "learned" model, and its data as trained model data 196. For example, also illustrated in the figures, the output interface may constitute a data storage interface 180, wherein in these embodiments, the interface is an input / output ("IO") interface, through which the trained model data 196 can be stored in the data storage device 190. For example, a data representation 194 defining an "untrained" model may be at least partially replaced by the data representation 196 of the trained model during or after training, because the model's parameters—such as weights, hyperparameters, and other types of model parameters—may be adapted to reflect training on the training data 192. In other embodiments, also as... Figure 1 As shown, data representation 196 can be stored separately from data representation 194 defining the "untrained" model. In some embodiments, the output interface can be separate from the data storage interface 180, but in general it can have the type described above for the data storage interface 180.

[0041] Figure 2 A computer-implemented method 200 for training a machine-learnable model to classify objects in spatial data is shown. Method 200 may correspond to... Figure 1 The system 100 operation, but does not need to correspond to Figure 1The operation of system 100 is shown because it can also correspond to the operation of another type of system, apparatus, or device, or because it can correspond to a computer program. Method 200 is shown to include accessing training data 210 in the step entitled "Accessing Training Data for Training" and providing a machine-learnable model 220 in the step entitled "Providing a Machine-Learnable Model". Method 200 is further shown to include generating content-information-specific feature maps 240 in the step entitled "Training the Machine-Learnable Model" and training a content classification part 245 on the content-information-specific feature maps in the sub-step entitled "Generating Content-Information-Specific Feature Maps"; generating a location-information-specific feature map 250 in the sub-step entitled "Generating Location-Information-Specific Feature Maps"; and training a location classification part 255 on the location-information-specific feature maps in the sub-step entitled "Generating Location-Information-Specific Feature Maps". Method 200 is further shown to include, in a sub-step entitled “Providing an Outlier Detection Part” (which is a sub-step of providing 220 of the machine-learnable model), providing 225 at least one outlier detection part as part of the machine-learnable model for detecting outliers in the input data of the machine-learnable model that do not conform to the distribution of the training data. Method 200 is further shown to include, in a sub-step entitled “Generating Pseudo-Outlier Feature Maps” (which is a sub-step of training 230), generating 260 pseudo-outlier feature maps by modifying one or more previously generated feature maps generated as instances of spatial data to simulate the presence of actual outliers in the input data of the machine-learnable model; and in a sub-step entitled “Training the Outlier Detection Part” (which is a sub-step of training 230), training 265 the outlier detection part on the pseudo-outlier feature maps.

[0042] The following uses an image example to illustrate the classification of objects in spatial data. However, it will be understood that the measures described below can also be applied to other types of spatial data that may not be directly considered as images.

[0043] Figure 3 The diagram illustrates the relationship between content and location information in spatial data and object classification within that data. It also illustrates the limitations of deep classifiers, such as deep neural networks, when classifying objects. In other words, broadly speaking, object classes in spatial data can be classified based on two fundamental factors: content, such as the presence of elements / parts that make up the object; and location, such as the spatial placement and arrangement of these parts. Figure 3A real-life example is shown in the form of an image 300 containing a human face. Human observers typically classify a face as a face based on content information, such as textures and edges representing features like eyes, nose, and hair, and location information, such as the spatial arrangement of eyes, nose, and hair.

[0044] A standard classification network can be biased towards the spatial location of elements, and thus could classify the second image 320, thereby identifying the fruit placed in the facial arrangement of the first image 300 as a face. Alternatively, a standard classification network might be biased towards the elements themselves, such as content, and thus classify the third image 330, which has mixed facial elements, as a face. However, these images do not present a challenge for human observers. That is, while humans will to some extent identify all three images as faces, they will recognize that only the first image 300 shows a real face, and the other images only show specific facial attributes.

[0045] The machine-learnable / learning model described in this specification represents a classification framework capable of mimicking human responses. That is, the classification framework can recognize real faces on the one hand, and detect outliers on the other, providing an explanation of their properties, for example, by including real facial elements in incorrect locations. This ability to detect and interpret outliers during classification can be extremely important in many applications and, for example, can be used for proactive labeling to improve network generalization or for quality control in the manufacturing industry. In particular, the described classification framework can learn feature representations that are particularly sensitive to content and location factors. This can provide additional explanations regarding image blurriness in terms of content and location, in addition to standard classification.

[0046] Figure 4 The diagram illustrates the joint space of location (or spatial arrangement, x-axis) and content (y-axis) factors in object classification within spatial data, as location and content jointly define two object classes: face and fruit. Each point in this 2D plot resembles a possible observation sample of the joint distribution of these two factors—in this case, an image—and these two factors are, for example, specific content (y-axis) placed at a particular spatial location (x-axis).

[0047] In this example, two classes (face and fruit) require that both content and location fall within selected edge distributions, whereby the selected edge distributions jointly define the object class. The edge distributions are schematically indicated at the corresponding axes as corresponding graphs, where Figure 400 defines two edge distributions based on location (spatial arrangement)—namely, a face-like arrangement and a vertically stacked arrangement—and where Figure 410 defines two edge distributions based on content—namely, face features (e.g., nose, ears, eyes) and fruit features (e.g., banana, apple, cherry). Throughout this specification, samples where both factors fall within the corresponding edge distributions and then jointly define the object class will be referred to as “in-distribution” samples, see reference numeral 420. These inner values ​​belong to two possible object classes: faces, where content information falls within the “face features” edge distribution and location information falls within the “face-like arrangement” edge distribution; and fruits, where content information falls within the “fruit features” edge distribution and location information falls within the “vertically stacked arrangement” edge distribution. All other samples may be referred to as “out-of-the-box samples” or “outliers”, see reference numbers 430-450. As a group, out-of-the-box samples can be further divided into three distinct parts: 1. Both factors are within their marginal distributions, but their association is an erroneous joint distribution. Sample 430, for example, a sample with content of one class but position of another class. Figure 4 (The area bounded by dashed lines). Figure 3 The second image 320 is an example of this, in which fruit is placed in a facial arrangement. 2. A marginal distribution sample of 440 where one factor is outside its marginal distribution and another factor is inside its marginal distribution. Figure 4 (the striped area in the text). Figure 3 The fourth image 340 in the text is an example of this, which has randomly shuffled facial elements. 3. A complete out-of-distribution sample of 450 where all factors are outside their marginal distributions. Figure 4 (The dashed area in the text).

[0048] In general, the structure of outliers in distributions may be more complex, for example, having more factors in addition to content and location. However, for simplicity, we consider the case of two factors that lead to the three types of outliers mentioned above.

[0049] As will also refer to Figure 5To determine which of these out-of-distribution types a given sample falls into, two separate classifiers can be provided: one sensitive to the content present in the image and insensitive to the location / spatial arrangement of the content (invariant), and another sensitive to the location / spatial arrangement and insensitive to the content in the image (invariant). This can be achieved by intentionally removing or masking information (content or location) that the respective classifiers should be invariant to. Therefore, the classifiers can jointly distinguish between in-distribution and out-of-distribution cases. That is, in the former case, the classification outputs of the two classifiers may be consistent, while in the latter case, their classification outputs may be inconsistent. Accordingly, while location-sensitive and content-invariant classifiers may be... Figure 3 The second image 320 in the example is misclassified as a face. Content-sensitive and position-invariant classifiers may misclassify the second image 320 as a fruit object (this is a misclassification because the "fruit" class can be defined as fruit placed in a vertically stacked arrangement; see also...). Figure 4 Because the two classifiers may be inconsistent, the second image 320 can be identified as an outlier outside the marginal distribution by the classification results of the two classifiers.

[0050] However, these two classifiers may fail to distinguish between two other types of out-of-distribution: joint out-of-distribution and completely out-of-distribution samples. To also distinguish the joint out-of-distribution case, a separate outlier detector can be provided within the classification framework. To train this outlier detector, hard negative examples can be generated in the feature space by mixing information from different samples. This "joint" outlier detector can now detect samples where both content and location information are valid factors for a class, but they are incorrectly associated. For example, Figure 3 The face in the third image 330 has all the elements of a face, but its position has been replaced.

[0051] The aforementioned classifier—also known as an "edge-sensitive" classifier—provides high-confidence classification even in cases of complete out-of-distribution. To address this, additional outlier detectors can be added to each classifier. To predict outliers, each outlier detector can be trained in a manner similar to its corresponding classifier, i.e., by intentionally removing or masking certain information from the internal representation used as input to the corresponding classifier, thus mimicking the presence of outliers in the input of the machine-learnable model by modifying the internal representation. The output of the outlier detector can be used to distinguish between in-range values ​​and corresponding out-of-distribution outliers in the classification results of the edge-sensitive classifier. Notably, the machine-learnable model combining the above classification framework can be trained using only in-distribution samples.

[0052] In some examples, a machine-learnable model can therefore include at least two classifiers and three outlier detectors, which together provide additional information about the content and location factors of an image. More specifically, the output of each of the classifiers and outlier detectors can be used to distinguish between different types of outliers and to provide explanations for ambiguous situations. Such outputs can, for example, be stored or output to an operator, or used to make subsequent decisions. The following is a brief summary of three different outlier detectors and their outputs when applied to different types of input instances. It can be seen that each type of input instance can be uniquely identified by the combination of the outputs of the corresponding outlier detectors: Input Example Outlier detection at the content edge Location edge outlier detection Joint edge outlier detection Inner circumference value No outliers No outliers No outliers Outliers at the content edge Outliers No outliers Outliers Outliers at the edge of location No outliers Outliers Outliers Outliers in joint edges No outliers No outliers Outliers Outliers in two margins Outliers Outliers Outliers .

[0053] Figure 5 A schematic representation of an example classification framework is shown, where input instance 500 is mapped to two factor-sensitive classifiers and three outlier detectors. The corresponding classifiers and outliers can be considered as “branches” of the framework, which in some embodiments can be implemented by different heads of a deep neural network. The two distinct classifiers or classification branches 510, 520 can provide their outputs based on two distinct edges (e.g., location and content), which adds interpretability to the decisions. To further enhance interpretability, both the content-sensitive classifier 510 and the location-sensitive classifier 520 can each be provided with a corresponding outlier detector to predict outliers that are misaligned with the corresponding edge distribution. The outlier detector of the location-sensitive classifier 510 (LSC) can detect outliers relative to location edges, and thus detect those samples that do not fit the known spatial location distribution. Similarly, the outlier detector of the content-sensitive classifier 520 (CSC) can classify unknown or incomplete content as outliers, regardless of its spatial location. Furthermore, it may be desirable to detect outliers on the joint marginal distribution, and thus those samples that are inline values ​​in both marginals but whose association between the marginals is inaccurate. To detect such outliers, a third outlier detector, also known as the "outlier detector on the joint marginal distribution" (ODJ), can be provided. Overall, the framework with three outlier detectors along with two classifiers makes it possible to detect and provide interpretability of outliers belonging to marginal distributions, joint distributions, or complete distributions. Figure 5 The framework is depicted below, with the corresponding components described below.

[0054] Content-Sensitive Classifier (CSC, 520): This classifier's decisions can be based on the content of object elements / parts, independent of their spatial location. Let the input features of CSC branch 520 be mapped... F 输入 The dimension of 540 is H ×W × C ,in H , W , C These are the height, width, and channels of the feature map. Note the spatial resolution. H × W It can capture spatial information, and channels C It can encode feature representations, and such H × W × C The input feature map can also be considered as representing H × W A C-tuple of ×1 feature maps. When spatial information is removed from the channels, a content-sensitive classifier can be guided to respond to features encoded in different channels, regardless of their spatial location. For this purpose, spatial information... H × W Can cross channels C Aggregated to 522, in order to remove spatial information and allow the classifier to be based on 1×1× C Spatial gathering channels of varying sizes F c The 524 encoded feature representations are used to make decisions, which can also be referred to elsewhere as content-specific feature mappings. F 输入 In the passage k Spatial aggregation at a location can be formulated as follows: .

[0055] Outlier Detection at Content Edges (ODC): Given a CSC 520 that can classify input samples into one of the object classes, it might be desirable to provide an outlier detector to identify outliers where content lies outside the edge distribution. When using only inline samples from the training data, it might be desirable to detect outliers that were not seen during training. The following explains how potential outliers can be generated and how an outlier detector can be trained in a self-supervised manner. That is, by augmenting intermediate feature maps before spatial aggregation 522. F 输入 540 examples of hard negatives are used to generate outliers. Here, samples where all content is present can be considered inline values, while samples with missing, incomplete, and / or mismatched content can be considered outliers. By removing some information or mixing sample content from one class with sample content from another class, improvements can be made to the feature map. F 输入 This generates such outliers. For example, it can be achieved by using values ​​from different classes. F 输入 ( Not a class1) Replacement of the same patch size from the feature map F 输入 ( kind 1) The size is h × w × C The patch is used to achieve mixing, where and For example, information can be removed by setting all values ​​in the patch to 0. Such self-generated outliers in the feature space can be called pseudo-outliers. F 伪 These outlier sets can be generated in each training iteration, and the outlier detector can be trained on the ingress and pseudo-outlier feature maps. Note that the CSC can be trained only on the valid ingress training data, and the pseudo-outliers can be used only to train the outlier detector (ODC).

[0056] Location-Sensitive Classifier (LSC, 510): This classifier makes decisions based on the spatial location of parts / elements of an object, rather than their content. Consider classes with dimensionality... H × W × C LSC input feature mapping F 输入 540. Spatial resolution H × W It can contain spatial information, and channels C Feature representations can be encoded. It may be desirable to capture only spatial information, and thus potentially discard feature representations. Similar to spatial aggregation in CSC branch 520, feature aggregation 512 can be applied to intermediate feature maps. F 输入 540, to integrate content information, thereby generating dimensional... H × W The feature map of ×1 514. In F 输入 The feature aggregation at each position i, j in the matrix can be formulated as follows: (for This feature aggregation may weaken content representation but does not affect spatial information. Classifier branches with this feature aggregation as components ( Figure 5 Branch 510 in the middle can be sensitive only to the spatial location of the object elements used for decision-making, regardless of their exact contents.

[0057] Outlier Detection by Location Edges (ODL): This outlier detector classifies samples with unknown or incomplete spatial locations of object elements as outliers because they may not correspond to known location edge distributions. Similar to ODC, outlier detection can be trained in a self-supervised manner. For example, it can be trained from clustered feature maps. 514. Generating pseudo-outliers. As mentioned above, latent outliers are samples with unknown or incomplete spatial arrangements. Such outliers can be identified from the feature map by removing some information or randomly shuffling their positions. Generation. It was found that ODL may not need to be implemented as a separate outlier detector, but outlier detection can be implemented as an additional class in the classification branch 510. It is assumed that learning the spatial arrangement may be "easier" than learning different feature representations, and therefore the former can be included as an additional class in the classification branch, which in Figure 5 The diagram illustrates the classification of N+1 classes 516 compared to N classes 526 in other classification branches. Therefore, the inline values ​​can provide their class values ​​as the underlying ground truth, while pseudo-outliers can be treated as outlier classes. For example, the standard cross-entropy loss can be used to train the inline values, while the loss for pseudo-outliers can include the standard cross-entropy loss on the outlier classes, and additionally, the entropy of pseudo-outliers on the inline classes can be increased. In a specific example, the outlier classification loss can be as follows: Here, P 离群值 P 类 These are the probabilities of outliers and inliers. In a specific example, It can be set to 0.05. Treating outliers as additional classes and calculating the above loss can improve outlier detection. As initial empirical evidence shows, this may not hold true for other outlier detectors.

[0058] Joint Marginal Distribution Outlier Detection (ODJ): This outlier detector treats samples that share similar attributes with the training data as inline values ​​and the remaining samples as outliers. Similar to ODC, pseudo-outliers can be generated from samples with dimensionality... H × W × C The intermediate feature map 540 is generated. In ODC, pseudo-outliers may be affected by spatial clustering 522, which may lead to a loss of spatial information. On the other hand, ODL can simulate pseudo-outliers from clustered feature maps and may lack clues about content information. Unlike these two cases, ODJ may be expected to be sensitive to both content and location edges. Therefore, after flattening 532, the dimension is... H × W × CThe entire feature map 540 can be used as input for outlier detection, ensuring that both content and spatial information persist. Augmentation strategies can be similar to ODC, where some information can be removed or information can be mixed between inter-class features. In addition to the above strategies, patch size... h × w × C Either of the two positions can also be swapped within the same feature map. This additional example allows the outlier detector to be sensitive to changes in either of the two edges.

[0059] Figure 6 It shows as Figure 5 Examples of pseudo-outlier values ​​generated by an outlier detector are provided, where labels a, b, c, and d represent corresponding activations in intermediate feature maps, and where pseudo-outlier values ​​can be obtained by perturbing the location and / or content in four different ways. More specifically, Figure 6 Intermediate feature maps 600 are shown for the inner boundary values ​​of class A 610 and class B 620. Class A is shown as labels a and b included in the depicted spatial arrangement, while class B is shown as labels c and d included in the depicted spatial arrangement. Pseudo-outliers can be generated by modifying one or both intermediate feature maps 600 to create an intermediate feature map 650 for pseudo-outliers. For example, intermediate feature map 660 can be generated by removing information from intermediate feature map 610—in this case, label b. Another example is intermediate feature map 670, which can be generated by randomly shuffling the positions of the labels, thereby moving labels a and b to new positions. Yet another example is intermediate feature map 680, which can be generated by mixing content between classes, for example, by replacing label c in intermediate feature map 620 with label b from intermediate feature map 610. Yet another example is intermediate feature map 690, which can be generated by swapping the positions of labels a and b in intermediate feature map 610.

[0060] Regarding training, note that inline examples from the training data and any generated pseudo-outliers can be used to train the outlier detectors. All three outlier detectors and the two classifiers can be trained independently of each other, but can be trained within the same training period, such as in the same or separate iterations of the training period. Specific examples of training parameters and architecture / network parameters are described below, but these examples are merely illustrative and entirely non-limiting.

[0061] In specific examples, both CSC and ODJ can be trained over 100 epochs starting with a learning rate of 0.001, with the learning rate decreasing to 0.1x after 80 epochs. It was found that LSC can be trained in fewer epochs than other branches. In specific examples, training with only a learning rate of 0.001 for 25 epochs is sufficient. In specific examples, a batch size of 128 and an Adam optimizer without weight decay can be used. In specific examples, Xavier initialization can be used to initialize the network weights. In specific examples, mean squared error loss and tanh activation function can be used for outlier detection in both CSC and ODJ. Here, the labels can also be flipped with a probability of 0.1 in each batch of training, thus flipping the labels of inner samples as outliers and flipping the labels of self-generated outliers as inner samples to avoid network overfitting on inner samples.

[0062] In a specific example, for CSC, training can begin with only the classification loss up to epoch 8, and then include the outlier detection loss. This setup allows for pre-training of the weights and stabilization of the outlier detection training. A classifier similar to CSC can be used in the ODJ branch to pre-process the weights up to epoch 8, and then use both the classification loss and the outlier detection loss to stabilize the remaining training. A classifier including the outlier classes in LSC can be trained from scratch.

[0063] In a specific example, the architectural details of LSC can be as follows: Input The kernel size of the convolutional layer can be respectively... .here, , , These refer to an input with one channel, a convolutional layer with 16 feature maps, and a classifier with four classes (three object classes + one outlier class). In a specific example, the basic architecture details of CSC can be as follows: The classification and outlier headers are as follows: and .here This represents a classifier for three types of objects, and Refers to the outlier detection neuron. In a specific example, the architecture details of ODJ can be as follows: The classification loss that stabilizes the training of ODJ can be as follows: .

[0064] In a specific example, to generate outlier samples in all three outlier branches during training, a 3×3 or 5×5 patch size can be chosen to remove or blend information from another class, or to swap positions within the feature map, as discussed elsewhere in this specification. In a specific example, the patch center can be selected based on the highest activation location in the feature map, rather than randomly selecting the entire patch. The size is... H × W × C Feature maps can be aggregated along channels to obtain a size of H × W The feature map, which can then be used in The intervals are normalized to treat it as a probability graph for selecting items with probabilities. p The location of the highest activation. In CSC and ODJ, channel aggregation can be performed only to pick locations with high activation, without needing to be considered as any input to the network, even though the feature maps may have already been aggregated in LSC. During training, this scheme can enable modification of the highest activation location that will potentially lead to outliers.

[0065] Experiments demonstrate that by training the machine-learnable model only (essentially) on samples within the distribution from the training data, the metrics described in this specification can allow for the detection of outliers in content edges, location edges, and joint edge distributions with high accuracy. A framework including an outlier detector along with a classifier can provide explicit clues for interpretable decisions regarding unseen outliers. The table below illustrates several examples of explicit clues for interpretable decisions. As shown in the table, it is possible to interpret the type of outlier from the output of the outlier detector and the class decision from the classifier. When using a machine learning model on new input data, the system and / or method can provide a final inference as output or use the final inference in its final decision. Input Example ODC ODL ODJ CSC LSC Final conclusion Example 1 Not an outlier Not an outlier Not an outlier Class A Class A Within the distribution Example 2 Not an outlier Not an outlier Outliers Class B Class A Joint distribution outside Example 3 Outliers Not an outlier Outliers Class A Class A Outliers in information, inner boundaries in location .

[0066] Figure 7 A system 700 for classifying objects in spatial data is shown. System 700 may include an input interface 780 for accessing trained model data 196 representing a machine learning model, such as data that can be generated by... Figure 1 System 100 or Figure 2 Method 200 or as described elsewhere generates it. For example, also as... Figure 7 As illustrated in the diagram, the input interface can be composed of a data storage interface 780, which can access the trained model data 196 from the data storage device 790. Generally, the input interface 780 and the data storage device 790 can have the following characteristics as shown in the reference diagram. Figure 1The same type is described for input interface 180 and data storage device 190. Figure 7 Further illustrated is a data storage device 792 including input data 722, which includes at least one instance of spatial data, wherein the instance of spatial data includes objects to be classified. For example, input data 722 may be or may include sensor data obtained from one or more sensors. In a specific example, input data 722 may represent the output of a sensor-based observation, such as sensor measurements of an environment containing objects, and a machine learning model may provide classification of the objects. Examples of environments are roads in front of (semi-)autonomous vehicles, the interior of warehouses, assembly lines, etc. In some embodiments, the sensor data as input data 722 may also be received directly from sensor 20, for example, via sensor interface 720 or via another type of interface, rather than accessed from data storage device 790 via data storage interface 780. In such embodiments, sensor data may be received, for example, in real-time "field" or pseudo-real-time.

[0067] System 700 may further include a processor subsystem 760, which may be configured to, during operation of system 700, apply a convolutional portion of a machine learning model to input data to generate one or more feature maps, generate a content-specific feature map by removing location information from one of the one or more feature maps, and apply a content classification portion to the content-specific feature map to obtain a content-based object classification result. Processor subsystem 760 may be further configured to generate a location-specific feature map by removing content information from one of the one or more feature maps, and apply a location classification portion to the location-specific feature map to obtain a location-based object classification result. Processor subsystem 760 may be further configured to apply an outlier detection portion to one or more previously generated feature maps generated for instances of spatial data to obtain an outlier detection result, and classify objects in the spatial data based on the content-based object classification result, the location-based object classification result, and the outlier detection result, wherein the classification includes classifying the input data according to the object class if both the content-based object classification result and the location-based object classification result indicate an object class, and if the outlier detection result does not indicate the presence of an outlier value.

[0068] Generally, the processor subsystem 760 can be configured to perform as previously referenced. Figure 3-6 And any functionality described elsewhere. In particular, the processor subsystem 760 can be configured to apply machine learning models of the type described in the training of the reference machine learnable / learning model. This will be understood, with Figure 1The same considerations and implementation options as processor subsystem 160 apply to processor subsystem 760. Further understanding will follow. Figure 1 The same considerations and implementation options as System 100 can generally be applied to System 700, unless otherwise stated.

[0069] Figure 7 Various optional components of system 700 are further illustrated. For example, in some embodiments, system 700 may include a sensor data interface 720 for direct access to sensor data 722 acquired by sensor 20 in environment 60. Sensor 20 may, but does not need to, be part of system 700. Sensor 20 may have any suitable form, such as an image sensor, lidar sensor, radar sensor, etc., or in general, any individual sensor or set of sensors that provides spatial data containing objects to be classified. Sensor data interface 720 may have any suitable form corresponding in type to the sensor type, including but not limited to low-level communication interfaces, electronic buses, or data storage interfaces of the type described above for data storage interface 780.

[0070] In some embodiments, system 700 may include actuator interface 740 for providing control data 742 to actuator 40 in environment 60. Such control data 742 may be generated by processor subsystem 760 to control actuator 40 based on classification results; such control data 742, when applied to input data 722, may be generated by a machine learning model. For example, actuator 40 may be an electric, hydraulic, pneumatic, thermal, magnetic, and / or mechanical actuator. Specific but non-limiting examples include electric motors, electroactive aggregates, hydraulic cylinders, piezoelectric actuators, pneumatic actuators, servo mechanisms, solenoids, stepper motors, etc. This type of control reference... Figure 8 A description is given for (semi)autonomous vehicles.

[0071] In other embodiments ( Figure 7 (Not shown in the image), system 700 may include an output interface to a presentation device such as a display, light source, speaker, vibration motor, etc., which can be used to generate a sensory-perceptible output signal based on the classification result of a machine learning model. The sensory-perceptible output signal may directly indicate the classification result of the machine learning model, but may also represent a derived sensory-perceptible output signal, for example, for use in the guidance, navigation, or other types of control of a physical system.

[0072] Generally speaking, each system described in this specification—including but not limited to— Figure 1 System 100 and Figure 7System 700 may be embodied in or contained within a single device or apparatus, such as a workstation or server. The device may be an embedded device. The device or apparatus may include one or more microprocessors executing appropriate software. For example, the processor subsystem of the corresponding system may be embodied by a single central processing unit (CPU), and also by a system or combination of such CPUs and / or other types of processing units. The software may have been downloaded and / or stored in a corresponding memory, such as volatile memory like RAM, or non-volatile memory like flash memory. Alternatively, the processor subsystem of the corresponding system may be implemented in the device or apparatus in the form of programmable logic, for example as a field-programmable gate array (FPGA). Generally, each functional unit of the corresponding system may be implemented in the form of circuits. The corresponding system may also be implemented in a distributed manner, for example involving different devices or apparatuses, such as distributed local or cloud-based servers. In some embodiments, system 700 may be part of a vehicle, robot, or similar physical entity, and / or may represent a control system configured to control the physical entity.

[0073] Figure 8 The above example illustrates a system 700 as a control system for a (semi-)autonomous vehicle 80 operating in environment 60. The autonomous vehicle 80 can be autonomous, as it may include an autonomous driving system or a driver assistance system, the latter also referred to as a semi-autonomous system. The autonomous vehicle 80 can, for example, incorporate system 700 to control the steering and braking of the autonomous vehicle based on sensor data obtained from a video camera 22 integrated into the vehicle 80. For example, system 700 can control electric motor 42 to perform (regenerative) braking in the event that the autonomous vehicle 80 anticipates a collision with a pedestrian. System 700 can control steering and / or braking to avoid a collision with a pedestrian. For this purpose, system 700 can classify objects such as pedestrians from the sensor data obtained from the video camera. If the vehicle's state—e.g., its position relative to a pedestrian—is expected to result in a collision, system 700 can take appropriate action.

[0074] Figure 9 A computer-implemented method 800 for classifying objects in spatial data is shown. Method 800 can correspond to... Figure 7 The system 700 can be used for operation, but it can also be used or performed by any other system, device or equipment.

[0075] Method 800 is shown to include, in a step entitled "Accessing the Machine Learning Model," accessing 810 a machine learning model as described elsewhere in this specification, and in a step entitled "Accessing Input Data," accessing 820 input data, which includes instances of spatial data, the instances of spatial data including objects to be classified. Method 800 is further shown to include, in a step entitled "Generating (One or More) Feature Maps," applying a convolutional portion of the machine learning model to the input data to generate 830 one or more feature maps, and in a step entitled "Generating Content-Specific Features," generating 840 content-specific feature maps by removing location information from one of the one or more feature maps. Method 800 is further shown to include, in a step entitled "Generating Location-Specific Features," generating 850 location-specific feature maps by removing content information from one of the one or more feature maps, and in a step entitled "Generating Content-Based Object Classification Results," applying a content classification portion 860 to the content-specific feature maps to obtain content-based object classification results. Method 800 is further shown to include, in the step entitled "Generating Location-Based Object Classification Results," applying 870 of the location classification portion to a location-specific feature map to obtain a location-based object classification result, and in the step entitled "Generating Outlier Detection Results," applying 880 of the outlier detection portion to one or more previously generated feature maps generated for instances of spatial data to obtain an outlier detection result. Method 800 is further shown to include, in the step entitled "Classifying Objects in Spatial Data," classifying objects in spatial data 890 according to a content-based object classification result, a location-based object classification result, and an outlier detection result, wherein the classification includes classifying the input data according to the object class if both the content-based object classification result and the location-based object classification result indicate an object class, and if the outlier detection result does not indicate the presence of an outlier value.

[0076] To understand, generally speaking Figure 2 and Figure 9 The operations or steps of the computer-implemented methods 200 and 800 may be performed in any suitable order, such as sequentially, simultaneously, or a combination thereof, which, where applicable, conform to a particular order necessary, for example, by input / output relationships.

[0077] Each method, algorithm, or pseudocode described in this specification can be implemented on a computer as a computer-implemented method, dedicated hardware, or a combination of both. For example... Figure 10As illustrated, instructions for a computer (e.g., executable code) may be stored on a computer-readable medium 900, for example, in the form of a series of machine-readable physical symbols 910 and / or as a series of elements having different electrical (e.g., magnetic) or optical properties or values. The executable code may be stored in a transient or non-transient manner. Examples of computer-readable media include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Figure 10 An optical disc 910 is shown. In an alternative embodiment of the computer-readable medium 900, the computer-readable medium may include model data 910 defining a machine learning model, as described elsewhere in this specification.

[0078] Examples, embodiments, or optional features—whether or not indicated as non-limiting—should not be construed as limiting the invention as claimed.

[0079] According to the abstract of the specification, systems and methods for classifying objects in spatial data using machine learning models, as well as systems and methods for training machine learning models, can be provided. The machine learning model may include a content-sensitive classifier, a location-sensitive classifier, and at least one outlier detector. These two classifiers can be jointly used to distinguish between objects in the spatial data that are within a distribution or outside a marginal distribution. The outlier detection component can be trained on in-circle examples from the training data, while the presence of actual outliers in the input data of the machine-learnable model can be modeled in the feature space of the machine-learnable model during training. The combination of these components can provide a more robust classification of objects in spatial data relative to outliers without increasing the size of the training data.

[0080] It should be noted that the embodiments mentioned above are illustrative rather than limiting of the invention, and those skilled in the art will be able to devise many alternative embodiments without departing from the scope of the appended claims. Any reference marks placed between parentheses in the claims should not be construed as limiting the claims. The use of the verb “comprising” and its inflections does not exclude the presence of elements or stages other than those set forth in the claims. The article “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. Expressions such as “at least one of…” when preceding a list or group of elements indicate the selection of all elements or any subset of elements from that list or group. For example, the expression “at least one of A, B, and C” should be understood to include only A, only B, only C, both A and B, both A and C, both B and C, or all A, B, and C. The invention can be implemented by means of hardware comprising several different elements, and by means of a suitably programmed computer. In device claims enumerating several components, several of these components can be embodied by the same item of hardware. The mere fact that certain measures are described in mutually different dependent claims does not indicate that a combination of these measures cannot be used advantageously.

Claims

1. A computer-implemented method (200) for training a machine-learnable model for classifying objects in space sensor data, wherein objects can be classified into different object classes by combining content information and location information contained in the space sensor data, the method comprising: - Access (210) training data, which includes instances of space sensor data, which include objects belonging to different object classes; - Provide (220) a machine-learnable model, wherein the machine-learnable model includes a convolutional portion for generating one or more feature maps from instances of spatial sensor data, the convolutional portion including one or more convolutional layers, wherein the one or more feature maps have a spatial dimension representing the spatial sensor data, wherein activation of the one or more feature maps at a particular location represents the presence of a feature of the representation content at the particular location; and provide a first classification portion and a second classification portion. - As part of training (230) a machine-learnable model, (240) A content-specific feature map is generated by removing location information from the one or more feature maps, and (245) a first classification part is trained on the content-specific feature map to obtain a content classification part; (250) A location-specific feature map is generated by removing content information from the one or more feature maps, and (255) a second classification part is trained on the location-specific feature map to obtain a location classification part; The method further includes: - Provided as part of a machine-learnable model (225) at least one outlier detection component, said at least one outlier detection component being trained to detect outliers in the input data of the machine-learnable model that do not conform to the distribution of the training data; and - As part of training (230) a machine-learnable model, (260) Pseudo-outlier feature maps are generated by modifying one or more previously generated feature maps generated as instances of space sensor data to simulate the presence of actual outliers in the input data of a machine-learnable model, wherein modifying the one or more previously generated feature maps includes at least one of the following: - Remove feature information from the feature map; - The positions of feature information in the feature map are pseudo-randomly shuffled; - Mix feature information between feature maps of different object classes; - Exchange feature information at different positions in the feature map, and The (265) outlier detection part is trained on the pseudo outlier feature map.

2. The method (200) of claim 1, wherein the machine-learnable model includes a location and content outlier detection portion, wherein the method includes generating (260) pseudo-outlier feature maps for the location and content outlier detection portion by modifying feature information contained in the one or more previously generated feature maps and associated with both location information and content information.

3. The method (200) of claim 1 or 2, wherein the machine-learnable model includes a location outlier detection portion, wherein the method includes generating a pseudo outlier feature map (260) for the location outlier detection portion by modifying feature information contained in the one or more previously generated feature maps and associated with location information.

4. The method (200) according to claim 3, wherein the location outlier detection part is implemented by the location classification part by providing pseudo-outlier feature maps to the location classification part as part of a separate outlier object class to be learned.

5. The method (200) of any one of claims 1 to 4, wherein the machine-learnable model includes a content outlier detection portion, wherein the method includes generating (260) pseudo outlier feature maps for the content outlier detection portion by modifying feature information contained in the one or more previously generated feature maps and associated with content information.

6. The method (200) according to any one of claims 1 to 5, wherein each of the one or more feature maps generated by the convolutional portion has at least two spatial dimensions associated with location information, and wherein the feature values ​​of the one or more feature maps at each spatial coordinate together form a feature vector representing content information at the corresponding spatial coordinate, wherein: - Removing location information from one of the one or more feature maps includes aggregating the one or more feature maps above the spatial dimension to form a content-specific feature map that includes a feature vector; Removing content information from one of the one or more feature maps involves aggregating feature values ​​for each spatial coordinate on the one or more feature maps to form a location-specific feature map with at least two spatial dimensions and one feature value channel.

7. The method (200) according to any one of claims 1 to 6, wherein the machine-learnable model is a deep neural network, wherein the convolutional part is the convolutional part of the deep neural network, and wherein the content classification part and the location classification part are the corresponding classification heads of the deep neural network.

8. A computer-implemented method (800) for classifying objects in space sensor data, wherein objects can be classified into different object classes by combining content information and location information contained in the space sensor data, the method comprising: - Access (810) a machine learning model, wherein the machine learning model is a machine learnable model trained by the method of any one of claims 1 to 7; - Access (820) input data, the input data including instances of space sensor data, the instances of space sensor data including objects to be classified; - Apply the convolutional part of the machine learning model to the input data to generate (830) one or more feature maps, wherein the one or more feature maps have a spatial dimension representing spatial sensor data, and wherein the activation of the one or more feature maps at a particular location represents the presence of a feature of the representation content at that particular location; - A content-specific feature map is generated (840) by removing location information from one of the one or more feature maps, and a content classification portion is applied (860) to the content-specific feature map to obtain a content-based object classification result; - A location-specific feature map is generated by removing content information from one of the one or more feature maps, and a location classification portion is applied (870) to the location-specific feature map to obtain a location-based object classification result; - Apply the outlier detection part (880) to one or more previously generated feature maps generated for instances of spatial sensor data to obtain outlier detection results; - Classify objects in spatial sensor data based on content-based object classification results, location-based object classification results, and outlier detection results (890), wherein the classification includes classifying the input data according to the object class if both the content-based object classification results and the location-based object classification results indicate an object class, and if the outlier detection results do not indicate the presence of outliers.

9. The method (800) of claim 8, further comprising performing the method (200) of any one of claims 1 to 7 before classifying objects in space sensor data using a machine learning model.

10. A computer-readable medium (900) comprising temporary or non-temporary data (910) representing a computer program, said computer program comprising instructions for causing a processor system to perform the method according to any one of claims 1 to 9.

11. A system (100) for training a machine-learnable model for classifying objects in space sensor data, wherein objects can be classified into different object classes by combining content information and location information contained in the space sensor data, said system comprising: - Input interface (180) for accessing training data (192), the training data including instances of space sensor data, the instances of space sensor data including objects belonging to different object classes; - The processor subsystem (160) is configured as follows: - Provide a machine-learnable model (194), wherein the machine-learnable model includes a convolutional portion for generating one or more feature maps from instances of spatial sensor data, the convolutional portion including one or more convolutional layers, wherein the one or more feature maps have a spatial dimension representing the spatial sensor data, wherein activation of the one or more feature maps at a particular location represents the presence of a feature of the representation content at the particular location; and provide a first classification portion and a second classification portion. - As part of training a machine-learnable model Content-specific feature maps are generated by removing location information from the one or more feature maps, and a first classification part is trained on the content-specific feature maps to obtain a content classification part. A location-specific feature map is generated by removing content information from the one or more feature maps, and a second classification part is trained on the location-specific feature map to obtain a location classification part. - As part of a machine-learnable model, at least one outlier detection component is provided, said at least one outlier detection component being trained to detect outliers in the input data of the machine-learnable model that do not conform to the distribution of the training data; and - As part of training a machine-learnable model Pseudo-outlier feature maps are generated by modifying one or more previously generated feature maps generated as instances of space sensor data to simulate the presence of actual outliers in the input data of a machine-learnable model, wherein modifying the one or more previously generated feature maps includes at least one of the following: - Remove feature information from the feature map; - The positions of feature information in the feature map are pseudo-randomly shuffled; - Mix feature information between feature maps of different object classes; - Exchange feature information at different positions in the feature map, and Training the outlier detection part on the pseudo-outlier feature map, and - Output interface (180) for outputting machine learning model data (196) representing the trained machine learnable model.

12. A system (700) for classifying objects in space sensor data, wherein objects can be classified into different object classes by combining content information and location information contained in the space sensor data, said system comprising: - Input interface (720, 780) for accessing input data (722), the input data including instances of space sensor data, the instances of space sensor data including objects to be classified; - The processor subsystem (760) is configured as follows: - Access to a machine learning model (196), wherein the machine learning model is a machine-learnable model trained by the method of any one of claims 1 to 7; - Apply the convolutional part of a machine learning model to the input data to generate one or more feature maps; - A content-specific feature map is generated by removing location information from one of the one or more feature maps, and a content classification component is applied to the content-specific feature map to obtain a content-based object classification result; - A location-specific feature map is generated by removing content information from one of the one or more feature maps, and a location classification component is applied to the location-specific feature map to obtain a location-based object classification result; - Apply the outlier detection component to one or more previously generated feature maps generated for instances of spatial sensor data to obtain outlier detection results; - Classify objects in spatial sensor data based on content-based object classification results, location-based object classification results, and outlier detection results, wherein the classification includes classifying the input data according to the object class if both the content-based object classification results and the location-based object classification results indicate an object class, and if the outlier detection results do not indicate the presence of outliers.

13. The system (700) of claim 12, wherein the input interface is a sensor interface (720) to a sensor (20), wherein the sensor is configured to acquire space sensor data.

14. The system (700) according to claim 12 or 13, wherein, The system is a control system configured to adjust control parameters based on the object's classification.

Images