A computer-based method, system, and computer-readable medium for training machine-learnable functions.

Abstract

A function is provided that is trained to predictably respond to differences. The invention relates to a computer-implemented method (600) of training a machine-learnable function, such as an image classifier or an image feature extractor. When applying such machine-learnable functions in the field of autonomous driving and similar applications, generalization capability is important. To improve the generalization capability, the machine-learnable function is rewarded for predictably responding to a set of differences between input observations at a layer of the machine-learnable function. This is done by means of a regularization objective included in an objective function used to train the machine-learnable function. The regularization objective rewards mutual statistical correlation between representations of input observations at a given layer, given difference labels indicative of differences between the input observations.

Description

Technical Field

[0001] This invention relates to a computer-based method for training machine-learnable functions, and a corresponding system. The invention also relates to a computer-readable medium comprising instructions and / or function parameters. Background Technology

[0002] Machine-learnable functions have the potential to perform tasks much more accurately than manually designed algorithms. For example, machine-learnable functions (e.g., based on convolutional neural networks or similar techniques) have shown the ability to achieve good accuracy on a variety of image classification tasks, such as recognizing digits or distinguishing between cats and dogs. Typically, such machine-learnable functions are trained on training datasets (e.g., images containing their expected classification labels). Generally, the more training examples, the better the accuracy of the trained function.

[0003] In many practical situations, the feasibility of obtaining training data that covers a sufficiently broad range of inputs is a major obstacle to the practical adoption of machine learning solutions. For example, in the field of (semi-)autonomous vehicles, machine-learnable functions have the potential to perform a variety of tasks required for controlling or monitoring vehicles, such as image analysis of camera images of the vehicle's environment, thereby, for example, warning of dangerous traffic situations. However, obtaining training data for such tasks requires driving around the vehicle; this is expensive and can even be dangerous. Furthermore, images of the vehicle's environment can vary in many different ways, such as in the overall configuration of the scene (whether there is another car ahead, whether there are pedestrians nearby, etc.) and also in many other aspects (e.g., the type of road on which the car is traveling; how traffic signs look, which varies by country; weather conditions; etc.).

[0004] This makes it nearly impossible to obtain a training dataset that covers all possible combinations of scenarios a vehicle might encounter, especially in extreme cases. On the other hand, it is crucial that machine-learnable functions reliably handle such extreme cases, especially when used for making decisions in automated driving. Therefore, it is desirable to use functions with high generalization capabilities, that provide accurate results even for inputs not directly covered in the training dataset.

[0005] One known machine learning technique that aids in generalization is the so-called group equivariant convolutional network, described, for example, in "Group Equivariant Convolutional Networks" by T. Cohen et al. (incorporated by reference and available at https: / / arxiv.org / abs / 1602.07576). Group equivariant convolutional networks are deep neural networks, i.e., functions that process an input (in this case, an image) by passing the input through multiple corresponding parameterized layers. In group equivariant convolutional networks, the representations at these corresponding layers are modeled as a linear space G, where G is a predefined mathematical group (i.e., a set of binary operations satisfying closure, correlation, identity, and invertibility) representing the set of transformations that can be applied. Equivariance means that transformations on a given layer input are equivariant. T g and layer output T g ' ,in g ∈ G identity This holds true. For example, rotating or mirroring the input should result in a corresponding rotation of the output. This additional structure, given to neural network layers, allows for more efficient modeling of data and improved generalization, essentially because knowledge about the desired function output can be shared between the inputs and their transformation variables.

[0006] Unfortunately, existing techniques for making functions equivariant have several drawbacks. They require complex analytical formulas for the transformation group G, and additionally, these transformations must be known in advance and hard-coded into the model architecture. Furthermore, using the transformation group when employing the function makes the model architecture dependent on the type of transformations used during training, thus making it incompatible with standard tools for applying machine learning techniques, such as standard implementations of convolutional neural networks. The transformations also need to conform to the group axioms of the transformation group G. Therefore, only a limited set of (geometric) transformations can be specified. Summary of the Invention

[0007] It would be desirable to train machine-learnable functions, such as image classifiers or image feature extractors, to improve generalization ability without integrating the transformations into the model architecture. Specifically, it would be desirable to reward the predictability of changes in input observations that do not depend on the mathematical group structure. It would also be desirable to improve the generalization ability of machine-learnable functions without affecting their model architecture, for example, by making the trained functions standard convolutional neural networks.

[0008] According to one aspect of the present invention, a computer-implemented method and corresponding system for training machine-learnable functions are provided. According to another aspect of the present invention, a computer-readable medium is described.

[0009] Each aspect involves training a machine-learnable function by optimizing an objective function (in other words, learning the set of parameters of a parameterized machine-learnable function). Many types of machine-learnable functions and the objective functions suitable for training them are known. For example, the learning task could be classification or regression. Training can be supervised, for example, based on a labeled dataset of training input observations and corresponding training outputs, with the objective function configured to minimize the difference between the machine-learnable function's output and its corresponding training output. Training can also be unsupervised, such as learning a feature extractor (feature extraction is also called representation learning). In either case, a conventional objective function can be used.

[0010] Interestingly, to improve generalization, the machine-learnable function can be rewarded for predictably responding to a set of differences between input observations at a layer of the machine-learnable function. This is similar to a group-equal-variable convolutional neural network, which enforces that applying a given G-action to an input observation results in applying the corresponding G-action to the layer's output. However, interestingly, the inventors envision rewarding predictability not by hardcoding a specific set of transformations into the model architecture, but by including a regularization objective in the objective function used to train the machine-learnable function. For example, a regularization objective that reduces predictability could be included in the loss function to be minimized, or a regularization objective that increases predictability could be included in the objective function to be maximized.

[0011] Specifically, the objective function can be based on first and second input observation pairs, and difference markers indicating the differences between these first and second observations. For example, the first and second observations can be images of broadly corresponding scenes; for instance, both could be vehicle environments where vehicles are stopping due to traffic lights. Difference markers can highlight specific differences between the first and second observations, such as differences in weather conditions; the type of vehicle in front of the vehicle; the country in which the images were obtained (and correspondingly, the type of current traffic lights and / or traffic signs); and so on. Preferably, the observation pairs are selected such that they correspond to each other as closely as possible, except for the differences indicated by the difference markers.

[0012] In particular, the difference label can represent a transformation from which a second observation can be obtained (or at least approximately obtained) from a first observation. Thus, swapping the first and second observations typically also results in another difference label. However, it is of interest that the set of differences from which the difference label is chosen does not need to have any particular mathematical structure; in particular, it does not need to be a group in the mathematical sense.

[0013] To evaluate a regularization objective used to encourage predictability at a particular layer, a machine-learnable function can be applied to the first and second observations to obtain the corresponding representations at that layer. The regularization objective can then reward predictability by encouraging cross-statistical correlations between these representations given difference labels. In other words, the amount of information one representation provides about another representation, given difference labels, can be maximized by the objective function. Such cross-statistical correlations can be computed, for example, based on conditional mutual information or interaction information, as discussed in more detail below.

[0014] A regularizer is obtained by encouraging the cross-statistical correlation of representations given a difference label. This regularizer improves the generalization ability and robustness of the trained function. That is, the function is effectively rewarded for learning how the difference between a pair of input observations affects the function's output. Therefore, the function is better able to handle differences if the same difference also applies to unseen observations.

[0015] In terms of the achieved results, this is similar to the improved generalization capability provided by group-equal-variable convolutional networks, and in fact, the inventors have been able to demonstrate that group-equal-variable convolutional networks provide optimal performance for the regularization objective when expressed based on conditional mutual information. However, it is of interest that the provided technique offers significantly greater flexibility than group-equal-variable convolutional networks, resulting in several practical advantages.

[0016] By using cross-statistical correlation-based discriminants, a metric applicable to a wide range of discriminant sets is obtained, without requiring assumptions about a specific mathematical group structure. This allows for capturing various types of discriminants that occur in real-world situations, such as differences in weather conditions, differences in the countries where observations take place, etc. It also does not require explicitly defining how the discriminants transform one input into another: having the input observation pair and the accompanying discriminant labels is sufficient, and the discriminant labels can even be learned from the data. The pair does not need to be entirely different, only different according to the type of discriminant; for example, there may be other discriminants between observations (although training is more efficient if there is a closer correspondence). For the representation to which the regularization objective is applied, no transformation needs to be given as input, and therefore, the function itself can learn the optimal way to represent the input.

[0017] Furthermore, inclusion in the objective function allows for a balance of predictability with respect to other training objectives. In fact, cross-statistical correlation can be viewed as a continuum between unpredictability and predictability. Because the regularization term typically weights other terms in the objective function, the function itself can automatically achieve an appropriate amount of statistical correlation for the current learning task, rather than being forced to strictly enforce equivariance regardless. This also improves the function's applicability to situations where overly strict equivariance enforcement would degrade performance.

[0018] Another advantage of encouraging predictability by adapting the objective function (e.g., instead of the model architecture) is that a trained function can be obtained whose architecture does not depend on the type of variance considered during training. For example, the hardware / software used to apply the trained function does not need to depend on or support various transformations that provide such variances. Therefore, compatibility with conventional software / hardware used to apply machine learning functions, such as convolutional neural network evaluation software / hardware, is improved.

[0019] Several options are possible regarding the layer to which the regularization objective is applied. The objective can be applied at the output layer regardless of the type of machine-learnable function. This also applies if the machine-learnable function does not otherwise follow a hierarchical structure, for example, it also applies to non-neural networks. The output layer is a particularly good choice if the machine-learnable function is a feature extractor, such as an image feature extractor, because in this case, the predictability provided by the regularization objective can then be fully utilized by other components that use it. For machine-learnable functions with inner layers, such as neural networks and deep Gaussian processes, it is possible to apply the regularization objective alternatively or additionally to the inner layers of the function, or to apply the corresponding regularization objective to multiple corresponding inner layers. Beneficial choices for the machine-learnable function and / or the layer to be regularized are always discussed.

[0020] Alternatively, the machine-learnable function can be an image classifier or an image feature extractor. Such image processing models particularly have to handle input variations and therefore benefit especially from improved generalization capabilities. Generally, an image classifier maps an input image to one or more classification outputs. For example, an image classifier can be a detection model, such as a function that provides a classification output indicating the presence of an object in the entire image or in an image segment. An image classifier can also be a semantic segmentation model, such as a function that provides multiple classification outputs indicating the presence of an object in individual pixels or segments of an image. An image feature extractor maps an input image to an image feature vector, for example, for use in classification or other downstream applications. By using the described training techniques, more efficient and robust image processing models can be obtained.

[0021] Alternatively, the machine-learnable function can be an image feature extractor used in the overall image classifier. In this case, it is possible to train the image features on an unlabeled training dataset (i.e., training without using any available classification labels), and then train the overall image classifier on a labeled training dataset. The labeled dataset can be, for example, a subset of the unlabeled training dataset for which labels are available, or it can be a separate dataset. Typically, unlabeled data is more readily available than labeled data. By separating the training of the feature extractor from the training of the classifier, unlabeled data can be used to train the feature extractor. When training the feature extractor alone, learning generalizable features is particularly important, and therefore, the provided regularization objective is beneficial. In this example, the regularization objective is typically applied to the output layer of the feature extractor. When training the classifier, the feature extractor can also be fine-tuned (which may or may not include the use of the regularization objective), but this is not required.

[0022] Optionally, the objective function may include multiple corresponding regularization objectives for multiple layers of a machine-learnable function. Different layers of a function (such as a neural network) tend to represent input observations at different levels of abstraction. By applying regularization objectives at several layers, generalization ability can therefore be rewarded at several levels of abstraction. For example, during training, at each layer, the regularization objective can be included and removed from the objective function to see if it has a beneficial effect at that layer.

[0023] Optionally, the first and second input observations are images representing the same type of scene, such as a scene including similar objects or similar configurations of multiple objects. For example, the two images could represent the environment of a vehicle, for use in a visual model of a (semi-)autonomous vehicle. For example, the two images in a pair could represent scenes such as a vehicle cruising on a highway, a vehicle stopping at a traffic light, a pedestrian accidentally crossing the road, and so on. However, other application areas where robustness to transformed regions is required are also possible.

[0024] In such cases, difference tags can indicate differences in attributes of objects or the scene itself (e.g., car brand; car type; whether the car should be on the left or right; or the country where the image was captured, which can be seen as proxies for several scene attributes). Difference tags can also indicate differences in the observation conditions of the scene, such as weather conditions; the level of occlusion of objects captured in the image; the position or orientation of objects in the image; and so on. By including a regularization objective that includes such semantically meaningful differences, the reward function can infer these differences and their impact on the function's output, resulting in a function that generalizes better to these differences.

[0025] Alternatively, the machine-learnable function can be a convolutional network. Convolutional networks—also known as convolutional neural networks (CNNs)—have proven effective for a variety of learning tasks, particularly image analysis. Their corresponding layers typically represent input observations at corresponding levels of abstraction. By selecting the inner layers of a CNN, various differences between input observations can be appropriately accounted for in the regularization objective.

[0026] Optionally, evaluating the regularization objective may include estimating the mutual information between the first and second representations given a difference label. This so-called conditional mutual information provides a suitable, information theory-driven way to determine the cross-statistical correlation between input observation representations. However, other mutual information about the three variables may also be estimated, such as the mutual information between the first representation and a difference label given a second representation, or the mutual information between the second representation and a difference label given a first representation. For example, the regularization objective may include interaction information, also as described below, which may be based on these mutual information. Instead of using a mutual information-based measure, other measures of cross-statistical correlation, such as the conditional entropy or information change of the second observation given a first representation and a difference label, are also possible.

[0027] One reason why using (conditional) mutual information is particularly beneficial is that it can be estimated efficiently from a trainable mutual information estimation model. While it is possible to directly compute or estimate (conditional) mutual information or related quantities such as conditional entropy, using a trainable estimation model significantly improves performance. Interestingly, for example, a known mutual information estimation model can be trained simultaneously with a machine-learnable function by including the learning objective of the mutual information estimation model in the objective function. In this way, the function used to estimate mutual information can be progressively updated as the machine-learnable function is trained, rather than having to be learned from scratch every time the machine-learnable function changes.

[0028] Optionally, the regularization objective can also be configured to suppress cross-correlation between the first and second representations that is not conditioned on the difference label, which is measured, for example, by the mutual information between the first and second representations (not conditioned on the difference label). In this way, the function can be suppressed from providing the same representation for two input observations; in other words, the function can be suppressed from determining representations that are not only equivariant but even invariant. Such invariant representations are likely to be local optima for the regularization objective. However, invariance can be detrimental to the function's generalization ability because it effectively rewards the function for discarding information about label differences rather than learning how to respond to it in a non-trivial way. In network structure functions, this could be due to the transformation information being propagated deeper and more orderly structured into the network, which invariant networks might discard. Therefore, better generalization can be achieved, especially when there is little training data for extreme cases.

[0029] Specifically, the regularization objective can include the difference between mutual information conditioned on the difference label and mutual information not conditioned on the difference label. This difference is also referred to as the interaction information between the first representation, the second representation, and the difference label. Note that the interaction information can also be calculated without explicitly calculating the mutual information mentioned above; however, it will be understood that, regardless of how the interaction information is calculated, it generally rewards both the statistical correlation between the first and second representations with a given difference label and suppresses the correlation between representations not conditioned on the difference label.

[0030] Several options are envisioned for obtaining input observation pairs and their corresponding difference labels. The first input observation typically comes from the training dataset. Optionally, given the difference labels, a second input observation can be generated from the first input observation and the difference labels. For example, transformations based on the difference labels can be applied, such as when objects in an image may be partially occluded; differences in weather conditions can be simulated, such as by adding snow or rain; and so on. Given observation pairs, it is also possible to determine the difference labels using machine-learnable labeling functions. For example, the labels of some observation pairs can be used to train a labeling function to label other pairs, thereby reducing the required labeling effort; or the labeling function can even be trained in an unsupervised manner, so that manual labeling is not required at all.

[0031] The techniques described in this paper can be applied to various types of input observation. This includes images, but also a variety of other types of sensor data, including radar data, ultrasound data, lidar data, motion data, etc. For example, in the context of a computer control system, sensor data can provide observations of the computer control system and / or its environment. In such cases, the computer control system can be monitored and / or controlled based on the output of a machine-learnable function. Examples of computer control systems include robots, (semi- or fully) autonomous vehicles, home appliances, power tools, manufacturing machines, personal assistants, and access control systems. Systems that can be monitored, for example, include surveillance systems and medical (imaging) systems.

[0032] Those skilled in the art will appreciate that two or more of the embodiments, implementations and / or optional aspects of the present invention mentioned above can be combined in any way that is considered useful.

[0033] Based on this description, those skilled in the art can make modifications and variations to any system and / or any computer-readable medium, which correspond to the modifications and variations described in the corresponding computer-implemented method. Attached Figure Description

[0034] These and other aspects of the invention will become clear from the embodiments described by way of example in the following description and from the accompanying drawings, which are further illustrated with reference to the embodiments and the accompanying drawings, in which: Figure 1 A system for training machine-learnable functions is shown; Figure 2 A system for applying machine-learnable functions is shown; Figure 3 A system for applying machine-learnable functions as part of a (semi)autonomous vehicle is shown; Figure 4 A detailed example of a machine-learnable convolutional network is shown; Figure 5a A detailed example of training a machine-learnable function using a regularization objective is shown; Figure 5b A detailed example of generating a second input observation for use in training a machine-learnable function is shown; Figure 5c A detailed example of training a machine-learnable function is shown, which is an image feature extractor used in an overall image classifier; Figure 6 A method for training machine-learnable functions is shown in a computer implementation. Figure 7 A computer-readable medium containing data is shown.

[0035] It should be noted that the figures are purely illustrative and not drawn to scale. In each figure, elements corresponding to those already described may have the same reference numerals. Detailed Implementation

[0036] Figure 1 A system 100 for training a machine-learnable function is illustrated. The machine-learnable function maps input observations to function outputs. The machine-learnable function may include one or more inner layers and / or an output layer. The function can be trained by a system applying the machine-learnable function (e.g., [system name]). Figure 2 The system 200) is used, in particular, to enable the monitoring and / or control of computer control systems based on the output of machine-learnable functions.

[0037] System 100 may include a data interface 120. Data interface 120 can be used to access a training dataset 030 that includes multiple training input observations. Data interface 120 can also be used to access a set of parameters 040 of a machine-learnable function being trained. Once learned, parameters 040 can be used to apply the machine-learnable function to the observations, as well as regarding... Figure 2 The subject of discussion. For example, as well as... Figure 1As illustrated, the data interface may consist of a data storage interface 120, which can access data 030 and 040 from data storage device 021 via data communication 122. For example, the data storage interface 120 may be a memory interface or a persistent storage interface (e.g., a hard disk or SSD interface), but it may also be a personal, local area, or wide area network interface (such as a Bluetooth, Zigbee, or Wi-Fi interface, or an Ethernet or fiber optic interface). The data storage device 021 may be an internal data storage device of system 100, such as a hard disk drive or SSD, but it may also be an external data storage device, such as a network-accessible data storage device. In some embodiments, data 030 and 040 may be accessed from different data storage devices, for example, via different subsystems of the data storage interface 120. Each subsystem may have the type described above for the data storage interface 120.

[0038] System 100 may further include a processor subsystem 140, which may be configured to learn a set of parameters of a machine-learnable function by optimizing an objective function during operation of system 100. Optimization may include rewarding the machine-learnable function for a predictable response to a set of differences between input observations at a layer (e.g., an inner layer or an output layer) of the machine-learnable function. Optimization may reward said predictability by means of a regularization objective included in the objective function.

[0039] To perform optimization, processor subsystem 140 may obtain: a first input observation from the training dataset; a corresponding second input observation; and a difference label from a difference set indicating the difference between the first and second input observations. Processor subsystem 140 may also apply a machine-learnable function to determine first and second representations of the first and second input observations, respectively, at that layer of the machine-learnable function. Processor subsystem 140 may also evaluate a regularization objective. The regularization objective may be configured to reward the cross-statistical correlation between the first and second representations given the difference label between the first and second input observations. System 100 may output the learned parameter set via an output interface, for example, by updating the parameter set in storage device 021 via data interface 120, by providing the parameter set to a third party via a communication interface, and so on.

[0040] Figure 2 The following are examples of applications, such as those described herein, via... Figure 1 System 100 or according to Figure 6The computer-implemented method 600 trains a machine-learnable function in system 200. For example, system 200 may be a vehicle control system, a robot control system, a manufacturing control system, or a building control system. System 200 may also optionally perform training; for example, system 200 may be combined with system 100.

[0041] System 200 may include a data interface 220 for accessing a set of parameters 040 of a trained machine-learnable function. The same considerations and implementation options apply. Figure 1 Data interface 120. The figure shows data interface 220, which accesses parameter 040 from storage device 022 via data communication 222, similar to... Figure 1 Storage device 021 and data communication 122.

[0042] System 200 may further include a processor subsystem 240, which can be configured to apply a machine-learnable function, parameterized by parameter 040, to observations of the computer control system and / or its environment during operation of system 200. Based on the output of the machine-learnable function, the computer control system can be monitored and / or controlled in a conventional manner. The same considerations and implementation options apply. Figure 1 The processor subsystem 140.

[0043] Figure 2 Various optional components of system 200 are also shown. For example, in some embodiments, system 200 may include sensor interface 260 for direct access to sensor data 224, which provides observation of the computer control system and / or its environment. For example, sensor data 224 may be acquired by sensor 072 in environment 082. The sensor may be positioned in environment 082, but may also be positioned away from environment 082, for example, if one or more quantities can be measured remotely. Sensor 072 may, but does not need to, be part of system 200. Sensor 072 may have any suitable form, such as an image sensor, lidar sensor, radar sensor, pressure sensor, container temperature sensor, etc. Sensor interface 260 may have any suitable form corresponding in type to the sensor type, including but not limited to low-level communication interfaces (e.g., I2C or SPI-based data communication), or data storage interfaces of the type described for data interface 220.

[0044] In some embodiments, system 200 may include actuator interface 280 for providing control data 226 to actuator 092 in environment 082. Such control data 226 may be generated by processor subsystem 240 to control the actuator based on the output of a machine-learnable function. The actuator may, but does not necessarily, be part of system 200. For example, the actuator may be an electrical, hydraulic, pneumatic, thermal, magnetic, and / or mechanical actuator. Specific but non-limiting examples include electric motors, electroactive polymers, hydraulic cylinders, piezoelectric actuators, pneumatic actuators, servo mechanisms, solenoids, stepper motors, etc. This type of control reference... Figure 3 This describes (semi)autonomous vehicles.

[0045] In other embodiments ( Figure 2 (Not shown in the diagram), system 200 may include an output interface to a presentation device (such as a display, light source, speaker, vibration motor, etc.), which can be used to generate a sensory-perceptible output signal, which may be generated based on a function output. The sensory-perceptible output signal may directly indicate the function output, but may also represent a derived sensory-perceptible output signal, for example, for use in guidance, navigation, or other types of control in a computer control system.

[0046] Generally speaking, each system described in this specification—including but not limited to— Figure 1 System 100 and Figure 2 System 200 may be embodied as, or embodied in, a single device or apparatus such as a workstation or server. The device may be an embedded device. The device or apparatus may include one or more microprocessors executing appropriate software. For example, the processor subsystem of the corresponding system may be embodied by a single central processing unit (CPU), but may also be embodied by a combination or system of such a CPU and / or other types of processing units. The software may have been downloaded and / or stored in corresponding memory, such as volatile memory (such as RAM) or non-volatile memory (such as flash memory). Alternatively, the processor subsystem of the corresponding system may be implemented in the device or apparatus as programmable logic, such as as a field-programmable gate array (FPGA). Generally, each functional unit of the corresponding system may be implemented as a circuit. The corresponding system may also be implemented in a distributed manner, for example, involving different devices or apparatuses, such as distributed local or cloud-based servers. In some embodiments, system 200 may be part of a vehicle, robot, or similar computer control system, and / or may represent a control system configured to control such a system.

[0047] Figure 3The example described above illustrates a control system 200 for a (semi-)autonomous vehicle 62 operating in environment 50. The autonomous vehicle 62 can be autonomous, as it may include an autonomous driving system or a driver assistance system, the latter also referred to as a semi-autonomous system. In this example, the input observation used by system 200 may be an image of the vehicle's environment. Difference markers used during training may indicate, for example, differences in weather conditions, and / or differences in the occlusion level and / or position and / or orientation of objects in the environment.

[0048] Autonomous vehicle 62 can, for example, be incorporated into system 200 to control its steering and braking based on sensor data obtained from video camera 22 integrated into vehicle 62. For instance, system 200 can control electric motor 42 to perform (regenerative) braking should a collision with a road user be anticipated. System 200 can control steering and / or braking to avoid a collision. To this end, system 200 can obtain images of the traffic conditions surrounding vehicle 62 from camera 22 and classify them according to whether the traffic conditions are safe. If a traffic condition is deemed unsafe, system 200 can take appropriate action. Since images of unsafe traffic conditions are typically difficult to obtain, generalization ability is particularly important when learning to recognize them.

[0049] Various embodiments involve using a regularization objective to reward a machine-learnable function for a predictable response to a set of differences between input observations at a certain layer of the function. Specifically, the regularization objective may include evaluating mutual information between representations of the input observations, given difference labels between them at that layer, and / or evaluating interactive information between representations and difference labels. These choices of regularization objectives are now reasonable from a mathematical point of view—particularly with reference to the concept of equivariance described by Cohen et al. in “Group Equivariant Convolutional Networks”.

[0050] As described in Cohen's references, for a mapping from one space to another... , ,group The group element g and group actions If the mapping and transformation are interchanged, that is: Then the mapping It can be called - Equivariant, or equivariant in action relative to group G. Specifically, if If the transformation in space is an identity transformation, then the mapping... It can be called -Unchanging, that is: Therefore, an isovariant mapping is an example of a mapping that predictably responds to known differences in its inputs, especially when the inputs are correlated with each other by applying known transformations to the inputs.

[0051] However, what is interesting is that the inventors envisioned defining such predictability statistically, for example, based on the relationship between random variables. This yields a more general definition that is less stringent and does not require transformation based on group structure. This statistically based definition of predictability can be called "information equivariance." This definition can be driven by the following.

[0052] Using notations similar to isovariance, by space random variables in x The input observations are labeled to the machine-learnable function. Let the random variable... Label the second input observation and let the random variable... g Marking x and The differences between them. For example, in the isovariance setting, g It can be from The sampled group elements, and It can be That is, the transformed input observation, which passes through space Group actions in the equation are defined as g and A deterministic combination. Note that if g Independent of x So from The sample can be considered to be from Another iid sample. Given a random or deterministic mapping. The representations of the first and second input observations based on this mapping can be respectively labeled as follows: and .here, z It is a random variable; for example, if If it is deterministic, then its conditional distribution yes The point quality at that location. By using this notation, the predictability of how the mapping responds to differences between input observations can be based on... z and g conditional entropy We define it this way: the smaller the entropy, the higher the predictability. Specifically, this can be mathematically shown for groups... of - Isovariability implies minimum conditional entropy.

[0053] By using known identities, conditional cross-entropy can be decomposed into... Its assumption gIndependent of x and z And therefore, they do not share mutual information. Therefore, instead of minimizing conditional entropy, it is possible to instead, given the difference label... g Maximize the first and second representations in the case of z , Mutual information between conditions This allows, for example, the use of mutual information estimation models, as described elsewhere. Note that in deterministic... In this case, z (and The entropy of ) can be derived from x The entropy has a fixed upper bound, and therefore, such maximization can also explicitly maximize z and The entropy of the two is maximized because their mutual information is maximized only when the entropy of each individual is maximized.

[0054] Furthermore, it can be noted that the potential degenerate optimality is relative to the condition arising from... g Mappings that indicate differences invariant (e.g., ).although Maximizing entropy prevents the solution from becoming the global optimum, but the possibility of obtaining it as a local optimum still exists, especially when mutual information is estimated using a mutual information estimation model. Similarly, since invariance is all that is required to solve various discriminative learning tasks, the occurrence of this local optimum is to be expected. However, this local optimum is undesirable in terms of generalization ability.

[0055] To suppress invariance in the trained machine-learnable function and thus further improve generalization ability, an additional term can be included in the regularization objective to suppress cross-correlation between the first and second representations that is not conditioned on the difference label. Specifically, and z Mutual information between them (not based on) g (Assuming the condition) can be minimized. For example, the following regularization objective can be used: .

[0056] Interestingly, the regularization objective corresponds to the variable. , and g A known statistical measure of the interaction information among the three. This measure essentially represents the amount of information commonly contained among the three, which is not included in a subset of either of them. By using this interaction information, information equivariance mappings can be obtained without invariant solutions, and z and It can be rewarded as an independent entity.

[0057] Figure 4 A detailed, but non-limiting, example of the machine-learnable function MLF 040 is shown, in this case, a convolutional network. Such a convolutional network is also known as a convolutional neural network (CNN). A convolutional network is a special type of neural network. Neural networks are also called artificial neural networks. Neural networks typically consist of multiple layers of subsequent nodes. The parameters of a neural network typically include the weights of the neural network nodes. For example, the number of layers in the network can be at least 5 or at least 10, and the number of nodes and / or weights can be at least 1000 or at least 10000.

[0058] In a broad sense, a convolutional network (CNN) is a neural network in which at least one layer is a convolutional layer, as discussed below. Various CNN architectures are known, including so-called fully convolutional networks, where every layer of the network is a convolutional layer. This architecture is commonly used for semantic segmentation tasks in image classification, for example. Another known architecture is the one shown in the figure, where a CNN comprises two components: a convolutional part, including at least one convolutional layer; and a dense (e.g., fully connected) part, including at least one dense layer. This architecture is commonly used for detection tasks in image classification, for example.

[0059] For the sake of illustration, we will now describe how such a CNN can be used to obtain a function output FO 230 from an input observation IO 210. The input to a CNN is typically represented spatially, for example, as a fixed-size... The volume, where the dimensions are... d =1, for example, for grayscale images, and dimensions d =3 is typically used for color images. It is also known that other types of data (e.g., audio data) are represented as such a volume.

[0060] As shown in the figure, the input observation (IO) can be fed into a machine-learnable function (MLF), more specifically, into the first layer CL1 212-1 of multiple filters (or, in some cases, a single filter). Each filter can be convolved on the input observation (IO) to produce an activation volume or "latent representation" together. Each "slice" of the activation volume can be the output of one of the multiple filters. The latent representation can have a size similar to the input observation (IO). Compared to different sizes (e.g., spatial resolution), the latent representation can then be used as input to the subsequent layer CL2 212-2. Filters(s) of CL2 can then be convolved on the latent representation output from the first layer CL1 to produce a second latent representation. This process can continue for other layers 212. Throughout this convolutional portion of the CNN, the spatial relationship between the latent representation and the input observation is typically preserved, although the spatial dimension of the latent representation may vary depending on the latent representation and the spatial dimension of the input observation.

[0061] After the last layer CLk 212-k of the convolutional part of the CNN (often referred to as the last convolutional layer, although in some embodiments the last layer CLk of the convolutional part may not necessarily be a convolutional layer, as long as it preserves the spatial relationships discussed above), the resulting latent representation LRk 222 is fed into the first layer DL1214-1 in the dense layer in this example. The dense layer 214 can be a fully connected layer, where each neuron in the fully connected layer is connected to all neurons in another layer. After passing through the dense part of the CNN, an output function can be output, such as a classification score or a set of classification scores.

[0062] Figure 5a A detailed but non-restrictive example of training a machine-learnable function using a regularization objective is shown.

[0063] The example shown is the machine-learnable function Mlf 520. A machine-learnable function Mlf maps input observations to function outputs. Generally, a machine-learnable function Mlf can be a deterministic function (e.g., a regular neural network), but it can also be stochastic (e.g., a Bayesian neural network).

[0064] For example, an input observation can be an image or other types of sensor data represented as an image, or sensor data represented in different ways. For instance, an input observation can include a time series of (direct or indirect) measurements of a physical quantity, or multiple such time series. Various features are explained below for images, but these features also apply to other types of sensor data.

[0065] The output of a machine-learnable function (Mlf) can be of various types. For example, the output may include image classification output indicating whether an image or one or more segments of an image belong to one or more given classes. A machine-learnable function Mlf can be an image classifier, such as a detection model or a semantic segmentation model. For a detection model, the output may include one or more classification scores. For a semantic segmentation model, the output may include a classification output of fixed size that is similar to the input, representing the presence information of the corresponding segment for the input image. A machine-learnable function Mlf can also be, for example, an image feature extractor, in which case the output may be a vector of extracted features.

[0066] The output of the machine-learnable function Mlf is referred to herein as the “output layer” of the machine-learnable function, and the function output is referred to as the representation of the input observations at the output layer. For example, the representation of the input observations at the output layer may include at least 10, at least 100, or at least 1000 features. The input observations may include at least 100, at least 1000, or at least 1000 features. In some embodiments, such as in an object detection model, the output representation has fewer features than the input observations; however, in the case of a semantic segmentation model, for example, the number of features and / or spatial dimension of the output representation may be the same as the number of features and / or spatial dimension of the input observations.

[0067] Besides the output layer, a machine-learnable function (Mlf) may have one or more internal layers. That is, evaluating a machine-learnable function Mlf may involve evaluating one or more layers sequentially, with the output of a previous layer used as the input to the next layer. Machine-learnable functions with layers include neural networks (especially deep neural networks) and deep Gaussian processes. For example, a machine-learnable function Mlf could be a convolutional network (e.g., as referenced). Figure 4 (As described), especially fully convolutional networks. However, a hierarchical structure is not required for machine-learnable functions (MLFs); for example, a machine-learnable function can be a support vector machine, etc.

[0068] The machine-learnable function Mlf can be parameterized by a set of parameters (not shown in the figure), which can be learned during the training of the machine-learnable function. For example, parameters can include neural network weights, etc. For example, the number of parameters can be at least 1000 or at least 10000.

[0069] In the optimization operation Opt 570, parameters can be learned by optimizing an objective function. Various appropriate objective functions are known for the corresponding learning tasks, such as classification, regression, feature extraction, and autoencoding, and can be used here. Conventional training techniques can be used, including iterative optimization of the objective function. For example, stochastic optimization techniques such as stochastic gradient descent can be used, for instance, using the Adam optimizer disclosed in Kingma and Ba's "Adam: A Method for Stochastic Optimization" (available at https: / / arxiv.org / abs / 1412.6980, and incorporated herein by reference). As is known, such optimization methods can be heuristic and / or reach local optima. Training can be performed on an instance-by-instance basis or in batches (e.g., at most or at least 64 or at most or at least 256 instances).

[0070] This figure illustrates supervised learning on a training dataset based on multiple input observations and their corresponding training outputs. In this case, the objective function evaluated by optimizing Opt typically includes a training loss (sometimes called a reconstruction loss) that minimizes the difference between the function output based on the machine-learnable function and the corresponding training output. The figure shows the input observation IO1 511, which is fed into the machine-learnable function to obtain the corresponding function output FO1 531'. For the input observation, optimizing Opt compares the function output FO1 with the training output FO1 531 and adjusts the parameters of the machine-learnable function Mlf based on this comparison. Optimizing Opt can also perform unsupervised learning, in which case the machine-learnable function is typically still applied to the training instances, but the objective function is not based on the training output. Regardless of whether the training is supervised or unsupervised, the number of training input observations can be, for example, at least 1000, at least 10000, or at least 10000.

[0071] Interestingly, to reward the machine-learnable function Mlf for its predictable response to the set of differences between input observations at a layer of the machine-learnable function, a regularization objective Reg 560 can be included in the objective function being optimized. The layer to which the regularization objective is applied can be an output layer or an inner layer. It is also possible to include multiple corresponding regularization objectives for multiple corresponding layers of the machine-learnable function (e.g., inner or output layers). For ease of explanation, a single regularization objective is discussed here: multiple regularization objectives can be achieved by repeating the described operations for each regularization objective used.

[0072] The regularization target Reg can be defined based on the result of applying a machine-learnable function to a pair of input observations IO1 511 and IO2 512, which differ according to a given difference label DIF 513. Options for obtaining this pair of IO1, IO2 and the corresponding difference label DIF are discussed elsewhere. The set of difference labels is typically chosen to be discrete, for example, including at most or at least 5, at most or at least 10, or at most or at least 25 possible differences. However, in principle, it is also possible to use continuous difference labels. Given input observations IO1 and IO2, the machine-learnable function Mlf can be applied (at least partially) to obtain corresponding representations R1541 and R2542 of the input observations at the layer for which the regularization target Reg is defined. Based on representations R1 and R2 and the difference label DIF (typically based on multiple such triples R1, R2, DIF), the regularization target Reg can be evaluated.

[0073] The regularization objective REG can be configured to reward the cross-statistical correlation between the first representation R1 and the second representation R2 given the difference label DIF. As described elsewhere, various options exist for defining this regularization objective, including minimizing the conditional entropy of R2 given R1 and DIF, and maximizing the mutual information of R1 and R2 given DIF, for example... The regularization objective REG can also suppress cross-correlation between the first representation R1 and the second representation R2 that is not conditional on the difference label DIF, for example, by minimizing the mutual information between R1 and R2 that is not conditional on the difference label DIF. For example, the regularization objective can include interaction information between R1, R2, and DIF, such as... For example, based on the hyperparameter λ, the term... and Different weightings are also possible, for example... .

[0074] To evaluate the regularization objective REG, various conventional techniques for calculating or estimating the corresponding statistics can be used. However, this figure illustrates a regularization objective based on (conditional) mutual information, where (conditional) mutual information is estimated according to a trainable mutual information estimation model, Mie550. An example of such a trainable mutual information estimation model is disclosed in M. Belghazi et al., “MINE: Mutual Information Neural Estimation” (available at https: / / arxiv.org / abs / 1801.04062, and incorporated herein by reference). The corresponding mutual information estimation model can be used to estimate mutual information conditional on the difference label DIF, for example, ; and mutual information not conditional on difference labels, for example, For the former, MINE's technique can be adapted to estimate conditional mutual information directly by sampling from a distribution conditioned on difference labels. Interestingly, one or more mutual information estimation models can be optimized simultaneously with the training of a machine-learnable function Mlf in the optimization of Opt, said optimization being, for example, based on a single objective function. This allows the mutual information estimation to progressively adapt as the machine-learnable function is updated.

[0075] Note that, given the mutual information identity Interaction information can also be estimated, for example, using the mutual information estimation model described above. or .

[0076] Now, continuing with how to obtain the input observations and their corresponding difference labels in triples IO1, IO2, and DIF, several possibilities are envisioned. In some embodiments, such triples can be obtained as part of the training dataset. This can be referred to as a supervised setup. Figure 5b Another possibility illustrated in the diagram is that, within generation operation Gen 580, input observation IO2 is generated from input observation IO1 and difference type DIF. This can be referred to as a self-supervised setup. Any generation operation Gen suitable for the current application can be used. The difference type DIF can indicate the transformation to be applied to input observation IO1; for example, it can be a geometric transformation such as rotation and / or translation, and it can also be a synthetic modification of input observation IO1 to modify the properties of objects in the scene or modify the observation conditions. For example, an image of a vehicle environment can be adapted to introduce snow or rain, change a daytime scene to a nighttime scene, and so on.

[0077] A third option, not shown in the figure, is to determine the differential labels DIF from the input observations IO1 and IO2 using a machine-learnable labeling function. This function can be trained before performing the optimization Opt, but learning it simultaneously with the function Mlf (e.g., using multi-task learning) is generally preferred. The training of the labeling function can be supervised. For example, if differential labels are available for a subset of the input observations, the labeling function can be learned on that subset and applied to pairs where no labels are available. Interestingly, training can also be unsupervised, thus enabling the complete elimination of labeling effort. For example, the labeling function can be trained as described in M. Caron et al., “Deep Clustering for Unsupervised Learning of Visual Features” (available at https: / / arxiv.org / abs / 1807.05520, and incorporated herein by reference).

[0078] Figure 5c A detailed, but non-limiting, example is shown where the machine-learnable function being trained is an image feature extractor used in the global image classifier CI 590. That is, the global image classifier CI can classify an input observation IO, such as an input image, by applying a machine-learnable function Mlf to obtain a function output FO, which provides a feature vector extracted from the input observation IO; and then performing classification CI on the resulting extracted features FO using another machine-learnable function. In this case, to train the global image classifier, the machine-learnable function can be trained first, as described herein, without using classification labels. Therefore, input observation IOs without classification labels can also be used.

[0079] Subsequently, in optimization operation Opt2 579, the overall image classifier can be trained on training example IOs labeled with the desired training classification output TCO 595'. This optimization operation Opt2 learns at least the parameters of an additional machine-learnable function CI for the output classification; optionally, optimization Opt2 can also fine-tune the machine-learnable function Mlf. For this purpose, optionally, the regularization objective Reg for the machine-learnable function Mlf can be included in the objective function used to train the overall classifier.

[0080] Figure 6 A block diagram of a computer-implemented method 600 for training a machine-learnable function is shown. The machine-learnable function maps input observations to function outputs. The machine-learnable function may include one or more inner layers and / or an output layer. Method 600 may correspond to... Figure 1 The operation of system 100. However, this is not a limitation, because method 600 can also be performed using another system, apparatus or device.

[0081] Method 600 may include, in an operation entitled “Accessing Training Data, Parameters”, accessing 610 the following: a training dataset including multiple training input observations; and a set of parameters for a machine-learnable function.

[0082] Method 600 may include, in an operation titled "Learning Parameters," learning a set of parameters of a machine-learnable function 620 by optimizing an objective function. Optimization may include rewarding the machine-learnable function for predictably responding to a set of differences between input observations at a layer of the machine-learnable function. Optimization may reward said predictability by means of a regularization objective included in the objective function.

[0083] The optimization performed in learning operation 620 may include, in the operation titled "Obtaining Observations, Difference Labels", obtaining 622 the following: a first input observation from the training dataset; a corresponding second input observation; and difference labels from the difference set, indicating the difference between the first and second input observations. The optimization may also include, in the operation titled "Applying a Function to Determine Representations", applying 624 a machine-learnable function to determine first and second representations of the first and second input observations, respectively, at that layer of the machine-learnable function. The optimization may further include, in the operation titled "Rewarding the Relevance of Representations Given Difference Labels", evaluating 626 a regularization objective, wherein the regularization objective may be configured to reward the cross-statistical correlation between the first and second representations, given the difference labels between the first and second input observations.

[0084] To understand, generally speaking Figure 6The operations of method 600 can be performed in any suitable order, such as sequentially, simultaneously, or a combination thereof, which, where applicable, is subject to a particular order that is necessary, for example, by means of input / output relationships.

[0085] One or more methods can be implemented on a computer as a computer-implemented method, dedicated hardware, or a combination of both. Similarly, as... Figure 7 As illustrated, instructions for a computer (e.g., executable code) may be stored on a computer-readable medium 700, for example, in the form of a series 710 of machine-readable physical marks and / or as a series of elements with different electrical (e.g., magnetic or optical) properties or values. The executable code may be stored in a temporary or non-temporary manner. Examples of computer-readable media include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Figure 7 An optical disc 700 is shown. Alternatively, the computer-readable medium 700 may include temporary or non-temporary data 710 representing a set of parameters of a machine-learnable function trained by the computer-implemented methods provided herein.

[0086] Examples, embodiments, or optional features—whether or not indicated as non-limiting—should not be construed as limiting the claimed invention.

[0087] It should be noted that the embodiments mentioned above are illustrative rather than limiting of the invention, and those skilled in the art will be able to devise many alternative embodiments without departing from the scope of the appended claims. Any reference marks placed between parentheses in the claims should not be construed as limiting the claims. The use of the verb “comprising” and its variations does not exclude the presence of elements or stages other than those stated in the claims. The article “a” or “an” preceding an element does not exclude the presence of a plurality of such elements. Expressions such as “at least one of…” when preceding a list or group of elements indicate the selection of all elements or any subset of elements from that list or group. For example, the expression “at least one of A, B, and C” should be understood to include: only A; only B; only C; both A and B; both A and C; both B and C; or all A, B, and C. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In device claims enumerating several components, several of these components may be embodied by the same hardware item. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used advantageously.

Claims

1. A computer-implemented method (600) for training a machine-learnable function, the machine-learnable function mapping sensor data to a function output, the machine-learnable function comprising one or more inner layers and / or an output layer, the method comprising: Access (610): Training dataset including data from multiple training sensors; And the set of parameters for machine-learnable functions; (620) A set of parameters for a machine-learnable function is learned by optimizing an objective function, wherein the objective function includes a regularization term, and evaluating the regularization term includes the following steps: Obtain (622) first sensor data from the training dataset; corresponding second sensor data; and difference markers from the difference set, which indicate the differences between the first sensor data and the second sensor data; Apply (624) a machine-learnable function to determine a first representation of the first sensor data and a second representation of the second sensor data at a layer of the machine-learnable function, respectively; The method evaluates (626) the conditional mutual information of the first representation and the second representation given a difference label, wherein the machine-learnable function is an image classifier or an image feature extractor, wherein the machine-learnable function is an image feature extractor used in a global image classifier, the image feature extractor being trained on an unlabeled training dataset; the method further includes training a global image classifier on a labeled training dataset, and The method involves estimating mutual information based on a trainable mutual information estimation model, and further includes simultaneously training the mutual information estimation model with a machine-learnable function.

2. The method (600) of claim 1, wherein the objective function includes a plurality of corresponding regularization terms for a plurality of corresponding layers.

3. The method (600) according to claim 1 or 2, wherein, The first sensor data and the second sensor data are images representing the same type of scene, and the difference markers indicate differences in the attributes of objects in the scene and / or differences in the observation conditions of the scene.

4. The method (600) of claim 3, wherein the first sensor data and the second sensor data represent the environment surrounding the vehicle; the difference marker indicates at least one of the following: Differences in weather conditions, Differences in object occlusion levels in the environment Differences in object location within the environment, and Differences in object orientation within the environment.

5. The method (600) according to claim 1 or 2, wherein the machine-learnable function is a convolutional network.

6. The method (600) according to claim 1 or 2, comprising generating the second sensor data from the first sensor data and the difference marker.

7. The method (600) according to claim 1 or 2, comprising determining a difference label by applying a machine-learnable labeling function to the first sensor data and the second sensor data.

8. The method (600) according to claim 1 or 2 further includes obtaining sensor data via a sensor interface to provide observations of the computer control system and / or its environment; applying a machine-learnable function to the observations; and monitoring and / or controlling the computer control system based on the output of the machine-learnable function.

9. A system (100) for training a machine-learnable function that maps sensor data to a function output, the machine-learnable function including one or more inner layers and / or an output layer, the system comprising: Data interface (120) for accessing the following: training dataset (030) including multiple training sensor data; The set of parameters (040) of the machine-learnable function; The processor subsystem (140) is configured to learn a set of parameters of a machine-learnable function by optimizing an objective function, wherein the optimization function includes a regularization term, and wherein evaluating the regularization term includes the following steps: Obtain: first sensor data from the training dataset; corresponding second sensor data; and difference labels from the difference set, which indicate the differences between the first and second sensor data; A machine-learnable function is applied to determine a first representation of the first sensor data and a second representation of the second sensor data at a layer of the machine-learnable function, respectively. Estimating the conditional mutual information of the first representation and the second representation given a difference label, wherein the machine-learnable function is an image classifier or an image feature extractor, wherein the machine-learnable function is an image feature extractor used in an overall image classifier, the image feature extractor being trained on an unlabeled training dataset; The system also includes training an overall image classifier on a labeled training dataset, and The system estimates mutual information based on a trainable mutual information estimation model, and further includes training the mutual information estimation model simultaneously with a machine-learnable function.

10. A computer-readable medium (700) comprising temporary or non-temporary data (710) representing instructions that, when executed by a processor system, cause the processor system to perform the computer-implemented method of claim 1; and / or a set of parameters of a machine-learnable function trained by the computer-implemented method of claim 1.

Images