Physical-environment interactions with equivariant policies

Abstract

The present invention relates to a computer-implemented method (800) for interacting with a physical environment according to a policy. The policy determines multiple action probabilities for each action based on observable states of the physical environment. The policy includes a neural network parametrically specified by a set of parameters. The neural network determines the action probabilities by determining final layer inputs from the observable states and applying a final layer of the neural network to the final layer inputs. The final layer is applied by applying a linear combination of a set of equivariant basis weight matrices to the final layer inputs. The basis weight matrices are equivariant in the sense that, for a set of multiple predefined transformations of the final layer inputs, each transformation results in a corresponding predefined action permutation of the basis weight matrices output for the final layer inputs.

Description

[Technical field]

[0001] The present invention relates to a computer-controlled system for interacting with a physical environment according to policies, and to a corresponding computer-implemented method. The present invention further relates to a training system for configuring such a system, and to a corresponding computer-implemented method. The present invention further relates to a computer-readable medium comprising instructions for performing one of the above-mentioned methods, parameters for such policies, and / or basis weight matrix data for such policies. [Background technology]

[0002] It is well known to apply computer-implemented methods to interact with a physical environment. Typically, sensor data is obtained from one or more sensors, e.g., cameras, temperature sensors, pressure sensors, etc., and computer-implemented methods are applied to determine an action based on the sensor data, and actuators are used to realize the determined action in the physical environment, e.g., moving a robotic arm, activating a steering or braking system of an autonomous vehicle, or controlling the movement of an interventional medical (robotic) tool on a patient's body. The process by which an action is determined is often referred to as a computer-controlled interaction policy.

[0003] Computer-controlled systems include robotic systems, where a robot may perform one or more tasks automatically, for example under the control of an external device or an embedded controller. Further examples of systems that may be computer-controlled are vehicles and their components, domestic appliances, powered tools, manufacturing machinery, human assistance devices, access control systems, drones, nanorobots, and heating control systems. Various computer-controlled systems may operate autonomously in an environment, for example, as an autonomous robot, an autonomous agent, or an intelligent agent.

[0004] Examples in healthcare robotics, particularly in image-assisted therapy, include controlling the motion of imaging systems (e.g., x-ray, magnetic resonance, ultrasound systems) around a patient while taking into account the patient's anatomy, occlusions, and operating room equipment; robotically guiding diagnostic imaging devices within or outside a lumen, such as, but not limited to, a bronchoscope in the bronchi of the lungs or an intravascular ultrasound device in a vessel; steering deployable or non-deployable medical tools (e.g., flexible or non-flexible needles, catheters, guidewires, balloons, stents, etc.) toward an object based on x-ray or ultrasound images or other images, etc., to treat and / or measure biophysical parameters. A typical healthcare example of autonomous computer control is the dynamic adjustment of multiple imaging and display parameters (e.g., integration time, contrast), and filters for x-ray or ultrasound based on the current image content.

[0005] While interestingly in some instances it is possible to handcraft the policy, it is also possible to use a machine-learnable model, e.g. a neural network, as the policy. Such machine-learnable models are typically parametrically specified by a set of parameters that can be trained for a particular task. In "Proximal Policy Optimization Algorithms" by John Schulman et al., incorporated herein by reference and available at https: / / arxiv.org / abs / 1707.06347, a method for training such policies is disclosed. The method is from the field of reinforcement learning, where a set of parameters is optimized with respect to a given reward function. The method alternates between sampling data through interactions with the environment and optimizing parameters (in this case of a neural network) based on the sampled data. Summary of the Invention

[0006] According to a first aspect of the invention, a computer implemented method for interacting with a physical environment according to a policy is proposed as defined by claim 1. According to a further aspect of the invention, a computer implemented method for configuring a system for interacting with a physical environment according to a policy is proposed as defined by claim 8. According to further aspects, a computer controlled system for interacting with a physical environment and a training system for configuring such a system are proposed as defined by claims 12 and 13, respectively. According to other aspects of the invention, a computer readable medium is provided as defined by claims 14 and 15.

[0007] As is of course known, in various embodiments, a neural network, also known as an artificial neural network, is used as a policy for interacting with the physical environment. Such a neural network policy is parametrically specified by a set of parameters. The set of parameters is trained based on interaction data from real and / or simulated interactions. Having been trained to perform a particular task, the policy is deployed in a system to actually interact with the physical environment according to its set of parameters. During such interactions, observable states of the environment, e.g. camera images, are input to the policy. Based on this, an action is selected to be realized in the environment by an actuator, such as a robot arm. The set of actions is typically finite, e.g. a robot arm is driven left, right, up or down. The policy is stochastic in the sense that the policy output includes multiple action probabilities for performing each action, instead of directly returning a single action to perform, as is common in reinforcement learning.

[0008] As the inventors have realized, the (often physical) environments in which interactions take place exhibit various symmetries in terms of which actions are assumed to be beneficial in which observed states. For example, in an autonomous vehicle control system tasked with keeping the vehicle in a lane by steering left or right based on camera images, if the image is flipped horizontally, the desirability of steering left or right is similarly flipped. Various other kinds of symmetries in the observed states are possible as well, involving, for example, rotation, vertical or diagonal flips, etc. Multiple sensor measurements of the observable states are affected by symmetries in different ways, for example, if an autonomous vehicle uses camera images and left / right tilt sensors, horizontal mirror symmetry corresponds to a horizontal flip of the camera images and a negation of the angle measured by the tilt sensor. Possible actions are similarly affected by environmental symmetries in various ways, for example, some actions are swapped or otherwise substituted, while other actions are not affected by a particular symmetry at all. In general, symmetries are represented by a transformation (often a linear transformation) on the set of observable states and by a permutation of the set of possible actions. Such symmetries are also found in a healthcare environment by considering some symmetries in the patient's body (e.g., sagittal, frontal, and / or transverse planes and axes, bone motions around a central axis, symmetries between organs - e.g., right and left lungs, etc.), or in an operating room environment. This symmetry is found directly from the measurements or after pre-processing the results of those measurements to read out some symmetries.

[0009] The inventors have realized that by incorporating such symmetries in the neural network that computes the policy, much more efficient policies are obtained. For example, fewer parameters are needed to obtain a policy of the same quality with the same expected cumulative reward. Alternatively, for a fixed number of parameters, introducing symmetries allows policies to be obtained with higher expected cumulative reward. During training of the neural network, data efficiency is improved, e.g., the amount of observations required to reach a policy of a certain quality is reduced. The latter is particularly important when applying policies to physical (e.g., non-simulated) environments. In fact, the amount of data required to learn a policy can be very large. On the other hand, it is typically difficult to obtain a lot of observation data in non-simulated environments, e.g., interaction periods are limited, and failures have real-world costs.

[0010] In the field of image classification, the use of symmetries in neural networks is of course known. For example, "Group Equivariant Convolutional Networks" by TS Cohen and M. Welling (available at https: / / arxiv.org / abs / 1602.07576 and incorporated herein by reference) shows performing image classification on rotated handwritten digits. By incorporating translations, reflections, and rotations, a neural network is obtained that effectively learns to recognize digits regardless of how they are rotated. However, due to an important difference between image classification and reinforcement learning, such known group equivariant convolutional networks are not suitable for determining action probabilities. For example, the final layer of such a neural network outputs the same classification regardless of how the image is rotated. This is undesirable when determining action probabilities, since, as mentioned above, the desirability of performing an action changes with the transformation of the input. Furthermore, whereas standard group-homovariant convolutional networks typically rely on image invariance under translation, these are not usually expected to change in response to input translations in a predictable manner, so this is often not a useful kind of symmetry to consider when determining action probabilities.

[0011] Interestingly, however, the inventors have come up with a better kind of neural network that can effectively incorporate the combined state / action symmetry of the physical environment under consideration. In general, a neural network comprises multiple layers. Action probabilities are determined in the final layer of the neural network from the final layer inputs, which in turn are determined from the observable states. Interestingly, in various embodiments, the final layer is applied to the final layer inputs by applying a linear combination of a carefully defined set of basis weight matrices. The coefficients of this linear combination are included in the neural network parameter set. The output of applying the linear combination provides a pre-nonlinearity activation for some or all of the action probabilities, from which the action probabilities are computed, for example, using softmax. Interestingly, these basis weight matrices are defined to incorporate the combined state / action symmetry by being defined to be equivariant, and for a set of multiple predefined transformations of the final layer inputs, each transformation results in a corresponding predefined action permutation of the basis weight matrices output for the final layer inputs.

[0012] For example, the transformation of the final layer input is the matrix R θ The linear transformation is expressed as θ In this case, the basis weight matrix W is expressed as an equivariance relation P θ W=WR θ This means that for any final layer input, θ Wz=WR θ z. For example, the transformed final layer input R θ Transforming the final layer input z to obtain z is, in this case, an action permutation P of the original basis weight matrix output Wz of the untransformed final layer input z. θ The corresponding basis weight matrix output WR θ z is a substitution P θ Brings about becoming Wz.

[0013] If each basis weight matrix is ​​defined to be equivariant, then linear combinations of such basis weight matrices are also equivariant, thus preserving symmetries in the physical environment. Thus, the basis weight matrix set is effectively constrained to provide basis weight matrix outputs from which action probabilities are determined, reflecting the symmetries of the physical environment. Because symmetries are effectively taken into account, experience from one observed state is reused to derive actions in other transformed states. In particular, exploration is reused, as well as the vision part of the network as found in existing convolutional neural networks, which is particularly important in reinforcement learning due to the sparsity of rewards. Thus, by reflecting symmetries in the physical environment, fewer parameters are needed to obtain a neural network with a particular representation, providing improved efficiency both in use and training.

[0014] Interestingly, the techniques described herein are applied without the need to learn symmetries, e.g., from an environmental model or shaped reward symmetries; alternatively, symmetries can be specified a priori and used as disclosed herein, without the need to specify the entirety of policies, environmental models, or reward shaping. For example, it is not required to infer models as in various known model-based approaches, obviating the need for associated complex architectures containing many moving parts.

[0015] More specifically, with regard to the action permutations of the basis weight matrix output, these typically correspond to symmetries of the physical environment. It is assumed that a particular transformation of the observable state of the physical environment, such as a horizontal swap of an image of the environment, results in a particular permutation of actions being performed, such that performing a first action in the physical environment represented by the image corresponds to performing a second action in the physical environment represented by the swapped image. The action permutations are predefined, e.g., manually defined. The action permutations are obtained, e.g., as inputs for training a neural network for use in generating the basis weight matrix, but may also be implied as constraints on the basis weight matrix obtained from an external source.

[0016] With regard to transformations of the final layer inputs, these reflect the symmetries of the physical environment in various ways. Preferably, the final layer inputs are determined from the observable states by equivariant techniques, e.g., given multiple state transformations of the observable states, each state transformation resulting in a corresponding transformation of the final layer inputs. These transformations on the final layer inputs then correspond to action permutations as described above. The transformations are typically predefined, e.g., manually defined as part of the neural network design. The transformations may be explicitly given as inputs to the training of the neural network, e.g., to generate a set of basis weight matrices, or may be implicit as constraints on basis weight matrices obtained from an external source.

[0017] For example, the physical environment is assumed to satisfy a symmetry, e.g., mirror symmetry θ, where the observable states are equivariant to the action probabilities, i.e., each transformation Q of an observable state x θ x is a permutation P of action probability y θ y. Thus, the final layer inputs are specified to be equivariant with respect to the observable states and with respect to the action probabilities, and the transformation R θ is the transformation R of the corresponding final layer input for an observable state x. θ z is transformed into an observable state Q θx is the final input to the layer. Furthermore, as mentioned above, for the basis weight matrix W, the transformation R θ is the above-mentioned substitution P θ For example, P θ W=WR θ It is.

[0018] Interestingly, by including the corresponding observable state transformations, final layer input transformations, and action permutations, the entire neural network satisfies equivariance as well, providing, for example, action probabilities that reflect the environmental symmetries corresponding to these transformations and permutations. This may be true regardless of how strictly the previous layers guarantee equivariance (several possibilities are presented for this). By maintaining symmetry throughout the network, particularly efficient learning and accurate results are obtained.

[0019] For example, the final layer inputs are determined from the observable states by applying one or more layers of a known group equivariant neural network, e.g., as disclosed in "Group Equivariant Convolutional Networks." For example, a known neural network designed for image classification that is invariant to translations and swaps is used. A transformation of the inputs to such a network, e.g., a translation or a swap, results in a corresponding transformation of the layer outputs. Such layer outputs are used as final layer inputs for a policy as described herein, where transformations of the internal layer outputs that correspond to environmental symmetries are used to define the basis weight matrices of the final layer. These environmental symmetries typically do not include translations.

[0020] However, it is not required to use the group equivariant neural network disclosed in "Group Equivariant Convolutional Networks", and in particular, it is not required to use an inner layer that preserves translation as described in "Group Equivariant Convolutional Networks". Consistently, an example is provided. It is not even required to use a neural network to determine final layer inputs that explicitly take into account environmental symmetry, for example, a neural network is trained using the observable states and their transformations as inputs, where the neural network is encouraged via its loss function to determine final layer inputs that transform in an equivariant manner when applied to the transformed observable states. In fact, simply using the final layer as described herein is already sufficient to encourage the neural network to provide final layer inputs that transform according to the observable state transformations.

[0021] Regardless of the exact transformation of the final layer inputs and the corresponding action permutation, the basis weight matrix is ​​defined in various ways. For example, the basis weight matrix is ​​represented as a matrix, as a vector set applied as an inner product to the final layer inputs, etc. The basis weight matrices are derived from a sub-matrix for one input and / or output channel, for example. As described below, the basis weight matrix is ​​predefined, for example, pre-computed by a computer or manually, or computed when needed. In either case, typically, at least some of the basis weight matrices affect the multiple outputs of the final layer, thus reflecting that environmental symmetry limits the possible outputs of the neural network. Thus, the influence of the multiple outputs by the basis weight matrix can be considered as a kind of weight sharing between different outputs, thereby achieving a reduction in the number of parameters of the neural network.

[0022] The basis weight matrices may span the entire space of weight matrices reflecting the transformation and action permutation, but interestingly, this is not required and the basis weight matrix set may span only a subspace of allowed basis weight matrices, for example a randomly sampled subspace. This allows a very significant reduction in the number of parameters, especially since the number of basis weight matrices would otherwise be overwhelming. It is therefore possible to maintain equivariance and provide a trade-off between performance and expressiveness of the neural network layers.

[0023] The linear combination of basis weight matrices described above is applied both when interacting with the physical environment according to the policy, and when configuring, e.g., training, a system for implementing such environment interactions. In both cases, fewer parameters are required for the final layer of the neural network, thereby increasing the efficiency of both training and using the neural network.

[0024] Optionally, the environmental symmetries that govern the transformations and permutations form a mathematical group, e.g., the environmental symmetry set includes the identity symmetry, is closed under composition of transformations, satisfies the associative law, and is closed under inverse. These properties are natural properties for symmetries, e.g., if a policy must reflect one rotation of an image by 90 degrees, it must also reflect repeated rotations by 90 degrees and -90 degrees. By considering the full symmetry group, the available symmetries are more optimally used by the model.

[0025] Optionally, the sensor data includes images of the physical environment. In various applications, the images provide useful information about environmental conditions, such as traffic conditions for autonomous vehicles, medical environments for medical tools, intermediate products for manufacturing robots, etc. Images often exhibit various symmetries, such as rotations or mirroring, corresponding to action displacements of actions performed using actuators. In such cases, the use of techniques provided herein to incorporate such state / action state symmetries is particularly effective.

[0026] Optionally, the feature transformation corresponds to a rotation of the image of the physical environment and / or the feature transformation corresponds to a reflection of the image. For example, the reflection may be, for example, a mirroring on a central axis of the image or 3D scene. In one embodiment, the feature transformation corresponds to a 180 degree rotation of the image. In one embodiment, the set of feature transformations includes a 90 degree rotation, a 180 degree rotation, and a 270 degree rotation. In one embodiment, the set of transformations includes a horizontal mirroring and a vertical mirroring. Such environmental symmetries often occur in practice and are therefore particularly useful for incorporating into neural networks for policies.

[0027] Optionally, the sensor data includes one or more further sensor measurements in addition to the image. Symmetries of the physical environment affect such measurements in a different manner. For example, the one or more further sensor measurements are invariant under environmental symmetries that affect the image, e.g., the image is swapped, whereas the temperature measurements are not affected. Interestingly, however, the one or more further sensor measurements also change with the input image, e.g., when the input image is swapped horizontally, the measurement of the angle with respect to the horizontal plane is inverted. Interestingly, such transformations of the further sensor measurements are also taken into account by the neural network, allowing such sensor measurements to be effectively used to determine action probabilities.

[0028] Optionally, applying the final layer comprises applying a further linear combination of the basis weight matrix set to the final layer inputs, apart from the original linear combination mentioned above. The coefficients of the further linear combination are included in the parameter set in addition to the coefficients of the original linear combination. For example, the set of possible actions to be performed includes multiple subsets, each of which is separately affected by the action permutation. For example, actions a1 and a2 are swapped by the action permutation, and actions a3 and a4 are swapped independently. In such a case, instead of obtaining a basis weight matrix set to simultaneously compute outputs for a1 through a4, a basis weight matrix set is obtained, which is then applied to first compute outputs for a first subset of actions a1 and a2, and then further compute outputs for a second subset of actions a3 and a4 using a different parameter set. Thus, the basis weight matrix sets can be reused, reducing the storage required to maintain them and, if applicable, the computational resources required to further compute them.

[0029] Optionally, applying the final layer further comprises applying a further linear combination of a further basis weight matrix set to the final layer input. The further basis weight matrix set is obtained in the same way as the original basis weight matrix set. However, interestingly, the further basis weight matrix set is equivariant to another set of transformations, i.e., for this further set of multiple predefined transformations of the final layer input, each transformation results in a corresponding further predefined action permutation of the further basis weight matrix output to the final layer input. Thus, the set of possible actions includes a first subset, the action probabilities of which are determined using the original basis weight matrix set, and a second subset, the action probabilities of which are determined using the further basis weight matrix set. Although the entire one basis weight matrix set can be further used to determine action probabilities for both sets of possible actions, interestingly, a smaller number of basis weight matrices is sufficient, and the basis weight matrices themselves are smaller, so that using different basis weight matrices is more efficient. Preferably, the final layer inputs are equivariant to both the original and further transformation sets as described above.

[0030] Optionally, determining the action probabilities further comprises applying a softmax to the output of applying the linear combination of the basis weight matrices to at least the final layer inputs. The linear combination of the basis weight matrices provides a value indicating the relative desirability of performing each action. By applying a softmax to such relative desirability values, a probability distribution over the actions is obtained. For example, a softmax is applied to the output of the linear combination of the basis weight matrices and, optionally, other linear combinations that provide desirability values ​​for further possible actions.

[0031] Optionally, the final layer input includes multiple feature vectors corresponding to respective transformations of the observable state. For example, transforming the observable state according to state-action symmetry results in permutations of multiple feature vectors. In other words, the transformations of the final layer input include permutations of multiple feature vectors and can therefore be relatively efficiently evaluated. For example, known swarm convolutional neural networks typically provide this type of feature vector. The feature vector includes, for example, one, up to or at least two, or up to or at least five features. Similarly, some or all other layers of the neural network include multiple feature vectors corresponding to respective transformations of the observable state.

[0032] Optionally, features of the final layer input, or inputs to earlier layers, are determined by average pooling over feature vectors corresponding to translations of the observable states. For example, earlier layers of the neural network provide feature vectors each corresponding to performing both a translation of the observable states and another transformation, e.g., as disclosed in "Group Equivariant Convolutional Networks." The feature vectors corresponding to the various translations and the particular other transformation are average pooled to obtain a feature vector for the other transformation, thus allowing the object recognition capabilities of the translation-equivariant neural network to be used in earlier layers, while providing a more compressed input to later layers.

[0033] Optionally, the multiple action probabilities include at least one action probability that is invariant under each of the action permutations and at least one action probability that is not invariant under some of the action permutations. For example, two actions, such as "move left" and "move right", are swapped under an action permutation that corresponds, for example, to mirroring the input image, while another action, such as "do nothing", is unaffected by this transformation of the input. Interestingly, the techniques provided herein are powerful enough to represent these kinds of action permutations, and more generally, other kinds of action permutations that do not have a one-to-one correspondence with the symmetry of the physical environment.

[0034] Optionally, further basis weight matrices are obtained for further layers of the neural network. In particular, a set of further basis weight matrices for the further layer of the neural network is obtained, where transforming the further layer inputs according to a transformation from a set of multiple predefined transformations results in a corresponding predefined transformation of the further basis weight matrices output for the further layer inputs. To evaluate this further layer, a linear combination of the set of further basis weight matrices is applied to the further layer inputs. For example, the transformation of the further layer inputs and the further layer outputs corresponds to the symmetry of the physical environment in the same way as the final layer inputs, as described above. By using the linear combination of the basis weight matrices in other layers of the neural network, preferably all layers of the neural network, a better reduction of the parameter set of the neural network is achieved.

[0035] Optionally, when training the policy, in other words optimizing its parameter set, a basis weight matrix set for the final layer is automatically determined from a number of predefined transformations and corresponding predefined action permutations. Further sets of basis weight matrices for further layers are similarly automatically determined. In some examples, it is possible to manually determine the basis weight matrix set, but such manual operations may be infeasible and cumbersome to perform multiple times, especially for larger layer sizes and / or more symmetries. For larger layer sizes, approximate basis weight matrix sets are determined, e.g., basis weight matrix sets that provide equivariance but do not necessarily span the entire set of possible equivariances, thus providing a further reduction in the number of parameters without affecting equivariance.

[0036] Determining the set of basis weight matrices for the final layer may, in various embodiments, be done by: θ , and each permutation P of the basis weight matrix output θ For the other layers, the formula P θ W=WR θ This is expressed as determining a set of weight matrices W that satisfy the formula P θ W=WR θ yields a linear system in the elements of W that can be solved using conventional techniques. Similar equations can be defined for the other layers.

[0037] Optionally, the basis weight matrix is ​​obtained by taking an initial weight matrix W, applying a transformation and inverse projection of the corresponding action permutation to the initial weight matrix, and adding the transformed and permuted initial weight matrices together. θ , R θ In contrast, the inventors have determined that W'=Σ θ P θ -1 WR θ To calculate P θ , R θ The equivariance relation P associated with each of θ W=WR θIt has been noticed that the randomized approach provides a weight matrix that satisfies: ##EQU1## Thus, a candidate basis weight matrix is ​​obtained. Optionally, the candidate basis weight matrix set obtained by this approach is further improved by orthogonalizing the basis weight matrix, for example, by vectorizing the basis weight matrix, orthonormalizing the vectorized basis weight matrix, and de-vectorizing the orthonormalized vectorized basis weight matrix. Thus, a basis weight matrix set that provides a good representation of the entire basis weight matrix set is obtained by the randomized approach.

[0038] Optionally, the parameter set is optimized using a policy gradient algorithm. Various policy gradient techniques, such as the PPO method disclosed in "Proximal Policy Optimization Algorithms" by John Schulman et al., are combined with the neural network provided herein. Due to the incorporation of state / action symmetry in the neural network and the resulting reduction in the number of parameters, the technique provided herein allows for significant improvements in data efficiency over policy gradient algorithms. It should be noted that although PPO is a so-called model-free reinforcement learning technique, the technique described herein is also applicable to model-based reinforcement learning.

[0039] It will be appreciated by those skilled in the art that two or more of the above-described embodiments, implementations and / or optional aspects of the invention may be combined in any manner deemed useful.

[0040] Modifications and variations of any system and / or any computer-readable medium corresponding to the described modifications and variations of the corresponding computer-implemented method may be implemented by one of ordinary skill in the art based on this description.

[0041] These and other aspects of the invention will be apparent from and will be elucidated with reference to the embodiments described in the following description, by way of example only, with reference to the accompanying drawings, in which: FIG. [Brief description of the drawings]

[0042] [Figure 1] FIG. 1 illustrates a computer-controlled system for interacting with a physical environment according to a policy. [Diagram 2] FIG. 1 illustrates a training system for configuring a computer-controlled system that interacts with a physical environment according to a policy. [Diagram 3] FIG. 1 illustrates a computer-controlled system for interacting with a physical environment, in this case an autonomous vehicle. [Figure 4] FIG. 2 illustrates a detailed example of a neural network policy for interacting with a physical environment. [Figure 5a] FIG. 13 illustrates an example of a transformation of an observable state. [Figure 5b] FIG. 13 illustrates an example of transformation of final layer input. [Figure 5c] FIG. 13 illustrates an example of action substitution of an action to be performed. [Figure 6] FIG. 1 illustrates a computer-implemented method for interacting with a physical environment according to a policy. [Figure 7] FIG. 1 illustrates a computer-implemented method for configuring a system that interacts with a physical environment according to a policy. [Figure 8] FIG. 1 illustrates a computer readable medium containing data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0043] It should be noted that the figures are only schematic and are not drawn to scale, in which elements corresponding to elements already described may be provided with the same reference numbers.

[0044] FIG. 1 shows a computer controlled system 100 for interacting with a physical environment 081 according to a policy. The policy determines a number of action probabilities for respective actions based on observable states of the physical environment 081. The policy includes a neural network parametrically defined by a parameter set 040. The neural network determines the action probabilities by determining final layer inputs from the observable states and applying a final layer of the neural network to the final layer inputs. The system 100 includes a data interface 120 and a processor subsystem 140 in internal communication via a data communication 121. The data interface 120 is for accessing the parameter set 040 of the policy. The data interface 120 is for accessing basis weight matrix data 030 as described below. The parameter set 042 and / or the basis weight matrix data 030 are determined according to the methods described herein, for example, by the system 200 of FIG. 2.

[0045] The processor subsystem 140 is configured to access the data 030, 040 during operation of the system 100 and using the data interface 120. For example, as shown in FIG. 1, the data interface 120 provides access 122 to an external data storage 021 comprising the data 030, 040. Alternatively, the data 030, 040 is accessed from an internal data storage that is part of the system 100. Alternatively, the data 030, 040 is received over a network from another entity. For example, while the system 100 is configured, the data 030, 040 is obtained from the system 200 of FIG. 2, for example, multiple times for each environment interaction. In general, the data interface 120 takes various forms, such as a network interface to a local or wide area network, for example the Internet, a storage interface to an internal or external data storage, etc. The data storage 021 takes any known and suitable form.

[0046] The system 100 includes an image input interface 160, or any other type of input interface, for acquiring sensor data 124 from one or more sensors, such as, for example, a camera 071, indicative of observable conditions of a physical environment. For example, the camera is configured to capture image data 124, and the processor subsystem 140 is configured to identify the observable conditions based on the image data 124 acquired via data communication 123 from the input interface 160. The input interface is configured for various types of sensor signals indicative of physical quantities of the environment and / or the device 100 itself, and combinations thereof, such as video signals, radar / LiDAR signals, ultrasonic signals, etc.

[0047] In some embodiments, the sensor is located in the environment 081. In other examples, the sensor is located remotely from the environment 081, for example if the quantity can be measured remotely. For example, a camera-based sensor is located outside the environment 081, but nevertheless measures a quantity related to the environment, for example the position and / or orientation of a physical entity in the environment. The sensor interface 180 further accesses the sensor data from other cases, for example from a data storage or network location. The sensor interface 180 may have any suitable form, including but not limited to a low-level communication interface, for example based on I2C or SPI data communication, as well as a data storage interface, for example a memory interface or a persistent storage interface, or a personal, local, or wide area network interface, for example a Bluetooth, ZigBee, or Wi-Fi interface, or an Ethernet, or a fiber optic interface. The sensor is part of the system 100.

[0048] The system 100 comprises an actuator interface 180 for providing actuator data to the actuator, which causes the actuator to realize an action in the physical environment 081 of the system 100. For example, the processor subsystem 140 is configured to determine the actuator data based at least in part on an action probability determined by a policy as described herein. For example, the policy detects an exceptional condition, e.g., a collision risk, based on which a safety system, e.g., a brake, is activated. There may further be a number of actuators implementing the respective actions. The actuators may be electric, hydraulic, pneumatic, thermal, magnetic, and / or mechanical actuators. Specific, but non-limiting examples include electric motors, electroactive polymers, hydraulic cylinders, piezoelectric actuators, pneumatic actuators, solenoids, stepper motors, servo mechanisms, etc. The actuators are part of the system 200.

[0049] The processor subsystem 140 is configured during operation of the system 100 and using the data interface 120 to obtain basis weight matrix data 030 representing a basis weight matrix set for a final layer of the neural network, where for a set of multiple predefined transformations of the final layer inputs, each transformation results in a corresponding predefined action permutation of the basis weight matrix output for the final layer input. The processor subsystem 140 is further configured to control an interaction with the physical environment by repeatedly obtaining sensor data from one or more sensors indicative of an observable state of the physical environment via the sensor interface 160, determining action probabilities, and providing actuator data 126 to the actuator via the actuator interface 180 to cause the actuator to realize an action in the physical environment based on the determined action probabilities. The processor subsystem 140 is configured to determine the action probabilities based on the observable state, including applying the final layer of the neural network by applying a linear combination of the basis weight matrix set to the final layer inputs, where coefficients of the linear combination are included in the parameter set.

[0050] Various details and aspects of the operation of system 100, including optional aspects thereof, are further described with reference to FIGS.

[0051] Generally, the system 100 is embodied as or in one device or apparatus, e.g., a workstation, e.g., a laptop, or desktop-based, or a server. The device or apparatus comprises one or more microprocessors that execute appropriate software. For example, the processor subsystem may be embodied by one central processing unit (CPU), or by a combination or system of such CPUs and / or other types of processing units. The software may be downloaded and / or stored in a corresponding memory, e.g., a volatile memory, e.g., a RAM, or a non-volatile memory, e.g., a flash. Alternatively, the functional units of the system, e.g., the data interface and the processor subsystem, are realized in the device or apparatus in the form of programmable logic, e.g., a field programmable gate array (FPGA) and / or a graphics processing unit (GPU). Generally, each functional unit of the system is realized in the form of a circuit. It is noted that the system 100 may also be realized in a distributed manner, e.g., with different devices or apparatus, e.g., distributed servers, e.g., in the form of cloud computing.

[0052] 2 shows a training system 200 for configuring a computer-controlled system that interacts with a physical environment according to policies as described herein. For example, training system 200 is for configuring system 100. Training system 200 and system 100 may be combined into one system.

[0053] The training system 200 comprises a data interface 220 and a processor subsystem 240 in internal communication via a data communication 221. The data interface 220 is for accessing a parameter set 040 of a policy. The data interface 220 is also for accessing basis weight matrix data 030 representing a set of basis weight matrices for a final layer of a neural network, where for a set of multiple predefined transformations of the final layer inputs, each transformation results in a corresponding predefined action permutation of the basis weight matrix output for the final layer input.

[0054] The processor subsystem 240 is configured to access the data 030, 040 during operation of the system 200 and using the data interface 220. For example, as shown in FIG. 2, the data interface 220 provides access 222 to an external data storage 022 containing the data 030, 040. Alternatively, the data 030, 040 is accessed from an internal data storage that is part of the system 200. Alternatively, the data 030, 040 may be received over a network from another entity. In general, the data interface 220 takes various forms, such as, for example, a network interface to a local or wide area network, such as the Internet, a storage interface to an internal or external data storage, etc. The data storage 022 takes any known and suitable form.

[0055] The processor subsystem 140 is configured to optimize a parameter set 040 of the policy to maximize an expected reward of interacting with the environment accordingly during operation of the system 100 and using the data interface 120. To optimize the parameter set, the processor subsystem 140 is configured to repeatedly acquire interaction data indicative of sequences of observed environmental states and corresponding actions performed by the configured computer-controlled system, determine rewards for the interactions, determine action probabilities of a policy that selects a corresponding action in an observed state of the sequence of observed environmental states, and adjust the parameter set 040 to increase the expected reward based on the determined reward and action probabilities. To apply the final layer of the neural network, the processor subsystem 140 applies a linear combination of the basis weight matrix set 030 to the final layer inputs, the coefficients of the linear combination being included in the parameter set 040.

[0056] The system 200 further comprises a communication interface (not shown) configured for communication with configured systems, such as the system 100. For example, the system 200 obtains interaction data of one or more environment interactions of the other systems via the communication interface. The interaction data is obtained before optimizing the parameter set and / or during optimizing the parameter set. In the latter case, the system 200 further provides the current parameter set 040 of the policy to the other systems to have them interact with the environment according to the current policy. Various known types of communication interfaces are used and configured, for example, for direct communication with the other systems 200, for example, using a USB, IEEE 1394, or similar interface, or via a computer network, such as a wireless personal area network, Internet, intranet, LAN, WLAN, etc. The communication interface may be an internal communication interface, such as a bus, an API, a storage interface, etc.

[0057] Various details and aspects of the operation of system 200, including optional aspects thereof, are further described with reference to FIGS.

[0058] Generally, the system 200 is embodied as or in one device or apparatus, e.g., a workstation, e.g., a laptop, or desktop-based, or a server. The device or apparatus comprises one or more microprocessors that execute appropriate software. For example, the processor subsystem may be embodied by one central processing unit (CPU), or by a combination or system of such CPUs and / or other types of processing units. The software may be downloaded and / or stored in a corresponding memory, e.g., a volatile memory, e.g., a RAM, or a non-volatile memory, e.g., a flash. Alternatively, the functional units of the system, e.g., the data interface and the processor subsystem, are realized in the device or apparatus in the form of programmable logic, e.g., a field programmable gate array (FPGA) and / or a graphics processing unit (GPU). Generally, each functional unit of the system is realized in the form of a circuit. It is noted that the system 200 may also be realized in a distributed manner, e.g., with different devices or apparatus, e.g., distributed servers, e.g., in the form of cloud computing.

[0059] Figure 3 illustrates one example as mentioned above in that an automobile control system 300 is shown for controlling a vehicle 62 comprising a system for interacting with a physical environment according to a policy according to one embodiment, such as the system 100 of Figure 1 (not shown separately). The vehicle 62 is an autonomous or semi-autonomous vehicle, although this is not required, for example the system 300 may be a driver assistance system for a non-autonomous vehicle 62. For example the vehicle 62 incorporates an interaction system for controlling the vehicle based on images acquired from a camera 071. For example the automobile control system 300 comprises a camera interface (not shown separately) for acquiring images of the vehicle's environment 081 from the camera 071.

[0060] The control system 300 further comprises an actuator interface (not shown separately) for providing actuator data to the actuators, which causes the actuators to realize actions for controlling the vehicle 62 in the physical environment 081. The vehicle control system 300 is configured to determine actuator data for controlling the vehicle 62 based on the action probabilities determined by the policy, and to provide the actuator data to the actuators via the actuator interface. For example, the actuators are caused to control the steering and / or braking of the vehicle. For example, the control system controls or assists the steering of the vehicle 62 by turning the wheels 42. For example, the wheels 42 are turned left or right to keep the vehicle 62 within the lane. The policy is configured in this case to be equivariant under horizontal swaps, e.g., if the camera image 071 is swapped horizontally, the action of turning the wheels left or right is replaced.

[0061] 4 shows a detailed, non-limiting example of a neural network NN 400 of a policy for interacting with a physical environment, for example, the neural network NN applied in the system 100 of FIG 1 and / or in the system 200 of FIG 2.

[0062] As shown in the figure, the neural network NN is configured to determine multiple action probabilities AP1 491 to APn 492 for each action. A different number of action probabilities are possible, e.g., two, three, up to or at least five, or up to or at least ten. Each action corresponds to a signal provided to one or more actuators to perform a particular action, e.g., "move left", "move right", "move up", "move down", etc. One of the actions is a no-operation, e.g., an action that does not affect the physical environment. The neural network NN is configured to determine the action probabilities in such a way that the action probabilities sum to one, e.g., by a softmax function.

[0063] The neural network NN determines the action probabilities APi based on the observable state OS 410 of the physical environment. The observable state typically includes one or more sensor measurements, e.g. an image of the physical environment obtained from a camera, and / or one or more further sensor measurements. The observable state OS is represented by a feature vector of, e.g., up to or at least 100, or up to or at least 1000 features. The observable state OS includes or is based on a number of previous sensor measurements, e.g. the observable state OS includes a rolling history of a fixed number of recent sensor measurements, a rolling average of recent sensor measurements, etc. Depending on the application, further different types of processing, e.g. image processing such as scaling, may be performed to obtain the observable state OS from the sensor measurements. In general, the observable state OS encompasses different types of sensor data, audio data, video data, radar data, lidar data, ultrasonic data, or different individual sensor readings, or a history thereof.

[0064] The neural network NN is parametrically defined by a parameter set PAR 440. For example, the parameter set PAR includes the coefficients of the basis weight matrix for the final layer or for other layers as described herein. For example, the number of layers of the neural network NN is at least 5 or at least 10, and the number of parameters PAR is at least 1000 or at least 10000. It is beneficial from the point of view of training efficiency to use a neural network NN that is adapted to gradient-based optimization, e.g. continuous and / or differentiable in its parameter set. Neural networks are also known as artificial neural networks.

[0065] In various embodiments, the parameters PAR of the neural network NN are optimized to maximize the expected reward of interacting with the environment according to the corresponding policy. For example, the expected reward is the expected cumulative reward defined by modeling the environmental interactions as a Markov decision process (MDP). Mathematically, an MDP is a tuple (S, A, R, T, γ), where S is the space of possible environmental states OS, A is the space of possible actions,

number

[0066] In various embodiments, the neural network NN is trained to incorporate state / action symmetries of the physical environment in which the interaction occurs. The symmetry set is denoted as Θ. The set Θ is typically assumed to have a mathematical group structure, e.g., Θ has identity symmetry, symmetry composition

number

number

number

number

number

[0067] Symmetries typically affect both the observable states OS and the set of possible actions for which action probabilities AP1, ..., APn are determined. For example, for each symmetry θ, a transformation Q θ For example, the transformation Q θ rotates or reflects the input image contained in the observable state. Typically, Q θ is a linear transformation, e.g., represented by a matrix. Furthermore, for each symmetry θ, a permutation P θis defined and further represented, for example, by a matrix. The techniques provided herein are powerful enough to support various kinds of permutations, e.g., in some embodiments, the action probabilities include at least one action probability that is invariant under each action permutation and at least one action probability that is not invariant under some of the action permutations. Interestingly, in various embodiments, the neural network NN is configured to be equivariant with respect to these observable state transformations and action permutations, e.g., it is performed or at least stimulated that transforming the states OS and then computing the action probabilities APi results in the same output as computing the action probabilities APi and then permuting them, e.g., P θ [π ω ](·|s)=π ω (|Q θ [s]) where Q θ is the transformation of the observable state s, and P θ is the corresponding action permutation. Symmetries and the corresponding transformations of observable states, action probabilities, and hidden layer feature vectors are typically specified manually.

[0068] As shown in the figure, the neural network NN determines the action probabilities APi in an operation Ls 420 by determining the final layer inputs FLI 450 from the observable state OS and then applying the final layer of the neural network to the final layer inputs FLI. Θ (θ∈Θ) is the corresponding transformation R of the final layer input F Θ The operation Ls is constructed to be equivariant to the symmetry Θ in the sense that it yields: However, as will be explained, this is not strictly required to be accomplished, e.g., equivariance can be approximated. Various techniques for determining the final layer input FLI are described in more detail below.

[0069] Interestingly, the final layer of the neural network NN is configured to maintain equivariance through the use of a set of basis weight matrices BWM 430 that are equivariant with respect to a desired set of symmetries of the physical environment. For example, each transformation R of the final layer input z corresponding to such symmetries θ z is the corresponding action permutation P of the basis weight matrix output Wz of applying the basis weight matrix to the final layer input z θ For example, for the final layer input FLI, z, and the symmetry θ∈Θ, θ WZz=WR θ z. The parameter set PAR includes coefficients corresponding to each basis weight matrix, and the final layer of the neural network NN is applied by applying a linear combination LC460 of the basis weight matrix set BWM to the final layer input FLI using the coefficients given by the parameter set PAR. Interestingly, if the basis weight matrices are equivariant, the linear combination is also equivariant, and thus an equivariant linear combination output LCO470 is obtained. Thus, a softmax SMX480 is applied to at least the linear combination output LCO to obtain the action probability APi.

[0070] As an example, the basis weight matrix W1, ..., Wk and the corresponding linear combination coefficients C1, ..., Ck are shown in the figure. When applying a neural network NN, the basis weight matrix and the coefficients are typically fixed. When training a neural network, at least the coefficients are typically trained, e.g., the expected reward of interacting with the environment is maximized with respect to the coefficients.

[0071] The basis weight matrix set BWM can be specified by various methods. For example, mathematically, the equivariance of the weight matrix W applied in the final layer of the neural network NN is expressed as follows: P θ Wz=WR θ z, ∀z∈Z, θ∈Θ P θ W=WR θ

number

number

number

number

[0072] The above set W may be further defined for one input and / or output channel, where the basis weighting matrices BWi are obtained from the matrix W∈W by applying the matrices to reflect the input channels and / or to obtain the respective output channels.

[0073] Interestingly, it is not required that the basis weight matrix set BWM spans the entire set of equivariant weight matrices. For example, BWM is sampled as a randomly sampled subspace of the space of equivariant weight matrices. In this way, a reduction in the number of parameters C1, ..., Ck is achieved while still maintaining equivariance, which is especially important when the space of equivariant weight matrices is relatively large.

[0074] In particular, one approach to automatically determine the basis weight matrix BWM is as follows: First, one or more initial weight matrices W i is obtained, e.g., its coefficients are randomly sampled, e.g., from a univariate Gaussian distribution or the like. Then, the basis weight matrix

number

number

number

number

number

number

number

number

[0075] Thus, it has been described how, by computing a linear combination of the basis weight matrices BWM, the linear combination output LCO of the final layer of the neural network is determined corresponding to one or more possible actions to be taken.

[0076] It is also possible to apply a further linear combination of the same basis weight matrix set BWM to the final layer input FLI to obtain a further linear combination output. The coefficients of this further linear combination are included in the parameter set PAR in addition to the original linear combination coefficients Ci. This is particularly attractive if the action corresponding to the further linear combination output must be replaced in the same manner as when applying environmental symmetry as the original linear combination output LCO. The use of larger basis weight matrices to simultaneously operate both sets of linear combination outputs, which requires the use of more and / or larger basis weight matrices, is avoided by applying the same basis weight matrix twice.

[0077] Further, alternatively or in addition, it is possible to apply a further different linear combination of a further set of basis weight matrices to the final layer input FLI. For a further set of multiple predefined transformations of the final layer input, each transformation results in a corresponding further predefined action permutation of the further basis weight matrices output to the final layer input. In other words, the actions whose probabilities are determined using this further linear combination are permuted according to the environmental symmetry in a different form than the actions for the original final layer output. The further translation set of the final layer input is equal to the original transformation set of the final layer input, e.g., the actions corresponding to the further different linear combination are equivariant to the same symmetry, but according to a different permutation. The further translation set of the final layer may be different, e.g., the actions corresponding to the further different linear combination are equivariant to different environmental symmetry sets. In this latter example, the final layer input FLI is preferably equivariant under both symmetry sets.

[0078] Proceeding now to the computation of the final layer inputs FLI based on the observable state OS and the definition of the transformation of the final layer inputs, various embodiments are envisaged.

[0079] In general, the final layer input FLI is obtained by applying one or more layers of a group-univariant convolutional network to the observable state OS. The filters of the group-univariant convolutional network are

number

number

number

number

number

[0080] Specifically, in some embodiments, the final layer input FLI includes multiple feature vectors FV1 451, ..., FVm 452, each corresponding to a respective transformation of the observable state. For example, the transformation of the final layer input permutes the multiple feature vectors according to the group structure of the environmental symmetry, e.g.,

number

number

[0081] However, regardless of how exactly the transformations of the intermediate layer inputs and outputs are specified, it is also possible to use linear combinations of basis weight matrices in some or all other layers of the neural network to compute layer outputs. For example, further basis weight matrix data representing a set of further basis weight matrices for a further layer of the neural network is obtained, where transforming the further layer inputs according to a transformation from a set of multiple predefined transformations results in a corresponding predefined transformation of the further basis weight matrix output for the further layer input. To evaluate the further layer, the linear combination of the set of further basis weight matrices is applied to the further layer inputs, which are also parametrically defined by the parameter set PAR. Again, in this case, for example, as described above,

number

[0082] In some embodiments, one or more earlier layers of the neural network NN are designed to be equivariant not only to the state-action symmetry Θ, but also to translations. For example, the observable states include images of a physical environment, where one or more earlier layers of the neural network are equivariant to translations of the images in addition to state-action symmetry. For example, a group-equivalent neural network layer of a "group-equivalent convolutional network" is used. Note that in many applications, translations of the observable states generally do not induce state-action symmetry, since they do not result in the desired action permutation. Still, by including translations in the earlier layers of the neural network, they are used in those earlier layers for object recognition tasks, which are common, for example, in convolutional neural networks. Thus, in later layers, the translation symmetry is effectively removed by average pooling over the translations of the observable states.

[0083] The neural network NN is used to interact with the physical environment by repeatedly acquiring sensor data indicative of observable states OS of the physical environment, determining action probabilities APi based on the observable states OS, and providing actuators with actuator data that cause the actuators to realize actions in the physical environment based on the determined action probabilities APi, e.g., the realized actions are sampled according to the action probabilities, or the action with the maximum probability is selected, etc.

[0084] The neural network NN is trained by repeatedly obtaining interaction data indicative of sequences of observed environmental states and corresponding actions performed by the system, determining rewards for the interactions, determining action probabilities APi of a policy that selects the corresponding actions in observed states of the sequence of observed environmental states, and adjusting a parameter set to increase the expected reward based on the determined rewards and action probabilities. Various reinforcement learning techniques known per se are applied, for example policy gradient algorithms such as those disclosed in "Proximal Policy Optimization Algorithms". As is known, such optimization methods are heuristic and / or reach a local optimum. It is possible to use both on-policy methods, where the interaction data is obtained from the interactions according to the current parameter set of the policy, and off-policy methods, where this is not the case. In either case, while optimizing a standard neural network policy traditionally involves updating all filter weights, interestingly, using the technique presented herein, the coefficients Ci of the basis weight matrix are updated instead, resulting in faster and more efficient learning.

[0085] 5a-5c show non-limiting examples of transformations of observable states and final layer inputs, and action permutations of action probability sets. 11 , z 12 , z 21 , z 22 , APn in FIG. 4. An observable state 510, e.g., the observable state OS of FIG. 4, is shown, which includes a 2×2 input image including p, and a further sensor measurement x. Also shown is a vector 550 of action probabilities π1, π2, π3, π4, π5, e.g., the action probabilities AP1, ..., APn of FIG.

[0086] As an example, the physical environment in which the observable state 510 is obtained and the action 550 is performed is assumed to be equivariant with respect to horizontal and vertical mirroring. In this example, there is an identity mapping I, a horizontal mirroring H, a vertical mirroring V, and a horizontal plus vertical mirroring V.

number

number

number

number

number

[0087] For example, horizontal mirroring of the observable state, indicated by arrow 520, results in a transformed state 511, in which the image is mirrored horizontally and further sensor measurements are negated, e.g., x represents the angle in the vertical plane. Similarly, vertical mirroring of the observable state, indicated by arrow 521, results in a transformed state 521, e.g., in which the image is mirrored vertically but the sensor measurements remain the same. The observable state 510 may also be mirrored both horizontally and vertically, e.g., to result in a transformed state 513.

[0088] In this example, the transformation Θ of the observable state is assumed to result in an action permutation of the action permutation set of the neural network. For example, based on domain knowledge information, the action probability set (π1,π2,π3,π4,π5) 550 must be permuted to action probability (π2,π1,π3,π4,π5) 551 under horizontal mirror symmetry 560, in other words, the first action is assumed to be as desirable in the original observable state as the second action in the transformed observable state, and vice versa. The other three actions in this example are not affected by the horizontal symmetry. Similarly, under vertical symmetry 561 the action probability must be permuted to (π1,π2,π4,π3,π5) 552, and under horizontal and vertical symmetry 553 the action probability must be permuted to (π2,π1,π4,π3,π5) 553.

[0089] Using the techniques presented herein, such equivariance between transformations of the observable states 510 and corresponding permutations of the action probabilities 550 output by the neural network is achieved by computing the final layer inputs 530 from the observable states 510 in an equivariant manner, and then computing the action probabilities 550 from the final layer inputs 530 in an equivariant manner. To this end, the final layer inputs 530 include a number of feature vectors corresponding to each transformation Θ. The feature vector y corresponding to the transformations I, H, V, and HV is I , y H , y V , and y HV are shown in the figure. The transformation Θ in this example permutes the final layer inputs 530 according to the group action of Θ. For example,

number

number

[0090] Thus, as shown in this example, given transformations of observable states 510 and action permutations 550, both of which are manually specified based on domain knowledge information of the physical system under consideration, transformations of final layer inputs 530 can be automatically determined. Given transformations of final layer inputs and action permutations, as disclosed herein, a set of basis weight matrices is automatically determined, in which case the final layer of the neural network remains equivariant. By further computing the final layer inputs 530 in an equivariant manner, e.g., using additional similarly shaped layers, the symmetries of the physical environment are effectively incorporated into the neural network.

[0091] 6 illustrates a block diagram of a computer-implemented method 800 for interacting with a physical environment according to a policy. The policy determines a number of action probabilities for respective actions based on observable states of the physical environment. The policy includes a neural network that is parametrically defined by a set of parameters. The neural network determines the action probabilities by determining final layer inputs from the observable states and applying a final layer of the neural network to the final layer inputs. Method 800 corresponds to the operations of system 100 of FIG. 1. However, this is not a limitation in that method 800 may be implemented using another system, apparatus, or device.

[0092] The method 800 includes accessing 810 a parameter set of the policy through an operation called "accessing a policy."

[0093] The method 800 includes obtaining (820) basis weight matrix data representing a set of basis weight matrices for a final layer of the neural network, via an operation referred to as “obtaining basis weight matrix data,” where, for a set of multiple predefined transformations of the final layer inputs, each transformation results in a corresponding predefined action permutation of the basis weight matrix output for the final layer inputs.

[0094] The method 800 includes controlling (830) an interaction with the physical environment by an operation called "controlling the interaction." To control the interaction, the computing (830) may include: - acquiring (832) sensor data from one or more sensors indicative of observable conditions of the physical environment, through an operation referred to as "acquiring sensor data"; - determining action probabilities based on the observable states, comprising applying a final layer of the neural network by applying a linear combination of a set of basis weight matrices to the final layer inputs by an operation called "determining action probabilities" (834), where coefficients of the linear combination are included in a parameter set; - providing actuator data (836), which causes the actuator to realize an action in the physical environment based on the determined action probability, in an operation called "providing actuator data"; The above steps are repeated.

[0095] 7 shows a block diagram of a computer-implemented method 900 for configuring a system for interacting with a physical environment according to a policy. For example, the system may use the method 800 of FIG. 8. The method 900 includes: - acquiring (910) interaction data indicative of a sequence of observed environmental conditions and corresponding actions performed by the system, in an operation called "acquiring interaction data"; - determining a reward for the interaction (920) by an operation called "determine reward"; - determining (930) action probabilities of a policy that selects corresponding actions in observed states of a sequence of observed environmental states, comprising applying a final layer of a neural network by applying a linear combination of a set of basis weight matrices to the final layer inputs by an operation called "determining action probabilities", where coefficients of the linear combination are included in a parameter set; - adjusting (940) a set of parameters to increase the expected reward based on the determined reward and the action probabilities, in an operation called "adjusting parameters"; The method comprises optimizing a set of parameters of the policy to maximize the expected reward of interacting with the environment according to the policy by iteratively performing

[0096] In general, it will be appreciated that the operations of method 800 of FIG. 6 and method 900 of FIG. 7 may be performed in any suitable order, e.g., sequentially, simultaneously, or a combination thereof, subject to, where applicable, any particular order required by, e.g., input / output relationships.

[0097] The method is implemented in a computer as a computer-implemented method, either as dedicated hardware or as a combination of both. As further shown in Fig. 8, instructions for a computer, e.g. executable code, are stored in a computer readable medium 1000, e.g. in the form of a series of machine-readable physical marks 1010 and / or as a series of elements with different electrical, e.g. magnetic or optical properties or values. The executable code is stored in a transitory or non-transitory manner. Examples of computer readable media include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Fig. 11 shows an optical disk 1000.

[0098] Alternatively or in addition, the computer-readable medium 1000 includes transient or non-transient data 1010 representing a parameter set of a policy for interacting with a physical environment as described herein, the policy determining a plurality of action probabilities for respective actions based on an observable state of the physical environment, the policy including a neural network, the neural network determining the action probabilities by determining final layer inputs from the observable states and applying a final layer of the neural network to the final layer inputs, the final layer of the neural network being applied by applying a linear combination of a set of basis weight matrices to the final layer inputs, the coefficients of the linear combination being included in the parameter set.

[0099] Alternatively or in addition, computer-readable medium 1000 includes transient or non-transient data 1010 representing basis weight matrix data representing a basis weight matrix set for a policy for interacting with a physical environment as described herein, the policy determining a plurality of action probabilities for respective actions based on an observable state of the physical environment, the policy including a neural network, the neural network determining the action probabilities by determining final layer inputs from the observable states and applying a final layer of the neural network to the final layer inputs, the final layer of the neural network being applied by applying a linear combination of the basis weight matrix set to the final layer inputs.

[0100] No examples, embodiments, or optional features, whether or not indicated as non-limiting, are to be understood as limiting the invention as recited in the claims.

[0101] It should be noted that the above-described embodiments illustrate rather than limit the invention, and that those skilled in the art can design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claims. Use of the verb "comprise" and its conjugations does not negate the presence of other elements or stages of what is recited in the claims. The reference of an element in the singular does not negate the presence of a plurality of such elements. The reference of "at least one of", when preceding a list or group of elements, denotes the selection of all or any subset of the elements from the list or group. For example, the reference of "at least one of A, B, and C" shall be understood to include A only, B only, C only, both A and B, both A and C, both B and C, or all of A, B, and C. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage. [Note] [Appendix 1] 1. A computer-implemented method for interacting with a physical environment according to a policy, the policy determining a plurality of action probabilities for respective actions based on an observable state of the physical environment, the policy including a neural network parametrically defined by a set of parameters, the neural network determining the action probabilities by determining final layer inputs from observable states and applying a final layer of the neural network to the final layer inputs, the computer-implemented method comprising: accessing the parameter set of the policy; obtaining basis weight matrix data representing a set of basis weight matrices for the final layer of the neural network, where for a set of multiple predefined transformations of the final layer inputs, each transformation results in a corresponding predefined action permutation of basis weight matrix outputs for the final layer inputs; Controlling an interaction with the physical environment, said controlling comprising: acquiring sensor data from one or more sensors indicative of the observable state of the physical environment; determining the action probabilities based on the observable states, comprising applying the final layer of the neural network by applying a linear combination of the set of basis weight matrices to the final layer inputs, where coefficients of the linear combination are included in the parameter set; providing actuators with actuator data that cause the actuators to effect actions in the physical environment based on the determined action probabilities; and controlling the temperature by repeating the above steps. 23. A computer-implemented method comprising: [Appendix 2] the sensor data includes images of the physical environment; 2. The computer-implemented method of claim 1. [Appendix 3] The feature transformation corresponds to a rotation and / or the feature transformation corresponds to a reflection; 3. The computer-implemented method of claim 2. [Appendix 4] the sensor data further comprises one or more additional sensor measurements; 4. The computer-implemented method of claim 2 or 3. [Appendix 5] applying the final layer further comprises applying a further linear combination of the set of basis weight matrices to the final layer inputs, the coefficients of the further linear combination being included in the set of parameters. 5. A computer-implemented method according to any one of claims 1 to 4. [Appendix 6] the step of applying the final layer further comprises applying a further linear combination of a further set of basis weight matrices to the final layer inputs, for a further set of predefined transformations of the final layer inputs, each transformation resulting in a corresponding further predefined action permutation of a further basis weight matrix output to the final layer inputs; 6. A computer-implemented method according to any one of claims 1 to 5. [Appendix 7] a layer input of a layer of the neural network includes a plurality of feature vectors corresponding to respective transformations of the observable states, and features of the layer input are determined by average pooling over the feature vectors corresponding to translational transformations of the observable states; 7. A computer-implemented method according to any one of claims 1 to 6. [Appendix 8] acquiring interaction data indicative of sequences of observed environmental conditions and corresponding actions taken by the system; determining a reward for the interaction; determining action probabilities for a policy that selects the corresponding action in an observed state of the sequence of observed environmental states, the policy comprising applying a final layer of a neural network by applying a linear combination of a set of basis weight matrices to final layer inputs, the coefficients of the linear combination being included in a parameter set; adjusting the set of parameters to increase expected reward based on the determined reward and the action probabilities; optimizing the set of parameters of the policy to maximize the expected reward of interacting with the physical environment in accordance with the policy by iteratively performing [Appendix 9] obtaining the set of basis weight matrices by determining the set of basis weight matrices from the plurality of predefined transformations and corresponding predefined action permutations. 9. The computer-implemented method of claim 8. [Appendix 10] determining a basis weight matrix by obtaining an initial weight matrix, applying a transformation and inverse projection of a corresponding action permutation to the initial weight matrix, and adding together the transformed and permuted initial weight matrices; 10. The computer-implemented method of claim 9. [Appendix 11] The method further comprises the step of orthogonalizing the determined set of basis weight matrices. 11. The computer-implemented method of claim 10. [Appendix 12] 1. A computer-controlled system for interacting with a physical environment according to a policy, the policy determining a plurality of action probabilities for respective actions based on an observable state of the physical environment, the policy including a neural network parametrically defined by a set of parameters, the neural network determining the action probabilities by determining final layer inputs from observable states and applying a final layer of the neural network to the final layer inputs, the system comprising: a data interface for accessing the parameter set of the policy; a sensor interface for acquiring sensor data indicative of the observable state of the physical environment from one or more sensors; an actuator interface for providing actuator data to the actuator that causes the actuator to effect an action in the physical environment; obtaining basis weight matrix data representing a set of basis weight matrices for the final layer of the neural network, for a set of a plurality of predefined transformations of the final layer inputs, each transformation resulting in a corresponding predefined action permutation of basis weight matrix outputs for the final layer inputs; and controlling an interaction with the physical environment, the controlling comprising: acquiring sensor data from the one or more sensors via the sensor interface that is indicative of the observable state of the physical environment; determining the action probabilities based on the observable states, comprising applying the final layer of the neural network by applying a linear combination of the set of basis weight matrices to the final layer inputs, where coefficients of the linear combination are included in the parameter set; providing actuator data via the actuator interface to the actuator causing the actuator to effect an action in the physical environment based on the determined action probability; a processor subsystem for controlling the operation of the control unit by repeatedly performing the above steps; A system comprising: [Appendix 13] 8. A training system for configuring a computer-controlled system which interacts with a physical environment according to a policy using the computer-implemented method of any one of claims 1 to 7, said training system comprising: a data interface for accessing a parameter set of the policy and basis weight matrix data representing a set of basis weight matrices for a final layer of the neural network, the data interface comprising: a set of predefined transformations of final layer inputs, each transformation resulting in a corresponding predefined action permutation of basis weight matrix outputs for the final layer inputs; a processor subsystem, the processor subsystem comprising: acquiring interaction data indicative of a sequence of observed environmental conditions and corresponding actions performed by the computer-controlled system; Determining a reward for the interaction; and determining action probabilities of the policy for selecting the corresponding action in observed states of the sequence of observed environmental states, comprising applying the final layer of the neural network by applying a linear combination of the set of basis weight matrices to the final layer inputs, where coefficients of the linear combination are included in the parameter set; adjusting the set of parameters to increase an expected reward based on the determined reward and the action probabilities; optimizing the parameter set of the policy to maximize the expected reward of interacting with the physical environment according to the policy by iteratively performing [Appendix 14] instructions which, when executed by a processor system, cause the processor system to perform the computer-implemented method of claim 1; A parameter set for a policy for interacting with a physical environment as recited in claim 1, wherein the policy determines a plurality of action probabilities for respective actions based on an observable state of the physical environment, the policy including a neural network, the neural network determining the action probabilities by determining final layer inputs from observable states and applying a final layer of the neural network to the final layer inputs, the final layer of the neural network being applied by applying a linear combination of a set of basis weight matrices to the final layer inputs, coefficients of the linear combination being included in the parameter set; basis weight matrix data representing a set of basis weight matrices for a policy for interacting with a physical environment as recited in Supplementary Note 1, the policy determining a plurality of action probabilities for respective actions based on an observable state of the physical environment, the policy including a neural network, the neural network determining the action probabilities by determining a final layer input from an observable state and applying a final layer of the neural network to the final layer input, the final layer of the neural network being applied by applying a linear combination of the set of basis weight matrices to the final layer input; A computer-readable medium containing transitory or non-transitory data representing one or more of: [Appendix 15] A computer-readable medium comprising transitory or non-transitory data representing instructions that, when executed by a processor system, cause the processor system to perform the computer-implemented method of claim 8.

Claims

1. 1. A computer-implemented method for interacting with a physical environment according to a policy, the policy determining a plurality of action probabilities for respective actions based on an observable state of the physical environment, the policy including a neural network parametrically defined by a set of parameters, the neural network determining the action probabilities by determining final layer inputs from observable states and applying a final layer of the neural network to the final layer inputs, the computer-implemented method comprising: accessing the parameter set of the policy; obtaining basis weight matrix data representing a set of basis weight matrices for the final layer of the neural network, where for a set of multiple predefined transformations of the final layer inputs, each transformation results in a corresponding predefined action permutation of basis weight matrix outputs for the final layer inputs; Controlling an interaction with the physical environment, said controlling comprising: acquiring sensor data from one or more sensors indicative of the observable state of the physical environment; determining the action probabilities based on the observable states, comprising applying the final layer of the neural network by applying a linear combination of the set of basis weight matrices to the final layer inputs, where coefficients of the linear combination are included in the parameter set; providing actuators with actuator data that cause the actuators to effect actions in the physical environment based on the determined action probabilities; and controlling the temperature by repeating the above steps.

23. A computer-implemented method comprising:

2. the sensor data includes images of the physical environment; 2. The computer-implemented method of claim 1.

3. The feature transformation corresponds to a rotation and / or the feature transformation corresponds to a reflection; 3. The computer-implemented method of claim 2.

4. the sensor data further comprises one or more additional sensor measurements; 4. A computer-implemented method according to claim 2 or claim 3.

5. applying the final layer further comprises applying a further linear combination of the set of basis weight matrices to the final layer inputs, the coefficients of the further linear combination being included in the set of parameters. A computer-implemented method according to any one of claims 1 to 4.

6. the step of applying the final layer further comprises applying a further linear combination of a further set of basis weight matrices to the final layer inputs, for a further set of predefined transformations of the final layer inputs, each transformation resulting in a corresponding further predefined action permutation of a further basis weight matrix output to the final layer inputs; A computer-implemented method according to any one of claims 1 to 5.

7. a layer input of a layer of the neural network includes a plurality of feature vectors corresponding to respective transformations of the observable states, and features of the layer input are determined by average pooling over the feature vectors corresponding to translational transformations of the observable states; A computer-implemented method according to any one of claims 1 to 6.

8. acquiring interaction data indicative of sequences of observed environmental conditions and corresponding actions taken by the system; determining a reward for the interaction; determining action probabilities for a policy that selects the corresponding action in an observed state of the sequence of observed environmental states, the policy comprising applying a final layer of a neural network by applying a linear combination of a set of basis weight matrices to final layer inputs, the coefficients of the linear combination being included in a parameter set; adjusting the set of parameters to increase expected reward based on the determined reward and the action probabilities; 8. A computer-implemented method of configuring the system to interact with the physical environment in accordance with the policy using the computer-implemented method of any one of claims 1 to 7, comprising optimizing the set of parameters of the policy to maximize the expected reward of interacting with the physical environment in accordance with the policy by iteratively performing:

9. obtaining the set of basis weight matrices by determining the set of basis weight matrices from the plurality of predefined transformations and corresponding predefined action permutations.

9. The computer-implemented method of claim 8.

10. determining a basis weight matrix by obtaining an initial weight matrix, applying a transformation and inverse projection of a corresponding action permutation to the initial weight matrix, and adding together the transformed and permuted initial weight matrices; 10. The computer-implemented method of claim 9.

11. The method further comprises the step of orthogonalizing the determined set of basis weight matrices.

11. The computer-implemented method of claim 10.

12. 1. A computer-controlled system for interacting with a physical environment according to a policy, the policy determining a plurality of action probabilities for respective actions based on an observable state of the physical environment, the policy including a neural network parametrically defined by a set of parameters, the neural network determining the action probabilities by determining final layer inputs from observable states and applying a final layer of the neural network to the final layer inputs, the system comprising: a data interface for accessing the parameter set of the policy; a sensor interface for acquiring sensor data indicative of the observable state of the physical environment from one or more sensors; an actuator interface for providing actuator data to the actuator that causes the actuator to effect an action in the physical environment; obtaining basis weight matrix data representing a set of basis weight matrices for the final layer of the neural network, for a set of a plurality of predefined transformations of the final layer inputs, each transformation resulting in a corresponding predefined action permutation of basis weight matrix outputs for the final layer inputs; and controlling an interaction with the physical environment, the controlling comprising: acquiring sensor data from the one or more sensors via the sensor interface that is indicative of the observable state of the physical environment; determining the action probabilities based on the observable states, comprising applying the final layer of the neural network by applying a linear combination of the set of basis weight matrices to the final layer inputs, where coefficients of the linear combination are included in the parameter set; providing actuator data via the actuator interface to the actuator causing the actuator to effect an action in the physical environment based on the determined action probability; a processor subsystem for controlling the operation of the control unit by repeatedly performing the above steps; A system comprising:

13. A training system for configuring a computer-controlled system that interacts with a physical environment according to a policy using the computer-implemented method of any one of claims 1 to 7, said training system comprising: a data interface for accessing a parameter set of the policy and basis weight matrix data representing a set of basis weight matrices for a final layer of the neural network, the data interface comprising: a set of predefined transformations of final layer inputs, each transformation resulting in a corresponding predefined action permutation of basis weight matrix outputs for the final layer inputs; a processor subsystem, the processor subsystem comprising: acquiring interaction data indicative of a sequence of observed environmental conditions and corresponding actions performed by the computer-controlled system; Determining a reward for the interaction; and determining action probabilities of the policy for selecting the corresponding action in observed states of the sequence of observed environmental states, comprising applying the final layer of the neural network by applying a linear combination of the set of basis weight matrices to the final layer inputs, where coefficients of the linear combination are included in the parameter set; adjusting the set of parameters to increase an expected reward based on the determined reward and the action probabilities; optimizing the parameter set of the policy to maximize the expected reward of interacting with the physical environment according to the policy by iteratively performing

14. 13. A computer readable medium comprising transitory or non-transitory data representing instructions that, when executed by a processor system, cause the processor system to perform the computer-implemented method of claim 1.

15. The temporary or non-temporary data is:

2. A parameter set for a policy for interacting with a physical environment according to claim 1, wherein the policy determines a plurality of action probabilities for respective actions based on an observable state of the physical environment, the policy including a neural network, the neural network determining the action probabilities by determining a final layer input from an observable state and applying a final layer of the neural network to the final layer input, the final layer of the neural network being applied by applying a linear combination of a set of basis weight matrices to the final layer input, coefficients of the linear combination being included in the parameter set; 2. Basis weight matrix data representing a basis weight matrix set for a policy for interacting with a physical environment according to claim 1, wherein the policy determines a plurality of action probabilities for respective actions based on an observable state of the physical environment, the policy including a neural network, the neural network determines the action probabilities by determining a final layer input from an observable state and applying a final layer of the neural network to the final layer input, the final layer of the neural network being applied by applying a linear combination of the basis weight matrix set to the final layer input; further represent one or more of the following:

15. The computer-readable medium of claim 14.

16. 10. A computer readable medium comprising transitory or non-transitory data representing instructions that, when executed by a processor system, cause the processor system to perform the computer-implemented method of claim 8.

Images