Method for training a neural network policy and method for controlling motion of a robot

Abstract

A computer-implemented method for training neural network policy 200. The method includes generating plurality of episodic data slots 210 based on a trajectory sequence from a training dataset. The me

Description

TECHNICAL FIELD

[0001] Various embodiments relate to a method for training a neural network policy and a method for controlling motion of a robot. BACKGROUND

[0002] Reinforcement learning (RL) is increasing being applied to low level motor control in robotics applications. The core idea is to use a dataset of robot operating data to train a neural network policy that can optimally apply motor controls to the robot. While this approach has several advantages over traditional motion planning, such as a reduced need for precision in motors and sensing hardware, allowing it to achieve the desired outcomes on lower cost robot hardware, and requiring fewer assumptions about the operating environment, it is limited by inconsistencies in robot hardware dynamics, which may be caused by factors such as wear and tear, and variations in the robot’s interaction with its environment. This is especially so, for long time horizons. In the context of traditional motion planning for long time horizons, an online system identification may be “engineered” to recall older information (e.g. by setting a particular type of system model, or by adding heuristics). However, RL-based solutions cannot be "engineered" in this way.

[0003] To solve this problem in the RL approach, synthetic training data may be generated with randomized dynamics and the reinforcement learning policy may learn to act conservatively for a short period of time to safely observe some data about the robot’s dynamic response, implicitly recognize the actual dynamics of the robot, and recall this information at subsequent time steps to correctly control the robot. However, this approach relies too much on the recurrent neural network implicitly learning the required behavior. Adaptive reinforcement learning algorithms that are explicitly conditioned on the robot’s immediate prior experience may be applied to reduce this dependency, and specific episodes or subsets of the observed data may be explicitly provided to the neural network. The challenge with this approach is that there is no optimal method to select the episodic data that should be provided as input to the neural network policy. SUMMARY

[0004] According to various embodiments, there is provided a computer-implemented method for training a neural network policy. The method includes generating a plurality of episodic data slots based on a trajectory sequence from a training dataset. The method further includes, for each episodic data slot of the plurality of episodic data slots: generating a selection output based on the episodic data slot, using a slot selection network, determining a noise magnitude based on the selection output, generating noise based on the determined noise magnitude, and adding the generated noise to the episodic data slot, resulting in an updated episodic data slot. The method further includes generating a control output based on the updated episodic data slots, using a controller network.

[0005] According to various embodiments, there is provided a computer-implemented method for controlling motion of a robot. The method includes inputting current episodic data to the slot selection network of a neural network policy trained according to the method of any one of the preceding claims; replacing one of a plurality of stored episodic data slots with the current episodic data, based on a selection output from the slot selection network, to result in an updated plurality of episodic data slots; and generating motion control output based on the updated plurality of episodic data slots, using the controller network.

[0006] According to various embodiments, a first computer program may be provided. The computer program may include the neural network policy trained according to the abovementioned computer-implemented method for training a neural network policy.

[0007] According to various embodiments, a second computer program may be provided. The computer program may include instructions when executed by a processor, performs the abovementioned computer-implemented method for controlling motion of a robot.

[0008] According to various embodiments, a motion controller device may be provided. The motion controller device may be used with a robot. The motion controller device may include a memory and a processor. The memory may store a computer program that includes instructions, when executed by the processor, performs the abovementioned computer-implemented method for controlling motion of a robot.

[0009] Additional features for advantageous embodiments are provided in the dependent claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In the drawings, like reference characters generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments are described with reference to the following drawings, in which:

[0011] FIG. 1 shows a diagram of a robot according to various embodiments.

[0012] FIG. 2 shows a schematic diagram illustrating the neural network policy according to various embodiments.

[0013] FIG. 3 shows a flow diagram of a computer-implemented method for training a neural network policy, according to various embodiments.

[0014] FIG. 4 shows a flow diagram of a computer-implemented method for controlling motion of a robot according to various embodiments.

[0015] FIG. 5 shows a conceptual diagram of a motion controller device according to various embodiments. DESCRIPTION

[0016] Embodiments described below in context of the devices are analogously valid for the respective methods, and vice versa. Furthermore, it will be understood that the embodiments described below may be combined, for example, a part of one embodiment may be combined with a part of another embodiment.

[0017] It will be understood that any property described herein for a specific device may also hold for any device described herein. It will be understood that any property described herein for a specific method may also hold for any method described herein. Furthermore, it will be understood that for any device or method described herein, not necessarily all the components or steps described must be enclosed in the device or method, but only some (but not all) components or steps may be enclosed.

[0018] The term “coupled” (or “connected”) herein may be understood as electrically coupled or as mechanically coupled, for example attached or fixed, or just in contact without any fixation, and it will be understood that both direct coupling or indirect coupling (in other words: coupling without direct contact) may be provided.

[0019] In this context, the motion controller device as described in this description may include a memory which is for example used in the processing carried out in the device. A memory used in the embodiments may be a volatile memory, for example a DRAM (Dynamic Random Access Memory) or a non-volatile memory, for example a PROM (Programmable Read Only Memory), an EPROM (Erasable PROM), EEPROM (Electrically Erasable PROM), or a flash memory, e g., a floating gate memory, a charge trapping memory, an MRAM (Magnetoresistive Random Access Memory) or a PCRAM (Phase Change Random Access Memory).

[0020] In order that the invention may be readily understood and put into practical effect, various embodiments will now be described by way of examples and not limitations, and with reference to the figures.

[0021] FIG. 1 shows a diagram of a robot 110 according to various embodiments. The robot 110 may include a motion controller 120. The robot 110 may further include at least one motor 112, an accelerometer 114 and an odometer 116. The odometer 116 may be, for example, a visual odometer or a LiDAR odometer. The robot 110 may be configured to pick up various loads while it travels on different types of surfaces. The motion controller 120 may have access to various vehicle data at each instant in time. The various vehicle data may include, for example, torque applied to each motor 112, the velocity of each motor 112, the linear acceleration and angular velocity from the accelerometer 114 and an external exteroceptive odometry signal from the odometer 116. Collectively, the vehicle data provides a time series of input data to the robot 110. The time series of input data may be divided into regular time intervals. Each time interval of input data is referred herein as “episodic data”. For example, in a robot application, each time interval may be around 5 seconds, and the episodic data may include input data received within the 5 seconds. The motion controller 120 may include a neural network policy 200 that may be trained to identify events, for example, if the robot 110 skidded while going around a corner. In training the neural network policy, episodic data relating to such events may be collected and input to the neural network policy 200.

[0022] FIG. 2 shows a schematic diagram illustrating the neural network policy 200 according to various embodiments. The motion controller 120 may be configured to store a fixed number of slots 210. The fixed number is denoted herein as S. Each slot 210 may store a respective amount of episodic data 212. The slots 210 may be referred herein as episodic data slots 210. Each episode of data to be stored may have the same number of time instants. The number of time instants is denoted herein as N. Each episode of data may be referred herein as episodic data 212. The neural network policy 200 may include an episodic data encoding network (EDEN) 202 that converts each piece of episodic data 212 into an embedding vector, also referred herein as episodic data embedding vector 214. Each episodic data embedding vector 214 may be placed in an episodic data slot 210. The episodic data slots 210 may be provided to a slot selection network 204, that is configured to determine whether the current episodic data 212 should replace earlier episodic data in one of the episodic data slots 210. The slot selection network may be trained using training data. The full processing pipeline is referred herein as the neural network policy.

[0023] In other words, each episodic data embedding vector 214 may include N time instants of input data, and each episodic data slot 210 may store one episodic data embedding vector 214. The motion controller 120 may store S episodic data slots 210, and therefore store S x N time instants of input data.

[0024] The neural network policy 200 may further include a controller network 206. The controller network 206 may be configured to recollect a small number of episodes from previous operations, and combine these together with the current episodic data, to determine appropriate control actions to take in the future. Concurrently, the slot selection network 204 may decide whether the current episodic data should be stored in episodic data slots for future use, and if so, which episodic data slot it should overwrite.

[0025] An example of a suitable neural network for the controller network 206 is disclosed in “Watch, Try, Learn: Meta-Learning from Demonstrations and Reward” by Zhou et. al., which is accessible at https: / / arxiv.org / abs / 1906.03352.

[0026] The neural network policy 200 may simulate “forgetting” instead of evicting stored data. During the training process of the neural network policy 200, the agent may be able to use an unbounded amount of episodic data slots 210, with instances of the episodic data 212 corrupted with noise, such as additive noise or multiplicative noise. In this way, no hard decision is made during training to discard any data, and no “soft” blending of instances of episodic data 212 is required either.

[0027] During a training process, all episodic data 212 may be retained. The output of the slot selection network 204 may be used to determine an appropriate noise scale for noise that is added to old episodic data. This way, the neural network policy 200 may learn which episodic data can be “forgotten” in a differentiable manner, without requiring any sampling or soft blending of episodic data during training.

[0028] In the training process, one trajectory sequence may be selected from the training dataset at a time. At each time instant, the neural network policy 200 may be executed, such that the last A time instants of input data, also referred herein as current episodic data 220, may be processed into a current episodic data embedding vector 214’ using the EDEN 202. The current episodic data embedding vector 214’ may be stored in a buffer, and fed as input to the slot selection network 204. The slot selection network 204 may be configured to select one of the S episodic data slots 210 for storing the new input data, i.e. the current episodic data 220. The output of the slot selection network 204 may be referred herein as the selection output. The selection output may be converted into a set of noise magnitudes, by a noise scale calculation module 222. New noise 226 may be generated based on the computed noise magnitudes. The new noise 226 may be added to the episodic data embedding vectors 214 stored in the episodic data slots 210. Following the addition of the new noise 226, the controller network output 206 may be backpropagated to continue the training of the neural network policy training 200.

[0029] In another embodiment, the noise magnitudes 226 may be reduced if the episodic data slot 210 was corrupted with noise at an earlier time instant 224, so that the noise magnitude is prevented from being arbitrarily large. For example, the magnitude of the noise at the earlier time instant 224 may be subtracted from the noise magnitudes 226, to determine the amount of new noise 226 to be added.

[0030] An example of the computations performed by the noise scale calculation module 222 may be the Gumbel-Softmax reparameterization, as disclosed in Categorical Reparameterization with Gumbel-Softmax by Jang et. al., which can be accessed at https: / / arxiv.org / abs / 1611.01144.

[0031] An example of a suitable training dataset may be the RobotNet, which may be accessed at https: / / github.com / SudeepDasari / RoboNet. While the RobotNet data was compiled for robot arms, the principle for training the neural network policy is similar, and as such the training dataset may also be applied to mobile robots. For this dataset, a suitable “reward” signal might be the negated root mean square value of the joint accelerations for a particular sequence of data.

[0032] After the neural network policy 200 is trained, it may be deployed in a robot 110. During an inference process, the following steps may be executed:

[0033] First, the episodic data slots 210 may be initialized to zero. This may be performed when the robot 110 is commissioned at a factory, or each time when the robot 110 is switched on.

[0034] Next, at every time instant, the last N time instants of input data, also referred herein as current episodic data 220, may be processed into a current episodic data embedding vector 214’ using the EDEN 202. The current episodic data embedding vector 214 may be concatenated with the S pieces of stored episodic data slots 210, resulting in S+l episodic data embedding vectors 214. The resulting S+l episodic data embedding vectors 214 may be provided as input to the slot selection network 204. The slot selection network 204 may be configured to select one of the S episodic data slots 210 for storing the new input data, i.e. the current episodic data 220. The output of the slot selection network 204 may be referred herein as the selection output. The slot selection network 204 may assess whether the current episodic data 220 should replace an existing stored piece of episodic data in one of the episodic data slots 210. The slot selection network 204 may determine S probability values, and find the highest probability value among the K probability values. If the highest probability value exceeds a predetermined threshold, the neural network policy 200 may replace the data of the corresponding episodic data slot 210 with the current episodic data 220.

[0035] In another embodiment, the neural network policy 200 may not inspect the existing data in each episodic data slot 210. Instead, the neural network policy 200 may inspect only the current episodic data 220. In other words, the current episodic data embedding vector 214’ may not be concatenated with the S pieces of stored episodic data slots 210. The slot selection network 204 may generate its selection output based on the current episodic data embedding vector 214’. The slot selection network 204 may be trained to store a particular type of data in each respective episodic data slot 210.

[0036] According to various embodiments, the EDEN 202 may be a recurrent neural network (RNN), such as the long short-term memory (LSTM) network. In an example, the EDEN 202 may have 20 inputs, 256 hidden units, 2 layers, resulting in an encoding vector of length 256.

[0037] According to various embodiments, the slot selection network 204 may include one of a graph neural network, a convolutional neural network (CNN), an RNN, or a transformer network. An example of the graph neural network may include the following layers: (i) GraphConv(in_channels=256, out_channels=256); (ii) Relu; (iii) Dropout(p=0.2); (iv) GraphConv(in_channels=256, out_channels=256); (v) Relu; (vi) Linear(256, 1); and (vii) Sigmoid.

[0038] FIG. 3 shows a flow diagram of a computer-implemented method 300 for training a neural network policy 200, according to various embodiments. The method 300 may include processes 302, 304, 306, 308, 310 and 312. The process 302 may include generating a plurality of episodic data slots 210 based on a trajectory sequence from a training dataset. The process 304 may include, for each episodic data slot of the plurality of episodic data slots: generating a selection output based on the episodic data slot 210, using a slot selection network 204. The process 306 may include determining a noise magnitude based on the selection output, for each episodic data slot 210 of the plurality of episodic data slots 210. The process 308 may include generating noise based on the determined noise magnitude, for each episodic data slot 210 of the plurality of episodic data slots 210. The process 310 may include adding the generated noise 226 to the episodic data slot 210, resulting in an updated episodic data slot. The process 312 may include generating a motion control output based on the updated episodic data slots, using a controller network 206. The resulting neural network policy 200 may be capable of recalling old, important information.

[0039] While the above embodiments described relate to the use of the neural network policy 200 for controlling motion of a robot, the neural network policy 200 may also be used for controlling other types of long-running processes.

[0040] According to an embodiment which may be combined with the above-described embodiment or with any below described further embodiment, generating noise based on the determined noise magnitude may include subtracting magnitude of a noise at an earlier time instant, from the determined noise magnitude, to result in a reduced noise magnitude, and generating the noise based on the reduce noise magnitude. This may prevent the noise level from reaching an overly large magnitude, as noise levels accumulate over the trajectory sequence.

[0041] According to an embodiment which may be combined with the above-described embodiment or with any below described further embodiment, the method 300 may further include

[0042] controlling motion of a robot based on the motion control output, and measuring quality of behaviour of the robot, resulting in an effectiveness score of the option, and backpropagating the effectiveness score. Consequently, the neural network policy may be trained to generate motion control outputs that maximise the quality of the robot behaviour.

[0043] According to an embodiment which may be combined with the above-described embodiment or with any below described further embodiment, wherein generating the plurality of episodic data slots 210 comprises generating each episodic data slot 210 comprises encoding N number of time instants of training data into an episodic data embedding vector 214, using an episodic data encoding network 202. The episodic data encoding network 202 may include a long short-term memory network (LSTM).

[0044] According to an embodiment which may be combined with the above-described embodiment or with any below described further embodiment, the slot selection network 204 may include at least one of a fully connected graph neural network and a transformer network.

[0045] According to an embodiment which may be combined with the above-described embodiment or with any below described further embodiment, the training data may include motion control commands provided to a robot 110.

[0046] FIG. 4 shows a flow diagram of a computer-implemented method 400 for controlling motion of a robot 110 according to various embodiments. The method 400 may include processes 402, 404 and 406. The process 402 may include inputting current episodic data 220 to the slot selection network 204 of a neural network policy 200 trained according to the method 300. The process 404 may include replacing one of a plurality of stored episodic data slots 210 with the current episodic data 220, based on a selection output from the slot selection network 204, to result in an updated plurality of episodic data slots. The process 406 may include generating motion control output based on the updated plurality of episodic data slots, using the controller network 206.

[0047] According to an embodiment which may be combined with the above-described embodiment or with any below described further embodiment, inputting the current episodic data 220 to the slot selection network 204 may include encoding the current episodic data 220 to result in a current episodic data embedding vector 214’, and further comprises concatenating the current episodic data embedding vector 214’ with episodic data embedding vectors 214 from the plurality of stored episodic data slots 210. As a result, the slot selection network 204 considers the contents of all of the episodic data slots 210, which may improve the accuracy of the selection output.

[0048] According to an embodiment which may be combined with the above-described embodiment or with any below described further embodiment, the slot selection network 204 may be configured to determine for each episodic data embedding vectors 214 of the respective stored episodic data slots 210, a respective probability value of the current episodic data embedding vector 214’ being a replacement of the episodic data. The probability value may indicate the usefulness of the episodic data. The probability value is useful for debugging the accuracy of the slot selection network, so that perturbative analysis of the decisions of the slot selection network may be performed.

[0049] According to an embodiment which may be combined with the above-described embodiment or with any below described further embodiment, replacing one of the plurality of stored episodic data slots 210 with the current episodic data embedding vector 214’ based on a selection output from the slot selection network 204 may include identifying a highest probability value amongst the probability values determined for the plurality of stored episodic data slots 210, comparing the highest probability value to a predetermined threshold, and replacing the stored episodic data slot 210 corresponding to the highest probability value based on the highest probability value exceeding the predetermined threshold. This prevents the episodic data slot 210 being replaced with the current episodic data in the scenario that the current episodic data is irrelevant, or redundant.

[0050] FIG. 5 shows a conceptual diagram of a motion controller device 500 according to various embodiments. The motion controller device 500 may be used with a robot 110, to direct the movements of the robot 110. The motion controller device 500 may include a memory 502 and a processor 504. The memory 502 may store a computer program that includes instructions, when executed by the processor 504, performs the computer-implemented method 400 for controlling motion of a robot 110. The memory 502 and the processor 504 may be coupled to one another, for example electrically and / or communicatively, by coupling line 550.

[0051] Various aspects of the method 400 described may also apply to the use of any of the abovedescribed motion controller device 500.

[0052] While embodiments of the invention have been particularly shown and described with reference to specific embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. The scope of the invention is thus indicated by the appended claims and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced. It will be appreciated that common numerals, used in the relevant drawings, refer to components that serve a similar or the same purpose.

[0053] It will be appreciated to a person skilled in the art that the terminology used herein is for the purpose of describing various embodiments only and is not intended to be limiting of the present invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0054] It is understood that the specific order or hierarchy of blocks in the processes / flowcharts disclosed is an illustration of exemplary approaches. Based upon design preferences, it is understood that the specific order or hierarchy of blocks in the processes / flowcharts may be rearranged. Further, some blocks may be combined or omitted. The accompanying method claims present elements of the various blocks in a sample order, and are not meant to be limited to the specific order or hierarchy presented.

[0055] The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein, but is to be accorded the full scope consistent with the language claims, wherein reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more.” The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects. Unless specifically stated otherwise, the term “some” refers to one or more. Combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof’ include any combination of A, B, and / or C, and may include multiples of A, multiples of B, or multiples of C. Specifically, combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof’ may be A only, B only, C only, A and B, A and C, B and C, or A and B and C, where any such combinations may contain one or more member or members of A, B, or C. All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims.

Claims

1. A computer-implemented method (300) for training a neural network policy, the method comprising:generating a plurality of episodic data slots based on a trajectory sequence from a training dataset (302);for each episodic data slot of the plurality of episodic data slots:generating a selection output based on the episodic data slot, using a slot selection network (304),determining a noise magnitude based on the selection output (306),generating noise based on the determined noise magnitude (308), andadding the generated noise to the episodic data slot, resulting in an updatedepisodic data slot (310); andgenerating a control output based on the updated episodic data slots, using a controller network (312).

2. The method (300) of claim 1, wherein generating noise based on the determined noise magnitude (308) comprisessubtracting magnitude of a noise at an earlier time instant, from the determined noise magnitude, to result in a reduced noise magnitude; andgenerating the noise based on the reduce noise magnitude.

3. The method (300) of any preceding claim, further comprising:controlling motion of a robot based on the control output;measuring quality of behaviour of the robot, resulting in an effectiveness score of the option; andbackpropagating the effectiveness score.

4. The method (300) according to any preceding claim, wherein generating the plurality of episodic data slots (304) comprises encoding N number of time instants of training data into an episodic data embedding vector, using an episodic data encoding network.

5. The method (300) according to claim 4, wherein the episodic data encoding network comprises a long short-term memory network.

6. The method (300) according to any preceding claim, wherein the slot selection network comprises at least one of a fully connected graph neural network and a transformer network.

7. The method (300) according to any preceding claim, wherein the training data comprises motion control commands provided to a robot.

8. A computer-implemented method (400) for controlling motion of a robot, the method (400) comprising:inputting current episodic data to the slot selection network (402) of a neural network policy trained according to the method (300) of any one of the preceding claims;replacing one of a plurality of stored episodic data slots with the current episodic data, based on a selection output from the slot selection network, to result in an updated plurality of episodic data slots (404); andgenerating motion control output based on the updated plurality of episodic data slots, using the controller network (406).

9. The method (400) according to claim 8, wherein inputting the current episodic data to the slot selection network (402) comprises encoding the current episodic data to result in a current episodic data embedding vector, and further comprises concatenating the current episodic data embedding vector with episodic data embedding vectors from the plurality of stored episodic data slots.

10. The method (400) according to any preceding claim, wherein the slot selection network is configured to determine for each episodic data embedding vectors of the respective stored episodic data slots, a respective probability value of the current episodic data embedding vector being a replacement of the episodic data.

11. The method (400) according to claim 10, wherein replacing one of the plurality of stored episodic data slots with the current episodic data embedding vector based on a selection output from the slot selection network comprisesidentifying a highest probability value amongst the probability values determined for the plurality of stored episodic data slots, comparing the highest probability value to a predetermined threshold, and replacing the stored episodic data slot corresponding to the highest probability value based on the highest probability value exceeding the predetermined threshold.

12. A computer program comprising a neural network policy trained according to any one of claims 1 to 7.

13. A computer program comprising instructions when executed by a processor, performs the method of any one of claims 8 to 11.

14. A motion controller (500) for use with a robot, the motion controller (500) comprising:a memory (502) storing the computer program of claim 13; anda processor (504) configured to run the instructions of the computer program.16

Images