METHOD FOR TRAINING A NEURAL NETWORK STRATEGY AND METHOD FOR CONTROLLING THE MOVEMENT OF A ROBOT

Abstract

According to various embodiments, a computer-implemented method for training a neural network strategy is provided. The method involves generating a plurality of episodic data slots based on a trajectory sequence from a training dataset. The method further involves, for each episodic data slot of the plurality: generating a selection output based on the episodic data slot using a slot selection network; determining a noise size based on the selection output; generating noise based on the determined noise size; and adding the generated noise to the episodic data slot, resulting in an updated episodic data slot. The method further involves generating a motion control output based on the updated episodic data slots using a control device network.

Description

TECHNICAL FIELD

[0001] Various embodiments relate to a method for training a strategy of a neural network and a method for controlling the movement of a robot. BACKGROUND

[0002] Reinforcement learning (RL) is increasingly being applied to low-level motor control in robotics applications. The core idea is to use a set of robot operating data to train a neural network strategy that can optimally utilize motor controllers for the robot. While this approach has several advantages over traditional motion planning, such as reducing the need for precision in motors and sensing hardware, enabling the desired results to be achieved on lower-cost robot hardware, and requiring fewer assumptions about the operating environment, it is limited by inconsistencies in robot hardware dynamics, which can be caused by factors such as wear and tear and variations in the robot's interaction with its environment.This is especially true for long-term horizons. In the context of traditional long-term motion planning, online system identification can be "constructed" to retrieve past information (e.g., by specifying a specific system model type or by adding heuristics). However, RL-based solutions cannot be "constructed" in this way.

[0003] To solve this problem in the RL approach, synthetic training data with randomized dynamics can be generated, and the reinforcement learning strategy can learn to act cautiously for a short period of time to safely observe some data across the robot's dynamic range, implicitly detect the robot's actual dynamics, and recall this information at subsequent time steps to properly control the robot. However, this approach relies too heavily on the recurrent neural network implicitly learning the required behavior. Adaptive reinforcement learning algorithms, explicitly conditioned on the robot's immediate prior experience, can be applied to reduce this dependence, and specific episodes or subsets of observed data can be explicitly provided to the neural network.The challenge with this approach is that there is no optimal procedure for selecting the episodic data that should be provided as input to the neural network strategy. SUMMARY

[0004] According to various embodiments, a computer-implemented method for training a neural network strategy is provided. The method comprises generating a plurality of episodic data slots based on a trajectory sequence from a training data set. The method further comprises, for each episodic data slot of the plurality of episodic data slots: generating a selection output based on the episodic data slot using a slot selection network, determining a noise magnitude based on the selection output, generating noise based on the determined noise magnitude, and adding the generated noise to the episodic data slot, resulting in an updated episodic data slot. The method further comprises generating a control output based on the updated episodic data slots using a controller network.

[0005] According to various embodiments, a computer-implemented method for controlling the motion of a robot is provided. The method comprises: inputting current episodic data to the slot selection network of a neural network strategy trained according to the method of any preceding claim; replacing one of a plurality of stored slots of episodic data with the current episodic data based on a selection output from the slot selection network to result in an updated plurality of slots of episodic data; and generating a motion control output based on the updated plurality of slots of episodic data using the controller network.

[0006] According to various embodiments, a first computer program may be provided. The computer program may include a neural network strategy trained according to the above-mentioned computer-implemented method for training a neural network strategy.

[0007] According to various embodiments, a second computer program may be provided. The computer program may comprise instructions that, when executed by a processor, perform the above-mentioned computer-implemented method for controlling the movement of a robot.

[0008] According to various embodiments, a motion control device can be provided. The motion control device can be used with a robot. The motion control device can comprise a memory and a processor. The memory can store a computer program comprising instructions that, when executed by the processor, perform the above-mentioned computer-implemented method for controlling the movement of a robot.

[0009] Additional features for advantageous embodiments are provided in the dependent claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In the drawings, like reference characters generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, but rather the emphasis is generally placed on illustrating the principles of the invention. In the following description, various embodiments are described with reference to the following drawings. In the following: Fig. 1 a diagram of a robot according to various embodiments. Fig. 2 is a schematic diagram illustrating the strategy of a neural network according to various embodiments. Fig. 3 is a flowchart of a computer-implemented method for training a neural network strategy according to various embodiments. Fig. 4 is a flowchart of a computer-implemented method for controlling the movement of a robot according to various embodiments. Fig. 5 is a conceptual diagram of a motion control device according to various embodiments. DESCRIPTION

[0011] Embodiments described below in connection with the devices apply mutatis mutandis to the respective methods, and vice versa. Furthermore, it goes without saying that the embodiments described below can be combined; for example, a part of one embodiment can be combined with a part of another embodiment.

[0012] It is understood that any property described herein for a specific device may also apply to any device described herein. It is understood that any property described herein for a specific method may also apply to any method described herein. Furthermore, it is understood that for any device or method described herein, not all of the described components or steps need necessarily be included in the device or method; rather, only some (but not all) of the components or steps may be included.

[0013] The term "coupled" (or "connected") may be understood herein as electrically coupled or as mechanically coupled, for example, attached or fixed, or merely in contact without any attachment, and it is understood that both direct coupling and indirect coupling (in other words, coupling without direct contact) may be provided.

[0014] In this context, the motion control device as described in this specification may include a memory used, for example, in the processing performed in the device. A memory used in the embodiments may be: a volatile memory, for example, a DRAM (Dynamic Random Access Memory) or a non-volatile memory, for example, a PROM (Programmable Read Only Memory), an EPROM (Erasable PROM), an EEPROM (Electrically Erasable PROM), or a flash memory, e.g., a floating-gate memory, a charge-trapping memory, an MRAM (Magnetoresistive Random Access Memory), or a PCRAM (Phase Change Random Access Memory).

[0015] In order that the invention may be easily understood and put into practice, various embodiments will now be described by way of example and not by way of limitation and with reference to the figures.

[0016] Fig. 1 shows a diagram of a robot 110 according to various embodiments. The robot 110 may include a motion controller 120. The robot 110 may further include at least one motor 112, an accelerometer 114, and an odometer 116. The odometer 116 may be, for example, an optical odometer or a LiDAR odometer. The robot 110 may be configured to accommodate various loads while traveling on different surface types. The motion controller 120 may have access to various vehicle data at any given time. The various vehicle data may include, for example, a torque applied to each motor 112, the speed of each motor 112, the linear acceleration and angular velocity from the accelerometer 114, and an external exteroceptive odometry signal from the odometer 116.Collectively, the vehicle data provides a time series of input data to the robot 110. The time series of input data may be divided into regular time intervals. Each time interval of input data is referred to herein as "episodic data." For example, in a robotic application, each time interval may be approximately 5 seconds, and the episodic data may comprise input data received within the 5 seconds. The motion controller 120 may include a neural network strategy 200 that can be trained to identify events, for example, if the robot 110 skidded while moving around a corner. In training the neural network strategy, episodic data concerning such events may be collected and input to the neural network strategy 200.

[0017] Fig. 2 shows a schematic diagram illustrating the neural network strategy 200 according to various embodiments. The motion controller 120 may be configured to store a fixed number of slots 210. The fixed number is denoted herein by S. Each slot 210 may store a respective volume of episodic data 212. The slots 210 may be referred to herein as episodic data slots 210. Each data episode to be stored may have the same number of time points. The number of time points is denoted herein by N. Each data episode may be referred to herein as episodic data 212. The neural network strategy 200 may include an episodic data encoding network (EDEN) 202 that converts each episodic data element 212 into an embedding vector, also referred to herein as an episodic data embedding vector 214.Each episodic data embedding vector 214 may be placed in an episodic data slot 210. The episodic data slots 210 may be provided to a slot selection network 204 configured to determine whether the current episodic data 212 should replace previous episodic data in one of the episodic data slots 210. The slot selection network may be trained using training data. The complete processing pipeline is referred to herein as a neural network strategy.

[0018] In other words, each episodic data embedding vector 214 may include N time points of input data, and each episodic data slot 210 may store one episodic data embedding vector 214. The motion controller 120 may store S slots 210 of episodic data and therefore store S × N time points of input data.

[0019] The neural network strategy 200 may further include a controller network 206. The controller network 206 may be configured to recall a small number of episodes from previous operations and combine them with the current episodic data to determine appropriate control actions to be taken in the future. At the same time, the slot selection network 204 may decide whether the current episodic data should be stored in episodic data slots for future use and, if so, which episodic data slot it should overwrite.

[0020] An example of a suitable neural network for the controller network 206 is disclosed in "Watch, Try, Learn: Meta-Learning from Demonstrations and Reward" by Zhou et al., which can be accessed at https: / / arxiv.org / abs / 1906.03352.

[0021] The neural network strategy 200 can simulate "forgetting" instead of forcibly deleting stored data. During the training process of the neural network strategy 200, the agent can be able to use an unlimited number of slots 210 of episodic data, including instances of the episodic data 212 affected by noise, such as additive noise or multiplicative noise. This way, no difficult decision is made during training to delete any data, nor is "soft" mixing of instances of episodic data 212 required.

[0022] During a training process, all episodic data 212 can be preserved. The output of the slot selection network 204 can be used to determine an appropriate noise scale for noise added to old episodic data. In this way, the neural network strategy 200 can learn which episodic data can be "forgotten" in a differentiable manner without requiring sampling or soft shuffling of episodic data during training.

[0023] During the training process, a trajectory sequence may be selected from the training data set at any time. At each time point, the neural network strategy 200 may be executed such that the last N time points of input data, also referred to herein as current episodic data 220, may be processed using the EDEN 202 into a current episodic data embedding vector 214'. The current episodic data embedding vector 214' may be stored in a buffer and input to the slot selection network 204. The slot selection network 204 may be configured to select one of the S slots 210 of episodic data to store the new input data, i.e., the current episodic data 220. The output of the slot selection network 204 may be referred to herein as the selection output. The selection output may be converted into a set of noise quantities by a noise scale calculation module 222.New noise 226 may be generated based on the calculated noise quantities. The new noise 226 may be added to the episodic data embedding vectors 214 stored in the episodic data slots 210. After adding the new noise 226, the controller network output 206 may be backpropagated to continue training the neural network strategy 200.

[0024] In another embodiment, if the slot 210 of episodic data was affected by noise at an earlier time 224, the noise magnitudes 226 may be reduced to prevent the noise magnitude from being arbitrarily large. For example, the magnitude of the noise at the earlier time 224 may be subtracted from the noise magnitudes 226 to determine the amount of new noise 226 to be added.

[0025] An example of the calculations performed by the noise scale calculation module 222 may be the Gumbel-Softmax reparameterization, as disclosed in “Categorical Reparameterization with Gumbel-Softmax” by Jang et al., which can be accessed at https: / / arxiv.org / abs / 1611.01144.

[0026] An example of a suitable training dataset is RobotNet, which can be accessed at https: / / github.com / SudeepDasari / RoboNet. While the RobotNet dataset was compiled for robot arms, the principle for training a neural network's strategy is similar, and in principle, the training dataset can also be applied to mobile robots. For this dataset, a suitable "reward" signal could be the negated root mean square of the joint accelerations for a given data sequence.

[0027] After the strategy 200 of a neural network has been trained, it can be deployed in a robot 110. During an inference process, the following steps can be performed:

[0028] First, the episodic data slots 210 can be initialized to the value "zero." This can be done when the robot 110 is commissioned in a factory or whenever the robot 110 is powered on.

[0029] Next, at any time, the last N time points of input data, also referred to herein as current episodic data 220, may be processed using EDEN 202 into a current episodic data embedding vector 214'. The current episodic data embedding vector 214 may be concatenated with the S slots 210 of stored episodic data items, resulting in S+1 episodic data embedding vectors 214. The resulting S+1 episodic data embedding vectors 214 may be provided as input to the slot selection network 204. The slot selection network 204 may be configured to select one of the S slots 210 of episodic data to store the new input data, i.e., the current episodic data 220. The output of the slot selection network 204 may be referred to herein as the selection output.The slot selection network 204 may assess whether the current episodic data 220 should replace an existing stored episodic data item in one of the episodic data slots 210. The slot selection network 204 may determine S probability values ​​and find the highest probability value among the K probability values. If the highest probability value exceeds a predetermined threshold, the neural network strategy 200 may replace the data of the corresponding episodic data slot 210 with the current episodic data 220.

[0030] In another embodiment, neural network strategy 200 may not examine the data present in each episodic data slot 210. Instead, neural network strategy 200 may only examine the current episodic data 220. In other words, current episodic data embedding vector 214' may not be associated with the S slots 210 of stored episodic data items. Slot selection network 204 may generate its selection output based on current episodic data embedding vector 214'. Slot selection network 204 may be trained to store a particular data type in each respective episodic data slot 210.

[0031] According to various embodiments, EDEN 202 may be a recurrent neural network (RNN), such as a long short-term memory network (LSTM). In one example, EDEN 202 may have 20 inputs, 256 hidden units, and 2 layers, resulting in a coding vector of length 256.

[0032] According to various embodiments, the slot selection network 204 may comprise one of a graph-based neural network, a convolutional neural network (CNN), an RNN, or a transformer network. An example of the graph-based neural network may comprise the following layers: (i) GraphConv(in_channels=256, out_channels=256); (ii) Relu; (iii) dropout (p=0.2); (iv) GraphConv(in_channels=256, out_channels=256); (v) Relu; (vi) Linear(256, 1); and (vii) Sigmoid.

[0033] Fig. 3 shows a flowchart of a computer-implemented method 300 for training a neural network strategy 200 according to various embodiments. The method 300 may include processes 302, 304, 306, 308, 310, and 312. The process 302 may include generating a plurality of episodic data slots 210 based on a trajectory sequence from a training data set. The process 304 may include, for each episodic data slot of the plurality of episodic data slots: generating a selection output based on the episodic data slot 210 using a slot selection network 204. The process 306 may include determining a noise magnitude based on the selection output for each episodic data slot 210 of the plurality of episodic data slots 210.Process 308 may include generating noise based on the determined noise magnitude for each episodic data slot 210 of the plurality of episodic data slots 210. Process 310 may include adding the generated noise 226 to the episodic data slot 210, resulting in an updated episodic data slot. Process 312 may include generating a motion control output based on the updated episodic data slots using a controller network 206. The resulting neural network strategy 200 may be capable of retrieving old, important information.

[0034] Although the embodiments described above relate to the use of the neural network strategy 200 to control the movement of a robot, the neural network strategy 200 may also be used to control other types of long-term processes.

[0035] According to an embodiment that may be combined with the embodiment described above or with any further embodiment described below, generating noise based on the determined noise magnitude may comprise subtracting the magnitude of a noise at a previous time from the determined noise magnitude to result in a reduced noise magnitude, and generating the noise based on the reduced noise magnitude. This may prevent the noise level from reaching an excessively large magnitude because noise levels accumulate over the trajectory sequence.

[0036] According to an embodiment that may be combined with the embodiment described above or with any further embodiment described below, the method 300 may further comprise:

[0037] Controlling the movement of a robot based on the motion control output and measuring the quality of the robot's behavior, resulting in an effectiveness value for the option, and backpropagating the effectiveness value. The neural network strategy can then be trained to generate motion control outputs that maximize the quality of the robot's behavior.

[0038] According to an embodiment that can be combined with the embodiment described above or with any further embodiment described below, wherein generating the plurality of episodic data slots 210 comprises generating each episodic data slot 210; encoding a number N of time points of training data into an episodic data embedding vector 214 using an episodic data encoding network 202. The episodic data encoding network 202 may comprise a long short-term memory (LSTM) network.

[0039] According to an embodiment that may be combined with the embodiment described above or with any further embodiment described below, the slot selection network 204 may comprise at least one of a fully connected graph-based neural network and a transformer network.

[0040] According to an embodiment that may be combined with the embodiment described above or with any further embodiment described below, the training data may include motion control commands provided to a robot 110.

[0041] Fig. 4 shows a flowchart of a computer-implemented method 400 for controlling the motion of a robot 110 according to various embodiments. Method 400 may include processes 402, 404, and 406. Process 402 may include inputting current episodic data 220 to the slot selection network 204 of a neural network strategy 200 trained according to method 300. Process 404 may include replacing one of a plurality of stored slots 210 of episodic data with the current episodic data 220 based on a selection output from the slot selection network 204 to result in an updated plurality of slots of episodic data. Process 406 may include generating a motion control output based on the updated plurality of slots of episodic data using the controller network 206.

[0042] According to an embodiment that may be combined with the embodiment described above or with any further embodiment described below, inputting the current episodic data 220 to the slot selection network 204 may include encoding the current episodic data 220 to result in a current episodic data embedding vector 214' and further comprising concatenating the current episodic data embedding vector 214' with episodic data embedding vectors 214 from the plurality of stored episodic data slots 210. As a result, the slot selection network 204 considers the content of all episodic data slots 210, which may improve the accuracy of the selection output.

[0043] According to an embodiment that can be combined with the embodiment described above or with any further embodiment described below, the slot selection network 204 can be configured to determine, for each episodic data embedding vector 214 of the respective stored episodic data slots 210, a respective probability value of the current episodic data embedding vector 214' that is a substitute for the episodic data. The probability value can indicate the usefulness of the episodic data. The probability value is useful for debugging the accuracy of the slot selection network so that perturbative analysis of the slot selection network's decisions can be performed.

[0044] According to an embodiment that may be combined with the embodiment described above or with any further embodiment described below, replacing one of the plurality of stored episodic data slots 210 with the current episodic data embedding vector 214' based on a selection output from the slot selection network 204 may comprise: identifying a highest probability value among the probability values ​​determined for the plurality of stored episodic data slots 210, comparing the highest probability value to a predetermined threshold, and replacing the stored episodic data slot 210 corresponding to the highest probability value based on the highest probability value exceeding the predetermined threshold.This prevents the episodic data slot 210 from being replaced by the current episodic data in the scenario where the current episodic data is irrelevant or redundant.

[0045] Fig. 5 shows a conceptual diagram of a motion control device 500 according to various embodiments. The motion control device 500 may be used with a robot 110 to direct the movements of the robot 110. The motion control device 500 may include a memory 502 and a processor 504. The memory 502 may store a computer program including instructions that, when executed by the processor 504, perform the computer-implemented method 400 for controlling the movement of a robot 110. The memory 502 and the processor 504 may be coupled to each other, for example, electrically and / or communicatively, by the coupling line 550.

[0046] Various aspects of the described method 400 may also apply to the use of any motion control device 500 described above.

[0047] While embodiments of the invention have been particularly shown and described with reference to specific embodiments, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. The scope of the invention is thus indicated by the appended claims, and all changes which come within the spirit and range of equivalence of the claims are therefore intended to be embraced. It will be appreciated that common reference numerals used in the relevant drawings refer to components serving a similar or the same purpose.

[0048] Those skilled in the art will appreciate that the terminology used herein is for the purpose of describing various embodiments only and is not intended to be limiting of the present invention. As used herein, the singular forms of the indefinite and definite articles are intended to include the plural forms as well, unless the context clearly indicates otherwise. Further, it is to be understood that the terms "comprises" and / or "comprising," when used in this specification, precisely indicate the presence of specified features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0049] It is understood that the specific order or ranking of blocks in the disclosed process / flowcharts is an illustration of exemplary approaches. Based on preferred embodiments, it is understood that the specific order or ranking of blocks in the process / flowcharts may be rearranged. Further, some blocks may be combined or omitted. The appended method claims present elements of the various blocks in a sample order and are not intended to be limited to the specific order or ranking shown.

[0050] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications of these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein, but are to be accorded the full scope of protection consistent with the language of the claims, with reference to an element in the singular not meaning "one and only one" unless expressly so stated, but rather "one or more." The word "exemplary" is used herein to mean "serving as an example, instance, or illustration."No aspect described herein as "exemplary" should necessarily be construed as preferred or advantageous over other aspects. Unless expressly stated otherwise, the term "some" refers to one or more. Combinations such as "at least one of A, B, or C," "one or more of A, B, or C," "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, C, or any combination thereof" include any combination of A, B, and / or C, and may include multiples of A, multiples of B, or multiples of C.In particular, combinations such as "at least one of A, B, or C," "one or more of A, B, or C," "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, C, or any combination thereof" may be A only, B only, C only, A and B, A and C, B and C, or A and B and C, where all such combinations may include one or more elements of A, B, or C. All structural and functional equivalents of the elements of the various aspects described within this disclosure that are known or later become known to one of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims.

Claims

[1] A computer-implemented method (300) for training a neural network strategy, the method comprising: generating a plurality of slots of episodic data based on a trajectory sequence from a training data set (302); for each episodic data slot of the plurality of episodic data slots: generating a selection output based on the slot of episodic data using a slot selection network (304), Determining a noise quantity based on the selection output (306), Generating noise based on the determined noise quantity (308) and adding the generated noise to the episodic data slot, resulting in an updated episodic data slot (310); and Generating a control output based on the updated episodic data slots using a controller network (312). [2] The method (300) of claim 1, wherein generating noise based on the determined noise magnitude (308) comprises: Subtracting the magnitude of a noise at an earlier time from the determined noise magnitude to result in a reduced noise magnitude; and Generate the noise based on the reduced noise magnitude. [3] Method (300) according to one of the preceding claims, further comprising: Controlling the movement of a robot based on the control output; Measuring the quality of the robot’s behavior, resulting in an effectiveness value of the option; and Backpropagation of the effectiveness value. [4] The method (300) of any preceding claim, wherein generating the plurality of slots of episodic data (304) comprises encoding a number N of time points of training data into an episodic data embedding vector using an episodic data encoding network. [5] The method (300) of claim 4, wherein the network for encoding episodic data comprises a long short-term memory network. [6] The method (300) of any preceding claim, wherein the slot selection network comprises at least one of a fully connected graph-based neural network and a transformer network. [7] The method (300) of any preceding claim, wherein the training data comprises motion control commands provided to a robot. [8] A computer-implemented method (400) for controlling the movement of a robot, the method (400) comprising: Inputting current episodic data into the slot selection network (402) of a neural network strategy trained according to the method (300) of any preceding claim; replacing one of a plurality of stored slots of episodic data with the current episodic data based on a selection output from the slot selection network to result in an updated plurality of slots of episodic data (404); and Generating a motion control output based on the updated plurality of slots of episodic data using the controller network (406). [9] The method (400) of claim 8, wherein inputting the current episodic data to the slot selection network (402) comprises encoding the current episodic data to result in a current episodic data embedding vector, and further comprising concatenating the current episodic data embedding vector with episodic data embedding vectors from the plurality of stored episodic data slots. [10] The method (400) of any preceding claim, wherein the slot selection network is configured to determine, for each episodic data embedding vector of the respective stored episodic data slots, a respective current episodic data embedding vector probability value that is a substitute for the episodic data. [11] The method (400) of claim 10, wherein replacing one of the plurality of stored slots of episodic data with the vector for embedding current episodic data based on a selection output from the slot selection network comprises: Identifying a highest probability value among the probability values determined for the plurality of stored slots of episodic data, comparing the highest probability value with a predetermined threshold, and replacing the stored slot of episodic data corresponding to the highest probability value based on the highest probability value exceeding the predetermined threshold. [12] A computer program comprising a neural network strategy trained according to any one of claims 1 to 7. [13] A computer program comprising instructions which, when executed by a processor, performs the method of any one of claims 8 to 11. [14] A motion control device (500) for use in a robot, the motion control device (500) comprising: a memory (502) storing the computer program according to claim 13; and a processor (504) configured to execute the instructions of the computer program.

Images