Device and method for training a robot device controller

Abstract

According to various embodiments, a method for training a robot device controller is described, comprising providing demonstrations of an execution of a skill by a robot device, wherein each demonstration comprises for each time point of a sequence of time points a state observation including sensor data for each sensor modality of multiple sensor modalities, determining, for each demonstration and each time point, an embedding of the sensor data for each sensor modality, determining actions by the robot device controller from the embeddings, determining a loss including an imitation loss between the demonstrations and trajectories of the robot device caused by the determined actions and including a latent space regularization loss that punishes differences between embeddings of sensor data of different modalities for the same time point and training the robot device controller to reduce the loss.

Description

[0001]R.415204 -1 -Description TitleDevice and Method for Training a Robot Device ControllerThe present disclosure relates to devices and methods for training a robot device controller. Robot manipulation represents a dynamically evolving subfield of robotics, focus- ing on empowering robots to interact with and alter their environments through physical actions. This domain integrates perception, planning, control, and learn- ing to enable autonomous task execution. Perception utilizes sensors, such as cameras and LIDAR, to gather environmental data. Planning determines the nec- essary sequence of movements, and control implements these plans with high precision. Two prominent robotic learning algorithms currently employed are Re- inforcement Learning (RL) and Imitation Learning (IL). Imitation learning allows a robot to directly acquire skills from expert demonstra- tions. IL involves directly mapping states and observations (e.g. camera feed) to actions based on these demonstrations, offering advantages such as sample effi-ciency, intuitive and quick robot programming and – in comparison to reinforce-ment learning – simple implementation without needing a complex reward func-tion. The challenge of relying solely on vision for robot manipulation involves severalkey issues, such as limited depth perception, vision occlusion and lack of surfacedetail information. To address these limitations, sensor fusion may be used which involves integrating data from multiple sensors (in particular of different modali- ties) to achieve a comprehensive understanding of the environment. However, R.415204 -2 -traditional sensor fusion methods often rely on predefined models that combinesensor inputs in relatively straightforward ways and therefore lack the contextualunderstanding and struggle with the variability and complexity of real-world ma- nipulation tasks. Accordingly, approaches for sensor fusion or learned sensor fu- sion in robot device control (i.e. robot device control based on sensor data of multiple modalities), in particular in an imitation learning context, are desirable. According to various embodiments, a method for training a robot device controller is described, comprising providing demonstrations of an execution of a skill by a robot device, wherein each demonstration comprises for each time point of a se- quence of time points a state observation including sensor data for each sensor modality of multiple sensor modalities, determining, for each demonstration and each time point, an embedding of the sensor data for each sensor modality, de- termining actions by the robot device controller from the embeddings, determin- ing a loss including an imitation loss between the demonstrations and trajectories of the robot device caused by the determined actions and including a latent space regularization loss that punishes differences between embeddings of sen- sor data of different modalities for the same time point and training the robot de- vice controller to reduce the loss.The method described above allows robot imitation learning with high data effi-ciency and robustness when working with multiple sensor modalities. Instead ofonly considering the imitation learning objective, that is, to reproduce robot ac-tions given input sensor data (i.e. state observations), a latent space regulariza-tion loss (and possibly further losses, see below) is used in addition. This loss (or these losses) reflect self-supervised regularization objectives which provides a robot device controller (which implements a machine learning model which repre- sents the learned control policy) with increased generalization capabilities and robustness. In the following, various examples are given. Example 1 is a method for training a robot device controller as described above. R.415204 -3 -Example 2 is the method of example 1, wherein determining the actions com- prises determining action features (i.e. encoded actions) from the embeddings and decoding the action features to actions. The determining of actions thus happens in a two-stage process, wherein in an encoding stage action features (which may be seen as encoded actions) are de- termined and in a second stage actions are determined from the action features. Based on the action features, a further loss component may be included as de- scribed in the following.Example 3 is the method of example 2, further comprising randomly masking em-beddings and determining action features from the randomly masked embed- dings, wherein the loss further includes a masking loss which punishes differ- ences between action features determined from the embeddings and action fea- tures determined from the randomly masked embeddings (referred to as MFM loss in the embodiments described below). This increases the robot device controller’s ability to decode and integrate infor- mation across various sensory inputs and temporal sequences.Example 4 is the method of example 3, wherein the masking loss is determinedby determining proprioception sensor data from the action features determinedfrom the randomly masked embeddings for one or more time points of the se-quence of time points and comparing the determined proprioception sensor data with proprioception sensor data included in the state observations for the one or more time points. This provides an effective determination of a masking loss with little computa- tional effort.Example 5 is the method of any one of examples 1 to 4, further comprising fusingthe embeddings to fused embeddings for each time point, randomly shuffling the R.415204 -4 -fused embeddings over the sequence of time points, determining action features from the shuffled fused embeddings, and estimating a correct order of the fused embeddings, wherein the loss further comprises a shuffling loss (607) which pun- ishes differences between the estimated order and the true order of the fusedembeddings (wherein the true order is the order of the embeddings according tothe sequence of time points). This increases the robot device controller’s ability to interpret and utilize temporal information.Example 6 is the method of any one of examples 1 to 5, wherein determining thelatent space regularization loss comprises projecting the embeddings for at leastsome of the modalities to a common sensor data space and determining at least a component (e.g. the component with respect to the modalities whose embed- dings where projected to the common sensor data space) of the latent space reg- ularization loss as a contrastive loss within the common sensor data space (if one of the modalities is already in the sensor data space that is chosen as the common sensor data space this projection is trivial, e.g. the identity).Example 7 is a method for controlling a robot device, comprising training a robotdevice controller according to any one of examples 1 to 6, supplying observed sensor data for each of the multiple sensor modalities, determining an embed- ding of the observed sensor data for each sensor modality, determining actionsby the robot device controller from the embeddings of the observed sensor dataand controlling the robot device to perform the actions.Example 8 is a robot device controller, configured to perform a method of anyone of examples 1 to 7.Example 9 is a computer program comprising instructions which, when executedby a computer, makes the computer perform a method according to any one of examples 1 to 7. R.415204 -5 -Example 10 is a computer-readable medium comprising instructions which, whenexecuted by a computer, makes the computer perform a method according toany one of examples 1 to 7.In the drawings, similar reference characters generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, empha- sis instead generally being placed upon illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings, in which: Figure 1 shows a robot. Figure 2 illustrates the training of a machine learning model according to an embodiment. Figure 3 shows an encoder 300 according to an embodiment. Figure 4 illustrates Multimodal Contrastive Learning (MCL). Figure 5 illustrates Masked Frame Modelling (MFM). Figure 6 illustrates Frame Order Modelling (FOM). Figure 7 shows a flow diagram illustrating a method for training a robot de- vice controller. The following detailed description refers to the accompanying drawings that show, by way of illustration, specific details and aspects of this disclosure in which the invention may be practiced. Other aspects may be utilized and struc- tural, logical, and electrical changes may be made without departing from the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure can be combined with one or more other aspects of this disclosure to form new aspects. R.415204 -6 -In the following, various examples will be described in more detail. Figure 1 shows a robot 100. The robot 100 includes a robot arm 101, for example an industrial robot arm for handling or assembling a work piece (or one or more other objects 113). The ro- bot arm 101 includes manipulators 102, 103, 104 and a base (or support) 105 by which the manipulators 102, 103, 104 are supported. The term “manipulator” re- fers to the movable members of the robot arm 101, the actuation of which ena- bles physical interaction with the environment, e.g. to carry out a task. For con- trol, the robot 100 includes a (robot) controller 106 configured to implement the interaction with the environment according to a control program. The last member 104 (furthest from the support 105) of the manipulators 102, 103, 104 is also re- ferred to as the end-effector 104 and includes a grasping tool (which may also be a suction gripper). The other manipulators 102, 103 (closer to the support 105) may form a position- ing device such that, together with the end-effector 104, the robot arm 101 with the end-effector 104 at its end is provided. The robot arm 101 is a mechanical arm that can provide similar functions as a human arm. The robot arm 101 may include joint elements 107, 108, 109 interconnecting the manipulators 102, 103, 104 with each other and with the support 105. A joint ele- ment 107, 108, 109 may have one or more joints, each of which may provide ro- tatable motion (i.e. rotational motion) and / or translatory motion (i.e. displacement) to associated manipulators relative to each other. The movement of the manipu- lators 102, 103, 104 may be initiated by means of actuators controlled by the controller 106. The term "actuator" may be understood as a component adapted to affect a mechanism or process in response to be driven. The actuator can implement in- structions issued by the controller 106 (the so-called activation) into mechanical R.415204 -7 -movements. The actuator, e.g. an electromechanical converter, may be config- ured to convert electrical energy into mechanical energy in response to driving. The term "controller" may be understood as any type of logic implementing entity, which may include, for example, a circuit and / or a processor capable of executing software stored in a storage medium, firmware, or a combination thereof, and which can issue instructions, e.g. to an actuator in the present example. The con- troller may be configured, for example, by program code (e.g., software) to con- trol the operation of a system, a robot in the present example. In the present example, the controller 106 includes one or more processors 110 and a memory 111 storing code and data based on which the processor 110 con- trols the robot arm 101. According to various embodiments, the controller 106 controls the robot arm 101 on the basis of a machine-learning model (e.g. includ- ing one or more neural networks) 112 stored in the memory 111. The machine-learning model may be trained using reinforcement learning (RL), e.g. using actor-critic RL. According to various embodiments, robot manipulation skills are learned (i.e. the machine-learning model 112 is trained) via kinesthetic teaching, or teleoperationsuch that these skills can be reproduced in new situations (i.e. situations notseen in training). The machine learning model 112 which takes sensor readings, such as, images, audio and robot states and produces robot end-effector trajec- tories. The machine learning model 112 is trained by recording demonstrations ofthe tasks (where the skills to be learned are performed, e.g. by a human operatorwho guides the robot arm 101), which consist of a trajectory of observations andactions. According to various embodiments, this is based on an Action ChunkingTransformer, which takes a history of observation and generates a trajectory of robot actions. According to various embodiments, multiple input sensor modali- ties are considered and a self-supervised regularization approach is used for im- proved imitation learning performance. R.415204 -8 -Figure 2 illustrates the training of a machine learning model (e.g. corresponding to the machine learning model 112) according to an embodiment. The machine learning model comprises a (multisensory) observation encoder213 which receives sensor data of multiple modalities 201, 202, 203. The obser-vation encoder 213 produces encoded actions 204, 205, i.e. (action) features,which include an aggregated action feature 204, denoted as YAgg, and actionfeatures 205, denotes as Yaction. Yagg is an output token of an aggregation to-ken Zagg. The Zagg is a “classification” token, that is, it is not a real feature butjust a placeholder. This token can pay attention to every other token, but no othertoken can pay attention to it. Thus, it functions as a “summary” or classificationtoken that does not influence feature token representation but rather tells some-thing about the features.The action features 205 are fed to an action decoder 206 which comprises anumber Ldec (e.g. Ldec = 7) of transformer layers. Specifically, the action fea-tures 205 are fed to a cross attention block 207 whose output is fed to a feed for-ward network 208. The output of the action decoder 206 is fed to an action head215 which generates a (predicted) action sequence 209, denoted as Apred. Fortraining, the predicted action sequence 209 is compared with the real (i.e. ob-served, e.g. demonstrated) action sequence, denoted as Areal 210, from thetraining data (e.g. demonstrations). Comparing here means the calculation of a loss 214, e.g. an L1 loss. The machine learning model (i.e. its weights) are then adjusted to reduce the loss (typically by backpropagation).Both the encoder 213 and the decoder 206 receive a positional encoding (e.g.2Dsinusoidal positional encodings) 212, denotes as Epos. In other words, the en-coder 213 and the decoder 206 share positional embeddings.In the decoder 206, to optimize the use of positional correlations among the in- puts, the sequence is initially processed through a multi-head self-attention layer211 equipped with a causal attention map. Subsequent to the multi-head atten-tion layer 211, the cross-attention block (i.e. layer) 207 integrates the information R.415204 -9 -derived from the observations by the encoder 213. Here, the output from the ob-servation encoder Yaction functions as both the key and value, while the outputfrom the multi-head causal self-attention 211 serves as the query.In the following, the multisensory observation encoder 213 is described in moredetail. Figure 3 shows an encoder 300 according to an embodiment. As described with reference to figure 2, the encoder 300 receives sensor data of multiple modalities as input, in this example vision inputs 301 (i.e. images, e.g. RGB and / or depth images), audio inputs 302 and proprioceptive inputs 302, also denoted by O. For the vision inputs V, defined by the tensor shape (B,N,3,HV,WV), the encoder comprises a pre-trained ResNet-18 model 313, enhancing its capability for spatial recognition. Each image in the respective batch undergoes individual processing. Subsequently, the channel dimensions of the initial convolutional layer in ResNet- 18 are adjusted to 5 to accommodate the augmented input. The Resnet’s stand- ard average pooling layer is replaced by an adaptive max pooling layer to better capture the maximal activation across spatial dimensions. The feature extraction culminates with a linear transformation (i.e. embedding) layer 314 that projects the image features to vision data embeddings 304 in a designated vision embed- ding space. Audio inputs A, represented as 1D waveform signals sampled at 16kHz, aretransformed into mel spectrograms 305 through a Short-Time Fourier Transform(STFT) employing a Hann window. The window and hop lengths are e.g. set to 50ms, creating non-overlapping windows, which simplifies synchronization with other modalities. For a standard 2.5-second audio segment, this process yields a mel spectrogram of dimensions 64 ÷ 51(64 x 101 for 5-second input). To facilitatetemporal alignment with the vision modality, each mel spectrogram 305 is seg-mented along the time axis into five patches, each of size 64 ÷ 10, for example. R.415204 -10 -To ensure even partitioning and mitigate overfitting, random columns within the mel spectrogram are omitted during this partitioning process. These patches are then flattened and passed through a linear embedding layer 315 to produce thefinal audio embeddings 306, matching the number of vision embeddings.Proprioceptive inputs O consist of 7-dimensional vectors. These inputs are di-rectly processed through a Multilayer Perceptron (MLP) 316. This MLP selec-tively maps the proprioceptive data to pose embeddings 307 in a pose embed-ding space, aligning the number of pose embeddings 307 with those of the vision and audio modalities. The extraction and embedding transformations thus yield three distinct sets of la-tent features 304, 306, 307, denoted as LV, LA, LO, respectively (i.e. sensor dataembeddings for the vision, audio and proprioceptive sensor modality) with a uni-form feature count N maintained across all groups. This configuration underpinsthe multimodal integration strategy, paving the way for effective downstream pro-cessing and fusion methodologies. For the fusion of these modalities, accordingto one embodiment, an Early Summation strategy is used, which is particularlyadvantageous for synchronizing modalities while maintaining temporal coher- ence. This approach involves the aggregation of feature vectors from corre- sponding time steps across different modalities.Specifically, for each triplet of features {LV, LA, LO} from a given time step, a mo-dality-adaptive fusion module 308 is used. The modality-adaptive fusion module308 operates through a weighted linear combination of the features, where the weights are modality-specific and learnable during training. The fusion process is mathematically represented as: In this equation, α, β and γ denote the trainable weights assigned to the vision,audio, and proprioception modalities, respectively. By leveraging these weights, the Early Summation technique not only facilitates the effective integration of R.415204 -11 -heterogeneous data but also ensures that the temporal alignment of the modali- ties is preserved. The features fused in this manner from all (considered) time steps are compiled into a cohesive set, resulting in a series of spatially sequential, fused cross-mo- dality observation feature vectors (also referred to as fused embeddings, since they are generated by fusing the embeddings LV, LA, LO) 309 denoted as Z =[Z1; Z2; ··· ; ZN].At the last part of the multisensory observation encoder 300, the cross-modalityobservation feature vectors 309 are subsequently processed through a cross-time transformer 310, an architecture related to the Vision Transformer (ViT). TheVision Transformer, originally conceived for computer vision tasks, segments animage into a series of fixed-size patches, akin to tokenizing text. This method in- volves flattening each patch and embedding it as a vector, much like embedding words in NLP applications. Given that the transformer does not natively handle sequential data, positional embeddings are integrated to maintain spatial relation- ships within the image. The cross-time transformer 310 diverges from the traditional ViT in that it is spe- cifically tailored to handle pre-derived tokens from cross-modality summation, by- passing the need for initial patchification or tokenization typically required in ViT. Further modifications can be made to the positional embedding mechanism tobetter suit the specific requirements of the robot skill learning use case. In con-trast to the typical trainable vectors used in standard vision encoder setups, ac- cording to various embodiments, traditional 2D sinusoidal positional embeddingsare used for the sequentially aligned input feature vectors (corresponding to thepositional encoding 212). These are effective in temporal data contexts. For addi-tional tokens, learnable embeddings are still utilized, allowing for greater flexibility and adaptation to the specific characteristics of the input data. According to oneembodiment, the original classification head from ViT is adopted as the aggrega-tion token 311, denoted as ZAgg, which serves to represent the entirety of theobservational trajectory, an approach that leverages the transformer’s flexibility in R.415204 -12 -handling a variable number of input tokens. This feature proves particularly ad- vantageous when incorporating longer observation histories or additional datatypes, such as a style vector 312, denoted by ZStyle, (e.g. indicating a userwhich has demonstrated a respective demonstration) in a conditional variational autoencoder (CVAE) training pipeline. The forward process of the Cross-TimeTransformer 310 can be mathematically represented as: where represents the attention block index while L represents the maximumnumber of attention blocks. MSA represents Multi-Head Self-Attention, MLP rep-resents Multi-Layer Perceptron, and Epos represents the positional embedding.The output from the multisensory observation encoder 300 includes, as men-tioned in context of figure 2,• An Aggregation Feature Vector YAgg: This vector encapsulates comprehen-sive information across the entire observational span, which could serve for downstream classification or regression tasks.• A Sequence of Action Feature Embeddings YAction or Ya: These embed-dings synthesize inputs from the three modalities—vision, audio, and proprio- ception—and are poised for use as key-value pairs in the cross-attention modules of subsequent transformer-based action decoders. The encoder architecture of figure 3, while effective in a supervised learning con- text, encounters significant challenges when applied to real-world data acquisi- tion for imitation learning. The process of recording data via teleoperation using dual robotic arms is not only time-intensive but also costly. Furthermore, in practi- cal application, a substantial portion of the recorded trajectories—approximately R.415204 -13 -half—are rendered unusable due to excessive noise or considerable movement errors. These constraints highlight a pressing need for more robust methodologies capa- ble of enhancing data utilization and model training efficiency. In view of this, according to various embodiments, the encoder’s generalizationcapabilities and its efficiency in processing multisensory observations are en-hanced by providing it with a better capability to understand cross-modality andcross-time relationship between observation from diverse data types. Accordingto various embodiments, the encoder is trained using self-supervised learning ap-proaches which can effectively utilize lower quality trajectories that would other-wise be discarded in a purely supervised framework. To this end, according to various embodiments, three specific pretext tasks areprovided to augment the encoder 300 (wherein one, two or all of them may beused). This means that for training the robot device controller, i.e. in particular themachine-learning model 112, in addition to the loss 214 (which can be seen asan imitation loss) further loss components are used, i.e. training is performed to reduce a (total) loss which includes the imitation loss 214, a Multimodal Contras- tive Learning (MCL) loss, which can in general be seen as a latent space regular- ization loss, a Masked Frame Modelling (MFM) loss, which can in general be seen as a masking loss and a Frame Order Modelling (FOM) loss, which can in general be seen as a shuffling loss.So, the encoder is trained to perform one or more pretext tasks which are de-signed to refine different aspects of the encoder’s functionality. They are Multi-modal Contrastive Learning (MCL), Masked Frame Modelling (MFM) and Frame Order Modelling (FOM) and are described in the following. Figure 4 illustrates Multimodal Contrastive Learning (MCL). R.415204 -14 -Multimodal Contrastive Learning (MCL) is a technique aimed at improving therepresentation learning of data from multiple modalities (such as video, audio, and text) by employing contrastive loss functions. These functions maximize the agreement between semantically similar pairs and minimize it between dissimilar pairs across different modalities. This technique allows learning robust embed- dings that can be effectively used in various downstream tasks such as classifi- cation, retrieval, and recognition tasks. In an embodiment with visual, audio and proprioception inputs like in the example of figure 2 and 3, this methodology is extended to include vision, audio, and pro- prioception inputs. According to one embodiment, an approach denoted as common space projec- tion is used which is designed to adeptly navigate the inherent challenges posed by the significant granularity and domain discrepancies among vision, audio, and proprioception inputs. Common space projection integrates with contrastive learning to enhance the training through a strategic alignment of multimodal data. According to various embodiments, a common space projection and contrastive learning are used in that common space to train the components of the encoder300. More specifically, given a set of latent features {LV, LA, LO} extracted fromraw multisensory inputs as described with reference of figure 2, i.e. a vision em-bedding 401, an audio embedding 402 and a pose embedding 403, respectively , a semantically hierarchical common space mapping is defined that allows directlycomparing vision-audio pairs as well as vision-proprioception pairs. Such a com-parison is more feasible if different levels of semantic granularity for these modal-ities are assumed. To achieve this, multi-level projections are defined as follows: R.415204 -15 -Here, a first projection head 404, denoted as and a second projectionhead 405, denoted as , are employed to transform the vision and audiofeature vectors into a shared vision-audio common space, denoted as SVA. Addi- tionally, the proprioception features and the vision embeddings from vision-audiocommon space SVA are further integrated by a third projection head 406, de-noted by and a fourth projection head 407 and fifth projection head408, denoted both by which map these features into a more com-prehensive vision-audio-proprioception space, SVAO.According to various embodiments, linear projections are used for and. For , according to one embodiment, a more complex two- layer projection with ReLU activations to accommodate the nuanced semanticdetails pre sent in visual data is used. Similarly, according to one embodiment,for , a non-linear projection involving ReLU is employed to ensure a robust fusion of the vision and audio embeddings into the proprioceptive space. To facilitate model training and stability, a normalization layer follows each pro- jection, enhancing the overall efficacy and reliability of the embedding transfor- mations.The multi-level projection approach of figure 4 allows exploiting the inherent dif-ferences in semantic granularity across various modalities. By embedding thesedifferences as an inductive bias within the projections, the encoder can process and analyze multimodal data more accurately and effectively. Furthermore, the hierarchical projection technique of figure 4enables the combination of feature vectors from distinct modalities into a single unified common space, SVAO, by introducing an additional projection step as follows: where the same projection head can also be used to map theaudio embeddings from vision-audio common space SVA. This approach allowsfeature vectors from different modalities, each characterized by unique levels of R.415204 -16 -granularity, to coexist within the same embedding space. This unified embedding space facilitates not only the visualization but also the subsequent comparative analysis of the multimodal data.A Cross-modality Noise Contrastive Estimation (NCE) is incorporated to align vi-sion audio pairs and to vision-proprioception pairs. The pairs are composed from different temporal locations in the multisensory input stream. Positive pairs from two modalities are constructed by sampling their corresponding streams from the same location in the video, and negative pairs are constructed by sampling from any non-matching locations in the video. specifically, according to various em- bodiment, for each group of latents {LV, LA, LO} encompassing multiple frames, sequential compatibility is enhanced by concatenating the common space em- beddings of adjacent frames within the same observation history along channel dimension, into a single composite feature vector. This process ensures that em- beddings of different modalities from the same temporal sequence are treated as positive samples. In contrast, embeddings from different observation histories, in our case, the different samples from the same batch, are designated as negative samples, thus enriching the model’s ability to discern and differentiate between coherent and incoherent multimodal sequences. So, according to various embodiments, NCE losses 409, 410 are added to the respectively, where N comprises all non-matching pairs from a given batch (of training data). The parameter τ, known as the temperature, is utilized to modulate the sensitivity of the loss function, fine-tuning the distinction between positive andnegative pairs by adjusting the sharpness of the softmax function used in thecontrastive learning process. R.415204 -17 -So, if Multimodal Contrastive Learning (MCL) is applied, the two Noise Contras- tive Estimation (NCE) losses 409, 410 are added to the loss 214. The MCL loss is calculated as the average loss of a batch of samples and given by Figure 5 illustrates Masked Frame Modelling (MFM).For each set of the triplet latent vectors 501, 502, 503 denoted as {LV, LA, LO},there exists a probability for each latent to be replaced with a mask token by arespective random masking 504, 505, 506 (e.g.25%). Three trainable vectors areused as mask tokens for each sensor modality. Consequently, after performingcross-modality fusion (summation) 507, N randomly masked cross-modalityframe features 508, denoted by Zm, are obtained.These features 508 are then input into a cross-time transformer 509 which lever-ages the interdependencies across time to reconstruct the masked action fea-tures 515, denoted by (which may also be aggregated to an aggre-gated masked action feature 516): In addition to the features 509, the cross-time transformer 509 receives the ag-gregated feature 513, denoted by ZAgg (generated by the cross-modality fusionmodule 507) as well as a style vector 514, denoted by ZStyle.Subsequently, a projection head 516 in form of a simple fully connected (FC)layer is employed to map the masked action feature back to theraw proprioceptive space and thus generate a recovered proprioception input510, represented as: R.415204 -18 -An L2 regression between the recovered proprioceptive input and the is used.So, if Masked Frame Modelling (MFM) is applied, an MFM loss 512 is added tothe loss 214. The MFM loss is for example computed as follows: This loss component significantly refines the machine learning model’s ability to decode and integrate information across various sensory inputs and temporal se- quences, thereby enhancing its overall predictive and interpretative capabilities. Figure 6 illustrates Frame Order Modelling (FOM).FOM facilitates a deeper comprehension of the temporal structures within multi-sensory inputs. Traditional models that analyze single image frames lack the ca- pability to interpret and utilize the sequential nature of sequential data, which is essential for accurately capturing the dynamics and causal relationship observed in demonstrations, which is essential for robotic imitation learning.FOM addresses the challenge of temporal sequence understanding by manipu-lating the frame order within a subset of frame features 601 provided by thecross-modality fusion (summation in the present embodiment) module 308. Spe-cifically, this involves random shuffling of these frame features 601. This is then used for a loss 607 to train the machine learning model to predict their correct chronological sequence. This task is conceptualized as a classification challenge where the machine learning model assigns each frame to its original temporal po- sition based on learned temporal dependencies.Specifically, after applying cross-modality summation by the cross-modality fu-sion module 308 a subset of the resulting features 601, with n original R.415204 -19 -timestamps is selected and shuffled, resulting in a new feature set.These shuffled features are then processed through a cross-time transformer 602to generate shuffled action features 603 (along with an aggregated action feature604). Notably, the traditional 2D sinusoidal positional embedding for the inputs tothis transformer are preserved, which aids in leveraging temporal interdependen-cies to retrieve the original ordering of the frames. The equation for this process is: The cross-time transformer 602 may further receive the aggregated feature 608 (generated by the cross-modality fusion module 308) as well as a style vector 609. Subsequently, a fully connected (FC) layer is employed as a classification head logits. Theselogits are then processed through a SoftMax layer to compute probability vectors 606, as represented by the equation The FOM loss reflects the objective to minimize the negative log-likelihood(cross-entropy loss) between the predicted probabilities of the shuffledsubset and the original timestamps i.e. true order indices 610. Ac- the Frame Order (FOM) loss is computed as follows: R.415204 -20 -It should be noted that that incorporating masked frame features akin to theMasked Frame Modeling approach described above, significantly enhances theperformance of FOM. Accordingly, according to various embodiments, a MaskedFrame Order Modeling (MFOM) is used which utilizes masked frame features topotentially reduce overfitting and maximize the use of complementary multisen- sory inputs. Specifically, after the random masking and cross-modality fusion, Nrandomly masked cross-modality frame features are used where a subset ofthese features is randomly selected and shuffled, resulting in a new shuffledmasked cross-modality frame features set .So, if Masked Frame Order Modelling (MFM) is applied, an MFM loss 607 is used as a loss added to the loss 214. It is Usage of this loss component in training enhances the machine learning model’scapability to interpret and utilize temporal information (i.e. improves its under-standing of the temporal dynamics). In summary, according to various embodiments, a method is provided as illus- trated in figure 7. Figure 7 shows a flow diagram 700 illustrating a method for training a robot de- vice controller.In 701, demonstrations of an execution of a skill by a robot device are provided,wherein each demonstration comprises for each time point of a sequence of time points a state observation (i.e. an observation of a state of the robot device and R.415204 -21 -possibly also of its environment) including sensor data for each sensor modality of multiple sensor modalities. The demonstrations may further (implicitly or ex- plicitly) include a specification of an action for each time point which was taken (i.e. demonstrated) at the time point. An action may be a (relative) movement (with respect to the current state, i.e. pose, or an acceleration etc.). The multiple sensor modalities are at least two, but possible more than two modalities, e.g. three like in the embodiments above (visual, audio, proprioception). For example, the multiple sensor modalities include at least proprioception, include at least vis- ual, or include at least proprioception and visual (and possibly one or more addi- tional modalities like audio). The robot device controller is to be trained to implement a control policy, wherein the control policy generates an action sequence, when the robot device has a certain state at a certain point in time, the robot device controller should provide an action to take. In training, the action selected by the robot device controller may be evaluated by determining whether the robot device has, by taking the se- lected action, reached the next state of the respective demonstration (e.g. whether a movement action was as demonstrated), i.e. from a loss between a state that was reached and a state that was demonstrated. In 702, for each demonstration and each time point, an embedding of the sensordata for each sensor modality is determined (e.g., as in the example above, a vis-ual data embedding, an audio data embedding, and a proprioception data em- bedding). In 703, the robot device controller determines actions from the embeddings. In 704, a loss is determined including ^an imitation loss between the demonstrations and trajectories of the robotdevice caused by the determined actions and ^a latent space regularization loss that punishes differences between em-beddings of sensor data of different modalities for the same time point. This means that for each time point, the embeddings of the sensor data of R.415204 -22 -the modalities should be similar. The more they differ, the higher the (la-tent space regularization) loss term: In the above example, this is the MCL loss which includes the two NCE losses which compare the visual embeddings and the audio embeddings and the proprioception embed- dings with the other two, respectively). In 705, the robot device controller is trained to reduce the loss (i.e. parameters such as machine learning model weights which represent the control policy are adapted, e.g. according to a backpropagation of the loss through neural network layers). According to various embodiments, in other words, a training method is provided which allows considering multimodal, time-series data for robot imitation learning. This allows for learning versatile skills, especially for contract rich robotic manipu- lation. When learning a control policy from demonstrations (observation-action trajectories), data-efficiency is crucial. Therefore, according to various embodi- ments, a self-supervised regularization approach to augment the imitation learn-ing objective is provided. The self-supervised objectives involve aligning differentsensor modalities in latent space, and optionally in addition learning embeddingconsistency across time using order prediction and token masking.Various embodiments may receive and use sensor data from various sensors ac- cording to different sensor modalities. These modalities for example include ^Light, i.e. sensor data from various visual sensors such as color (e.g.RGB) images, LiDAR point clouds etc.^ Sound^ Pressure^ Temperature^ robot proprioception, e.g. data from sensors that monitor joint angles,speeds, torques, and force, end-effector pose etc. R.415204 -23 -Sensors may be understood to have the same sensor modality if they measurethe same form of energy and process it in similar way. “Modality” may be under-stood as the raw input used by the sensors. The robot device controller (and, equivalently, the control policy learned by it) may be used to control a robot device (i.e. to generate a control signal for a robot device). Robot device may be understood as any technical system (with a me- chanical part whose movement is controlled), like e.g. a computer-controlled ma- chine, like a robot, a vehicle, a domestic appliance, a power tool, a manufacturing machine, a personal assistant or an access control system. According to various embodiments, a policy for controlling the technical system may be learnt and then the technical system may be operated accordingly.The method of figure 7 may be performed by one or more data processing de-vices (e.g. computers or microcontrollers) having one or more data processing units. The term "data processing unit" may be understood to mean any type of entity that enables the processing of data or signals. For example, the data or signals may be handled according to at least one (i.e., one or more than one) specific function performed by the data processing unit. A data processing unit may include or be formed from an analogue circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or any combination thereof. Any other means for implement- ing the respective functions described in more detail herein may also be under- stood to include a data processing unit or logic circuitry. One or more of the method steps described in more detail herein may be performed (e.g., imple- mented) by a data processing unit through one or more specific functions per- formed by the data processing unit. Accordingly, according to one embodiment, the method is computer-imple- mented.

Claims

R.415204 -24 -Claims1. A method for training a robot device controller (106), comprising:providing (701) demonstrations of an execution of a skill by a robot de- vice, wherein each demonstration comprises for each time point of a se- quence of time points a state observation including sensor data for eachsensor modality of multiple sensor modalities; determining (702), for each demonstration and each time point, an em-bedding of the sensor data for each sensor modality;determining (703) actions by the robot device controller (106) from theembeddings; determining (704) a loss includingan imitation loss (211) between the demonstrations and trajecto- ries of the robot device (101) caused by the determined actions and including a latent space regularization loss (409, 410) that punishes differ- ences between embeddings of sensor data of different modalities for the same time point; and training (705) the robot device controller (106) to reduce the loss.

2. The method of claim 1, wherein determining the actions comprises de-termining action features from the embeddings and decoding the action features to actions.

3. The method of claim 2, further comprising randomly masking embed-dings and determining action features from the randomly masked em-beddings, wherein the loss further includes a masking loss (512) which punishes differences between action features determined from the em-beddings and action features determined from the randomly masked em-beddings.

4. The method of claim 3, wherein the masking loss (512) is determined bydetermining proprioception sensor data from the action featuresR.415204 -25 -determined from the randomly masked embeddings for one or more timepoints of the sequence of time points and comparing the determined pro- prioception sensor data with proprioception sensor data included in the state observations for the one or more time points.

5. The method of any one of claims 1 to 4, further comprising fusing theembeddings to fused embeddings for each time point, randomly shuffling the fused embeddings over the sequence of time points, determining ac- tion features from the shuffled fused embeddings, and estimating a cor- rect order of the fused embeddings, wherein the loss further comprises a shuffling loss (607) which punishes differences between the estimated order and the true order of the fused embeddings.

6. The method of any one of claims 1 to 5, wherein determining the latentspace regularization loss (409, 410) comprises projecting the embed-dings for at least some of the modalities to a common sensor data space and determining at least a component of the latent space regularization loss (409, 410) as a contrastive loss within the common sensor data space.

7. A method for controlling a robot device (101), comprising training a robotdevice controller (106) according to any one of claims 1 to 6, supplying observed sensor data for each of the multiple sensor modalities, deter- mining an embedding of the observed sensor data for each sensor mo- dality, determining actions by the robot device controller (106) from the embeddings of the observed sensor data and controlling the robot de-vice (101) to perform the actions.

8. A robot device controller (106), configured to perform a method of anyone of claims 1 to 7.R.415204 -26 -9. A computer program comprising instructions which, when executed by acomputer, makes the computer perform a method according to any one of claims 1 to 7.

10. A computer-readable medium comprising instructions which, when exe-cuted by a computer, makes the computer perform a method according to any one of claims 1 to 7.

Images