Method for training a machine learning model to determine body poses and positions of a body with multiple body parts

Abstract

According to various embodiments, a method for training a machine learning model for determining body poses and positions of a body having multiple body parts is provided, comprising, for each training trajectory of a plurality of training trajectories, wherein each training trajectory specifies a position of each body part of the multiple body parts in a global coordinate system for each time point of a respective sequence of time points, transforming the training trajectory into a transformed training trajectory, predicting, by means of the machine learning model, positions of the body parts at one or more time points following the prediction start time point, and determining a loss by comparing the predicted positions with positions of the body parts specified by the training trajectory in the training trajectory for the one or more time points following the prediction start time point.

Description

[0001] The present disclosure relates to methods for training a machine learning model to determine body poses and positions of a body having multiple body parts.

[0002] The ability to predict human movements and recognize human activities is a crucial component for social robots and autonomous systems that operate in human spaces and must interact with humans in domestic and industrial settings. In collaborative manufacturing environments, for example, robots accurately track and predict the future poses and activities of human workers to provide safe and efficient assistance in collaborative assembly tasks. Autonomous vehicles benefit from tracking the status and attention of pedestrians to safely navigate urban streets.Automated systems, including mobile robots, manipulators, and vehicles, therefore increasingly rely on motion prediction for safe navigation, improved human-robot collaboration, human tracking, event observation, and simulation purposes.

[0003] Predicting the movement of humans (or animals) involves predicting a motion trajectory and a pose. Motion trajectory prediction focuses on the coarse prediction of the movement path of dynamic entities. Pose prediction (or whole-body pose prediction), on the other hand, deals with the fine-grained prediction of the 3D skeletal joints of humans relative to a fixed reference point on the body, usually the pelvis or torso.

[0004] For both, accurate and efficient methods (in terms of memory requirements, (training) data efficiency and computational effort) are desirable.

[0005] According to various embodiments, a method for training a machine learning model for determining body poses and positions of a body having multiple body parts is provided, comprising: • For each training trajectory of a plurality of training trajectories, each training trajectory specifying a position of each body part of the plurality of body parts in a global coordinate system for each time point of a respective sequence of time points, ◯ Transforming the training trajectory into a transformed training trajectory such that, for a point in time of the sequence of points in time defined as the prediction start time, the position of a given reference body part corresponds to a given point in the global coordinate system and the direction of movement of the reference body part corresponds to a given direction in the global coordinate system; ◯ Predicting, by means of the machine learning model, positions of the body parts at one or more points in time following the prediction start time by feeding the positions of the body parts specified by the transformed training trajectory up to the prediction start time to the machine learning model; and ◯ Determining a loss by comparing the predicted positions with positions of the body parts specified in the training trajectory for the one or more time points following the prediction start time; and • Adjust the machine learning model to reduce an overall loss that includes the identified losses.

[0006] The method described above enables joint prediction of motion trajectory and pose with balanced accuracy for motion trajectory and pose prediction in scenarios that are most important for a mobile robot. Furthermore, it enables prediction with high computational efficiency, making its use in real time feasible, particularly for human-centered motion planning for mobile robots, e.g., autonomous vehicles and intralogistics robots.

[0007] Prediction can be used as part of a pipeline for (particularly mobile) autonomous systems and / or for stationary manipulators and co-production robots: detection, tracking, prediction, planning, and control. An autonomous system navigating shared environments is thus able to detect and track other dynamic agents, plan its own navigation trajectory, and execute it through a series of control actions. For example, the prediction module provides information for tracking humans and thus for trajectory planning and control actions.

[0008] According to various embodiments, a graph attention-based transformer model (i.e., a neural network with a transformer architecture) with input data transformation is provided, enabling the prediction of motion trajectories and poses in the form of 3D positions of a set of body parts in a global coordinate system. This enables real-time predictions for robotic applications requiring immediate responses.

[0009] The method enables the prediction of posture dynamics and movement trajectories for different walking styles, including running, accelerating, decelerating and turning, i.e. the types of movements important for socially acceptable robot navigation.

[0010] Various examples of implementation are given below.

[0011] Embodiment 1 is a method as described above.

[0012] Embodiment 2 is the method according to embodiment 1, wherein the machine learning model comprises a graph attention network by means of which the positions of the body parts specified by the transformed training trajectory up to the prediction start time are processed by representing a respective pose as a graph for each of the times up to the prediction start time, by assigning a node with the respective specified position as node features to each of the body parts and by connecting nodes that are assigned to connected body parts (e.g. according to a skeleton model) by means of an edge, and the graphs are processed by means of the graph attention network.

[0013] In this way, connections between body parts (e.g. connections through bones) are effectively taken into account.

[0014] Embodiment 3 is the method according to embodiment 1 or 2, wherein the machine learning model has a transformer architecture.

[0015] A transformer enables effective consideration of temporal sequences, as required when predicting poses based on a sequence of previous poses. The use of attention mechanisms also allows for the consideration of semantic relationships between different poses (e.g., looking left and right before crossing a street: looking back may indicate that the person will move forward afterward).

[0016] Embodiment 4 is the method according to one of the embodiments 1 to 3, comprising determining spatial-temporal codings of the positions specified by the transformed training trajectory up to the prediction start time and supplying the spatial-temporal codings together with the positions of the body parts specified by the transformed training trajectory up to the prediction start time to the machine learning model.

[0017] The combination of spatial position encoding (as typically provided by the use of a transformer, e.g., for processing text) with temporal encoding allows the machine learning model to take into account the pose changes in the input sequence, which improves the prediction.

[0018] Embodiment 5 is the method according to one of the embodiments 1 to 4, wherein the loss contains a first loss component which, for each of the one or more points in time following the prediction start time and each of the body parts, contains the difference between the predicted position of the body part and the position of the body part specified by the training trajectory or transformed training trajectory as a loss contribution and / or wherein the one or more points in time following the prediction start time have one or more pairs of consecutive points in time and the loss contains a second loss component,which, for each of the one or more pairs and each of the body parts, contains the difference between the difference in the positions predicted for the time points of the pair and the difference in the positions of the body part specified for the time points of the pair by the training trajectory or transformed training trajectory as a loss contribution.

[0019] In particular, a combined loss can be used that includes both a pose loss (first loss component) and a trajectory loss (second loss component), so that the machine learning model predicts both good absolute and relative (i.e., consistent trajectories).

[0020] Embodiment 6 is a method for predicting one or more body poses and one or more positions of a body having multiple body parts, comprising: • Training a machine learning model according to one of the embodiments 1 to 5; • Acquiring a trajectory of the body that indicates, for each acquisition time of a sequence of acquisition times, a position of each body part of the plurality of body parts in the global coordinate system; • Transforming the acquired trajectory into a transformed acquired trajectory such that, for the last of the acquisition times, the position of the specified reference body part corresponds to the specified point of the global coordinate system and the direction of movement of the reference body part corresponds to a specified direction in the global coordinate system; and • Predicting, using the machine learning model, positions of the body parts by feeding the positions of the body parts specified by the transformed acquired trajectory to the machine learning model.

[0021] Embodiment 7 is a method according to Embodiment 6, further comprising controlling a robot device depending on the predicted positions of the body parts (ie, the method in this case is a method for controlling a robot device).

[0022] Embodiment 8 is a data processing system (in particular control device) which is configured to carry out a method according to one of the embodiments 1 to 7.

[0023] Embodiment 9 is a computer program including instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 7.

[0024] Embodiment 10 is a computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method according to any one of embodiments 1 to 7.

[0025] In the drawings, like reference characters generally refer to the same parts throughout the several views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings. Fig. 1 shows a vehicle. Fig. 2 illustrates pose prediction and motion trajectory prediction. Fig. 3 illustrates an input data transformation according to one embodiment. Fig. 4 shows a machine learning model for pose trajectory prediction according to one embodiment. Fig. 5 shows a flowchart illustrating a method for training a machine learning model to determine body poses and positions of a body having multiple body parts according to one embodiment.

[0026] The following detailed description refers to the accompanying drawings, which, by way of illustration, show specific details and aspects of this disclosure in which the invention may be practiced. Other aspects may be utilized, and structural, logical, and electrical changes may be made without departing from the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure may be combined with one or more other aspects of this disclosure to form new aspects.

[0027] Various examples are described in more detail below.

[0028] Fig. 1 shows a vehicle 101.

[0029] In the example of Fig. 1, a vehicle 101, for example a car or truck, is provided with a vehicle control device (also referred to as an electronic control unit, e.g. a control device, e.g. an Electronic Control Unit (ECU)) 102.

[0030] The vehicle control device 102 has data processing components, e.g., a processor (e.g., a CPU (central processing unit)) 103 and a memory 104 for storing control software 107 according to which the vehicle control device 102 operates, and data processed by the processor 103. The processor 103 executes the control software 107.

[0031] For example, the stored control software (computer program) includes instructions which, when executed by the processor, cause the processor 103 to perform driver assistance functions (i.e., the function of an ADAS (Advanced Driver Assistance System)) or even to control the vehicle autonomously (AD (Autonomous Driving)).

[0032] The control software 107 is transmitted, for example, from a computer system 105, e.g., via a communications network 106 (or also with the aid of a storage medium such as a memory card), to the vehicle 101. This can also occur during operation (or at least when the vehicle 101 is with the user), since the control software 107 is updated to new versions over time, for example.

[0033] The control software 107 determines control actions for the vehicle (such as steering actions, braking actions, etc.) from input data available to it that contain information about the environment or from which it derives information about the environment (such as by detecting other road users, e.g., other vehicles, pedestrians, cyclists, etc.). These are, for example, sensor data from one or more sensor devices 109, such as a camera of the vehicle 101, which are connected to the vehicle control device 102 via a communication system 110 (e.g., a vehicle bus system such as CAN (Controller Area Network)).

[0034] The control software 107 can be trained at least partially, for example, using machine learning (ML). This means that the control software 107 implements, for example, a machine learning model 108 (e.g., a neural network (NN)) that is trained based on training data, in this example, from the computer system 105. The computer system 105 thus implements an ML training algorithm for training one (or more) ML models 108.

[0035] For example, the ML model 108 (e.g., a neural network) is an ML model for predicting the behavior of other road users, such as pedestrians. This includes, for example, the prediction of movement trajectories (e.g., a sequence of 2D positions for a walking person) and the prediction of poses (e.g., the prediction of the positions of the whole-body joints relative to a given "central" body part or joint, e.g., the pelvis or hip). The latter is of interest because, for example, the direction in which a pedestrian's head is turned provides information about the direction they intend to walk. For example, the typical looking left and right can be an indication that a pedestrian intends to cross a street.

[0036] Both motion trajectory prediction and pose prediction are also of interest for other use cases, such as other mobile robots. In collaborative manufacturing environments, for example, it is desirable for robots to accurately predict the future positions and activities (e.g., arm movements) of human workers in order to provide safe and efficient support in collaborative assembly tasks.

[0037] Fig. 2 illustrates a prediction of poses 201 and, in a diagram 202, a motion trajectory prediction (ground truth and prediction, respectively, starting from the prediction start time 0s).

[0038] Motion trajectory prediction and pose prediction can be treated as separate problems, but then two machine learning models are required. For example, motion trajectories are expressed as displacements [dx,dy] (or velocities) of a 2D position above the ground plane. This encoding of the 2D motion trajectory provides a similar representation of different motion trajectories regardless of their absolute (x,y) coordinates, enabling generalization by the machine learning model for motion trajectory prediction. For pose prediction, in addition to the 2D position coordinates, 3D joint coordinates (or body part coordinates) are encoded relative to the 2D position.

[0039] Decoupled (separate) pose and motion trajectory prediction modules can also be combined at the end of a prediction pipeline to achieve simultaneous pose and motion trajectory prediction. However, this also results in a large (and thus memory-intensive) and computationally intensive machine learning model (e.g., neural network) that performs suboptimally due to the separate treatment of walking dynamics and body motion dynamics, which are in reality tightly coupled.

[0040] Therefore, according to various embodiments, a coupled approach (for a joint prediction of movement trajectory and pose) is provided in which training is carried out using an (input data) transformation in global coordinates, ie the entire sequence of joint coordinates of the 3D skeleton is expressed with respect to a single reference point (e.g. the position of the pelvic joint at the "prediction time", ie at the time of the last state of the input state sequence on the basis of which prediction is made).

[0041] Joint prediction can approximately halve the number of parameters of the machine learning model compared to an approach where there are two prediction modules whose results are merged as described above, and achieve faster inference time and more accurate prediction of the motion trajectory (which is the critical part of the problem when dealing with a mobile robot or an autonomous vehicle).

[0042] According to various embodiments, the coupled prediction of the movement trajectory and pose (e.g., of a walking person) is performed using a machine learning model (e.g., corresponding to the machine learning model 108) having a transformer architecture. 3D body part positions (in a global coordinate system) are used directly, which are derived, for example, from a 3D pose estimation pipeline for humans. Thus, the position of a reference body part (e.g., the hip) is not treated separately from the positions of other body parts, but rather all body part positions are used together as input. The input for the machine learning model is thus a (pose) trajectory in the form of an (input) sequence of states (or "frames") for a sequence of times up to the prediction start time, where each state represents the 3D position for a predetermined set of body parts (e.g.,Joint positions or limb center positions, etc.). The term "pose trajectory" (hereinafter also simply referred to as "trajectory") is used here for a sequence of poses. In contrast, "motion trajectory" refers only to the sequence of positions of the respective body (e.g., the position of a central body part such as the hip), but not the positions of multiple body parts in the global coordinate system (i.e., reference frame).

[0043] According to various embodiments, this input (i.e., a pose trajectory) is transformed using an (input data) transformation to couple pose and trajectory prediction streams in a unified architecture, thus simplifying the machine learning model for simultaneous pose and trajectory prediction as a coupled task. For training trajectories that, in addition to the input sequence of states (i.e., the past trajectory), also contain states for one or more additional points in time (after the prediction start time, i.e., a future trajectory) as ground truth, these states are also transformed for training, or the machine learning model's prediction is back-transformed before being compared with the ground truth for the loss calculation.

[0044] Fig. 3 illustrates the input data transformation according to one embodiment for four training trajectories 301, 302, 303, 304.

[0045] The four training trajectories 301-304 are shown in a first diagram 305 with global coordinates and, for simplicity, as a sequence of 2D positions. The dotted part is the past trajectory, and the solid part is the future trajectory (shown here for illustration but omitted for inference).

[0046] The transformation involves aligning the training trajectories 301-304 at the origin (X=0, Y=0) and the positive X-axis direction, as shown in more detail for a trajectory in the second diagram 306: The trajectory 307 is transformed into a transformed trajectory 308 such that the prediction start time is set to the origin and the direction of movement at the prediction start time corresponds to the X-axis direction.

[0047] The input data transformation serves to convert input trajectories (especially training trajectories) into a common reference frame (especially for learning) using rotation and translation. The result is transformed input trajectories 309. This enables generalization beyond the various training trajectories in global coordinates and training with absolute 3D coordinates without decomposition into separate pose and trajectory prediction data streams.

[0048] In addition to the input data transformation, according to various embodiments, a graph attention network (GAT) is used to generate graph embeddings that capture the spatial skeletal structure, thus informing the machine learning model about the skeletal hierarchy and the relative dependencies between individual joints. To this end, a GAT encoder is applied to the states of the input trajectory using the skeleton's adjacency matrix, resulting in a more accurate and realistic prediction of body dynamics.

[0049] Fig. 4 shows a machine learning model 400 for pose trajectory prediction (ie, joint prediction of motion trajectory and pose) according to one embodiment.

[0050] In the following, P(t) ∈ ℝ 3Nthe body pose (e.g., human pose) at time t, which includes N three-dimensional body part positions (e.g., joint positions): P(t) = {j1(t), j2(t), ... , jN(t)}, where each j i (t) ∈ ℝ 3 represents the three spatial coordinates (x, y, z) of the i-th body part positions (e.g., the i-th joint) in the robot's coordinate system at time t. An input trajectory (input sequence of states) is a sequence of poses from time 0 to the prediction start time T1: S∈ℝT1×3N, which all refer to the same person: S={P(0),P(1),...,P(T1)}

[0051] The goal of the machine learning model 400 is to predict a sequence of poses from time T1+1 to time T1+T2 with global translation Sout∈ℝ(T2−T1)×3N: Sout={P(T1+1),...,P(T1+T2)}∪{T(T1+1),...,T(T1+T2)}

[0052] As described above, for an input trajectory S∈ℝT1×3N first an input data transformation 401.

[0053] This serves to normalize the input trajectories to a common reference frame in order to generalize and predict human movement in global coordinates across different movement directions. As discussed with reference to Fig. 3, global invariance is achieved by ensuring that all predictions start from a consistent origin by performing a translation using the vector v, which is derived as the negative counterpart of the position of a reference body part (with index r (for “root”), e.g. hip) of the last state of the input sequence, i.e. v = -j r (T1). The translation with the vector v is applied to each state of the input sequence S applied, resulting in a sequence S' For each state (frame) of S is the corresponding shifted pose in S' given by: P'(t)=P(t)+v ∀t∈[0,T]

[0054] This shift anchors the human pose of the last state of the input sequence to the origin, providing a consistent starting point for subsequent motion predictions.

[0055] Furthermore, orientation invariance is established by the input data transformation 401 by canonically aligning the direction of movement along the positive x-axis. To this end, a rotation angle θ is calculated based on the direction of movement of the input trajectory (at the prediction start time) and the positive x-axis. This angle is calculated as the arctangent of the ratio of the differences between the y- and x-coordinates of the position of the reference body part of the last state of the input sequence T1 and the position of the reference body part at a previous time (T1-w), where w (for "window") is a predetermined time interval: θ=arctan 2(Δy,Δx){Δx=jr(T1)x−jroot(T1−w)xΔy=jr(T1)y−jroot(T1−w)y

[0056] Finally, a corresponding rotation matrix for a rotation around the z-axis is applied to the shifted sequence S', resulting in the rotated sequence S'': P''(t)=[cos(−θ)−sin(−θ)0sin(−θ)cos(−θ)0001]⋅P'(t) ∀t∈[0,T]

[0057] The pose sequence S'' is the result of the input data transformation 401.

[0058] A GAT (Graph Attention Network) 402 now generates, as briefly described above, spatial graph embeddings from the pose sequence S''. Each human pose can be represented as a graph 403, where each joint in the pose corresponds to a node and the connections between them (bones or body parts) are the edges. The input to the GAT 402 is a redesigned version of the pose sequence S'': Each joint (or more generally each body part) is a node in the graph 403 with three spatial features j i (t) ∈ ℝ 3(given by the position of the body part). The edges E are determined based on the kinematic chain of the body skeleton used, which can vary depending on the dataset. From this kinematic chain, an adjacency matrix A is derived such that A ij = 1 if there is a connection between the joints j i (t) and j j (t) exists, and otherwise 0. The GAT 402 calculates attention scores e between pairs of joints, which capture the importance of one joint compared to another: eij=LeakyReLU(aT[Wji(t)‖Wjj(t)]) where || denotes the concatenation and a and W are learnable parameters (vector and matrix, respectively). Using normalized attention coefficients, the joint features are updated by aggregating the information from neighboring joints, resulting in the joint embeddings.

[0059] The GAT 402 thus creates a new set of node characteristics χ∈ℝT1×N×Jdim as output and thus generates common embeddings for all poses of the input sequence.

[0060] Subsequently, the joint embeddings of the input sequence are flattened to obtain a total pose embedding of the input sequence in ℝ T1× (N × Jdim) The GAT 402 serves to facilitate attention mechanisms between body parts within a frame and effectively capture spatial relationships. The output of the GAT 402 is fed to a transformer (network) 404. The transformer is designed to detect and learn the temporal relationships between poses to ensure a comprehensive understanding of the spatial and temporal dynamics in the data.

[0061] In addition to the embeddings generated by the GAT 402, a further (spatial-temporal) position encoding 405 is used to capture human dynamics in detail.

[0062] To do this, sinusoidal spatial position encodings are generated that are specifically tailored to distinguish between the different joints in each image. For each joint, this approach generates a position encoding of dimension J dim , which takes all N joints into account. Furthermore, temporal codes are generated to capture the temporal progression of sequences. These temporal codes have dimensions J dim × N, consider all frames and highlight the sequential dynamics from one frame to the next. The spatial position encodings are flattened and then merged with the temporal encodings to obtain a unified 2D position encoding 405.

[0063] The transformer 404 has a classic transformer architecture with an encoder 406 and a decoder 407. The encoder 406 receives the output of the GAT 205 together with the (spatial-temporal) position encoding 405 and processes it through L layers 408. Each layer contains a self-attention block (masked single-headed self-attention with RPP (Relative Position Representations)) 409 and a feed-forward network 411 (each followed by a respective addition and normalization operation 410, 412). The layers 408 process the input to a sequence of elements in a latent space Z = [z1, z2, ..., z T ].

[0064] The decoder 407 then uses this sequence of latent space elements to generate an output position sequence 413. It contains several layers 414, each containing a self-attention attention block (Masked Single-Headed Self-Attention with RPP) 415, a multi-headed cross-attention attention block 417, and a feed-forward network 419 (again, each followed by a respective addition and normalization operation 416, 418, 420), followed by a final multi-headed attention block 421 followed by a final feed-forward network 422.

[0065] To ensure that a predicted pose depends only on previous poses (i.e., up to T1) and not on future poses, a "casual masked self-attention" is used for both the encoder 406 and the decoder 407. The core principle of masked self-attention is that during the calculation of the attention weights, certain weights are set to an extremely negative value, e.g., 10, by using a mask. -9 , are set. This mask is designed so that, for a given sequence position, all future positions in the sequence are marked as irrelevant. When the softmax function is subsequently applied to these weights during the attention mechanism, the values ​​corresponding to the masked positions become negligible.

[0066] In self-attention with relative position representation, attention values ​​for a given sequence position are weighted more heavily toward the immediately adjacent poses. This is advantageous for sequences with human poses, where not only the order of the positions is important, but also the relative transition between the images. This attention to relative distances can lead to a better understanding of human movement patterns.

[0067] The decoder's self-attention queries are initialized with x(T1) (as query 423), i.e., with the last input state, which is repeated several times according to the number of desired output poses.

[0068] After the decoding phase by the layers 414, the final multi-headed attention block 421 causes a multi-headed attention mechanism, also called end attention, where the output of the layers 414 is used as a query and the output of the GAT 402 is taken into account, which serves as both the key and the value.

[0069] Subsequently, the embeddings output by the multi-headed attention block 421 are propagated through the linear layers of the final feed-forward network 422 and, with the help of v and θ, are returned to the original motion orientation and the global coordinate space by means of a back transformation 424, resulting in the predicted 3D poses and the corresponding trajectories, i.e., the initial position sequence: Y^={y^T1+1,y^T1+2,…,y^T1+T2}

[0070] The machine learning model 400 can be trained with a combined pose and trajectory loss. Since each pose has a dimensionality of 3N (where N is the number of body parts), the predicted pose sequence is a sequence of vectors y^T1+1,y^T1+2,…,y^T1+T2 and the ground truth position sequence is a sequence of vectors yT1+1,yT1+2,…,yT1+T2. If L1 denotes the pose loss and L2 the trajectory loss, the combined loss (for a training trajectory) is L = L1 + L2 with L1=13N(T2−T1−1)∑t=T1+1T1+T2‖y^t−yt‖2 and L2=13N(T2−T1)∑t=T1+1T1+T2−1‖(y^t+1−y^t)−(yt+1−yt)‖2

[0071] Across a batch of training trajectories, this combined loss can be aggregated (or averaged) to an overall (batch) loss, and the parameters (weights, etc.) of the machine learning model can be adjusted towards decreasing overall loss.

[0072] In summary, according to various embodiments, a method is provided as described in Fig. 5 shown.

[0073] Fig. 5 shows a flowchart 500 illustrating a method for training a machine learning model to determine body poses and positions of a body having multiple body parts (e.g., a person or an animal, but also, e.g., a mechanical system having one or more degrees of freedom with respect to the relative position of components) according to one embodiment.

[0074] In 501, for each training trajectory of a plurality of training trajectories, wherein each training trajectory specifies a position of each body part (e.g., joint or connecting link between two joints) of the plurality of body parts in a global coordinate system for each time point of a respective sequence of time points, • in 502, the training trajectory is transformed into a transformed training trajectory such that, for a point in time of the sequence of points in time defined as the prediction start time, the position of a predetermined reference body part corresponds to a predetermined point in the global coordinate system (e.g., origin) and the direction of movement of the reference body part corresponds to a predetermined direction in the global coordinate system; • in 503, positions of the body parts at one or more points in time following the prediction start time are predicted (ie determined) by means of the machine learning model by feeding the positions of the body parts specified by the transformed training trajectory up to the prediction start time to the machine learning model; • in 504, a (single) loss is determined by comparing the predicted positions with positions of the body parts specified in the training trajectory for the one or more time points following the prediction start time point, from the training trajectory.

[0075] In 505, the machine learning model is adapted to reduce an overall loss that includes the determined losses (ie, the (trainable) parameters (e.g., weights) are adjusted in the direction of a decreasing overall loss that includes the individual losses).

[0076] The procedure of Fig.5 may be performed by one or more computers having one or more data processing units. The term "data processing unit" may be understood as any type of entity that enables the processing of data or signals. The data or signals may, for example, be handled according to at least one (i.e., one or more than one) specific function performed by the data processing unit. A data processing unit may include or be formed from an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array (FPGA) integrated circuit, or any combination thereof.Any other way of implementing the respective functions described in more detail herein may also be understood as a data processing unit or logic circuit arrangement. One or more of the method steps described in detail herein may be performed (e.g., implemented) by a data processing unit through one or more specific functions performed by the data processing unit.

[0077] According to various embodiments, the method is therefore particularly computer-implemented.

[0078] After training, the machine learning model can be used to generate a control signal for a robotic device by feeding it sensor data regarding its environment or derived (past) pose trajectories (e.g., of people), thereby generating predictions for one or more poses (in the global coordinate system). The term "robotic device" can be understood as referring to any technical system (with a mechanical part whose movement is controlled), such as a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant, or an access control system.

[0079] Various embodiments may receive time series of sensor data from various sensors such as video, radar, LiDAR, ultrasound, motion, thermal imaging, etc., and use it to determine past poses.

Claims

[1] A method for training a machine learning model (108, 400) for determining body poses and positions of a body having multiple body parts, comprising: For each training trajectory (307) of several training trajectories, wherein each training trajectory specifies, for each time point of a respective sequence of time points, a position of each body part of the plurality of body parts in a global coordinate system, Transforming (502) the training trajectory (307) into a transformed training trajectory (308) such that, for a point in time of the sequence of points in time defined as the prediction start time, the position of a predetermined reference body part corresponds to a predetermined point in the global coordinate system and the direction of movement of the reference body part corresponds to a predetermined direction in the global coordinate system; Predicting (503), by means of the machine learning model (108, 400), positions of the body parts at one or more points in time following the prediction start time, by feeding the positions of the body parts specified by the transformed training trajectory (308) up to the prediction start time to the machine learning model (108, 400); and determining (504) a loss by comparing the predicted positions with positions of the body parts specified by the training trajectory (307) in the training trajectory (307) for the one or more points in time following the prediction start time; and Adapting (505) the machine learning model (108, 400) to reduce a total loss including the determined losses. [2] The method according to claim 1, wherein the machine learning model (108, 400) comprises a graph attention network (402) by means of which the positions of the body parts specified by the transformed training trajectory (308) up to the prediction start time are processed by, for each of the times up to the prediction start time, representing a respective pose as a graph (403), by assigning each of the body parts a node with the respective specified position as node features and connecting nodes assigned to connected body parts by means of an edge, and processing the graphs (403) by means of the graph attention network (402). [3] The method of claim 1 or 2, wherein the machine learning model (108, 400) comprises a transformer architecture. [4] Method according to one of claims 1 to 3, comprising determining spatial-temporal codings (405) of the positions specified by the transformed training trajectory (308) up to the prediction start time and supplying the spatial-temporal codings (405) together with the positions of the body parts specified by the transformed training trajectory (308) up to the prediction start time to the machine learning model (108, 400). [5] Method according to one of claims 1 to 4, wherein the loss contains a first loss component which, for each of the one or more points in time following the prediction start time and each of the body parts, contains the difference between the predicted position of the body part and the position of the body part indicated by the training trajectory or transformed training trajectory as a loss contribution and / or wherein the one or more points in time following the prediction start time have one or more pairs of consecutive points in time and the loss contains a second loss component,which, for each of the one or more pairs and each of the body parts, contains the difference between the difference in the positions predicted for the time points of the pair and the difference in the positions of the body part specified for the time points of the pair by the training trajectory or transformed training trajectory as a loss contribution. [6] A method for predicting one or more body poses and one or more positions of a body having multiple body parts, comprising: Training a machine learning model (108, 400) according to one of claims 1 to 5; Acquiring a trajectory of the body that indicates, for each acquisition time of a sequence of acquisition times, a position of each body part of the plurality of body parts in the global coordinate system; Transforming the acquired trajectory into a transformed acquired trajectory such that, for the last of the acquisition times, the position of the specified reference body part corresponds to the specified point of the global coordinate system and the direction of movement of the reference body part corresponds to a specified direction in the global coordinate system; and Predicting, by means of the machine learning model (108, 400), positions of the body parts by feeding the positions of the body parts indicated by the transformed detected trajectory to the machine learning model (108, 400). [7] The method of claim 6, further comprising controlling a robot device (101) depending on the predicted positions of the body parts. [8] Data processing system (102, 105) arranged to carry out a method according to one of claims 1 to 7. [9] A computer program comprising instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 7. [10] A computer-readable medium storing instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 7.

Images