Computer-implemented method and system for hand pose generation in hand-object interaction (HOI)

Abstract

A computer-implemented method for hand pose generation in hand-object interaction (HOI) is described. In an embodiment, the computer-implemented method comprises: receiving a two-dimensional (2D) image of an object and an input mask adapted to specify at least a location for HOI with the object; processing, using an encoder, the 2D image and the input mask to generate encoder outputs; and generating, using a decoder, a two-dimensional (2D) hand pose based on the encoder outputs, the 2D hand pose being represented by points relating to joints of the 2D hand pose. A system for hand pose generation in HOI is also described in an embodiment.

Description

[0001] Computer-implemented method and system for hand pose generation in handobject interaction (HOI)

[0002] Technical Field

[0003] The present disclosure relates to a computer-implemented method and system for hand pose generation in hand-object interaction (HOI).

[0004] Hand pose estimation is crucial in computer vision and has been extensively studied in recent years. Traditional hand pose estimation techniques use depth sensors or multiview cameras to estimate 3D hand poses. While hand pose estimation focuses on inferring poses from input data, hand pose generation aims to synthesize plausible hand poses, often conditioned on specific constraints or contexts. Early approaches employed physics-based simulations and optimisation techniques to generate hand poses that satisfy kinematic constraints and avoid collisions. More recent approaches leverage machine learning techniques, including the use of a regression forest to map haptic controller inputs to hand poses for animation. Existing work also attempted to generate synthetic hand posture and motion datasets by applying constraints of the human hand, enabling data augmentation for hand pose estimation models.

[0005] Another approach for hand pose generation is to consider controllable affordance Hand- Object Interaction (HOI) generation, which is becoming an increasingly important area of research in computer vision. In HOI, a hand grasp is a crucial step for effectively controlling a geometry of the hand. Present approaches typically rely on depth sensors, motion capture systems, or 3D hand models. However, depth sensors can be affected by occlusions, lighting conditions, and surface properties, while 3D hand models may not accurately capture the nuances of individual hand shapes and poses.

[0006] Besides generating hand-object interactions, other works focus on reconstructing them. However, as hands and objects naturally block each other, the reconstruction process becomes highly challenging during interactions. To address this problem, existing work simplifies the problem to six-dimensional (6D) pose estimation, assuming the templates for both the object and hand are known as prior knowledge. Nonetheless, these methods require reconstructing hand poses from hand-object interaction images, which may not be accurate or not efficient given the requirements of a 6D pose estimation. Still further, existing metrics for evaluating the generated hand poses, such as the Frechet Inception Distance (FID) method and the Maximum Mean Discrepancy (MMD) method, are biased and inefficient for evaluating the generated hand poses. This inadvertently affects an improvement of accuracy of a hand pose generation method or system.

[0007] It is therefore desirable to provide a computer-implemented method and system for hand pose generation in hand-object interaction (HOI) which address the aforementioned problems and / or provide a useful alternative. Further, other desirable features and characteristics will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and this background of the disclosure.

[0008] Aspects of the present application relate to a computer-implemented method and system for hand pose generation in hand-object interaction (HOI).

[0009] In accordance with a first aspect, there is provided a computer-implemented method for hand pose generation in hand-object interaction (HOI). The computer-implemented method comprising: receiving a two-dimensional (2D) image of an object and an input mask adapted to specify at least a location for HOI with the object; processing, using an encoder, the 2D image and the input mask to generate encoder outputs; and generating, using a decoder, a two-dimensional (2D) hand pose based on the encoder outputs, the 2D hand pose being represented by points relating to joints of the 2D hand pose.

[0010] By providing a 2D image of an object and an input mask adapted to specify at least a location for HOI with the object, the present method allows generation of a 2D hand pose at a user-specified location for affordance hand object interaction. This allows the method to be better adapted at capturing the natural variations and subtleties of human hand poses, leading to more realistic and diverse hand pose generations. The ability of the present method to generate hand poses at user-specified locations by using only a 2D image of the object of interest also avoids the need to reconstruct hand poses from handobject interaction images, thereby improving accuracy and computing efficiency.

[0011] The input mask may include a lollipop-shaped mask having parameters including a palm size, a location and an approaching direction. The method may further comprise: receiving, at the decoder, positional encoding for providing spatial relationships between the joints, wherein the positional encoding includes learnable positional encoding defined using a learnable parameter vector p, associated with an 7-th joint’s position in the form of coordinates (x,, y). The 2D hand pose may include twenty-one joints. The decoder may include a transformer-based cross attention decoder. Learnable positional encodings allow the present encoder-decoder based model to optimise positional information during training and enables the encoderdecoder based model to better capture spatial dependencies and interactions between the joints.

[0012] The method may further comprise: incorporating one or more additional cross attention layers in the transformer-based cross attention decoder. By incorporating the one or more additional cross attention layers, improvement in the quality of the generated hand poses having a reduced pose - FID score can be achieved.

[0013] The encoder and the decoder may form an encoder-decoder based model and the computer-implemented method may comprise training the encoder-decoder based model using a Denoising Diffusion Probabilistic Model (DDPM) diffusion process.

[0014] Wherein training the encoder-decoder based model using the DDPM diffusion process may comprise: calculating a diffusion loss, ng a diffusion loss function given by: Ldiffusion here x0relates to original data, xtrelates to noisy data at time step t, e represents noise and network eerelates to prediction of noise at each time step.

[0015] Wherein training the encoder-decoder based model using the DDPM diffusion process may comprise: calculating a pose reconstruction loss, Lpose, using a pose reconstruction loss function given by where y represents ground truth 2D hand poses and y represents predicted 2D hand poses; and minimising an overall loss function, L, using: where A is a hyperparameter adapted to balance the diffusion loss and the pose reconstruction loss.

[0016] The computer-implemented method may comprise: generating a first population and a second population of 2D hand poses; and calculating a pose-FID score to capture a similarity between the first population and the second population of 2D hand poses, wherein calculating the pose-FID score comprises: calculating pairwise distances between each points to generate a descriptor vector for each 2D hand pose of the first population and the second population of 2D hand poses, a pairwise distance dtjbetween point / and point / is defined are positions of points / and j, respectively; calculating a mean, and a covariance, <J1, of descriptor vectors of the first population of 2D hand poses and a mean, p2, and a covariance, rr2, of descriptor vectors of the second population of 2D hand poses; and calculating the pose-FID score as: pose

[0017] In accordance with a second aspect, there is provided a computer-implemented method for evaluating a similarity between a first population and a second population of hand poses, where a hand pose is represented by points relating to joints of the hand pose. The computer-implemented method comprising: calculating a pose-FID score to capture the similarity between the first population and the second population of hand poses, wherein calculating the pose-FID score comprising: calculating pairwise distances between each points to generate a descriptor vector for each hand pose of the first population and the second population of hand poses, a pairwise distance dy between point / and point / is defined as: dtj= | | pt- py ||, where ptand pj are positions of points / and / , respectively; calculating a mean, p1, and a covariance, ar, of descriptor vectors of the first population of hand poses and a mean, g2, and a covariance, a2, of descriptor vectors of the second population of hand poses; and calculating the pose-FID score as: pose -

[0018] By calculating and obtaining the pose-FID score, the quality of generated 2D and / or 3D hand poses can be evaluated. The present evaluation method therefore provides versatility for evaluating the quality of generated hand poses for various applications, such as robotic manipulation training and interactive training simulations. The present evaluation method also evaluates the generated hand poses more holistically by considering both a central tendency (i.e. using the mean) and a spread (i.e. using the covariance) of the joint positions within the populations. The computer-implemented method for evaluating similarity using the pose-FID score therefore provides a comprehensive measure of the quality and realism of the generated hand poses.

[0019] In accordance with a third aspect, there is provided a computer readable medium storing processor executable instructions which when executed on a processor cause the processor to carry out any of the preceding method. In accordance with a fourth aspect, there is provided a system for hand pose generation in hand-object interaction (HOI), the system comprises a processor and a data storage storing computer program instructions operable to cause the processor to: receive a two- dimensional (2D) image of an object and an input mask adapted to specify at least a location for HOI with the object; process, using an encoder, the 2D image and the input mask to generate encoder outputs; and generate, using a decoder, a two-dimensional (2D) hand pose based on the encoder outputs, the 2D hand pose being represented by points relating to joints of the 2D hand pose.

[0020] The input mask may include a lollipop-shaped mask having parameters including a palm size, a location and an approaching direction.

[0021] The data storage may store computer program instructions operable to cause the processor to: receive, at the decoder, positional encoding for providing spatial relationships between the joints, wherein the positional encoding includes learnable positional encoding being defined using a learnable parameter vector p, associated with an / -th joint’s position in the form of coordinates (x,, yi). The decoder may include a transformer-based cross attention decoder.

[0022] The encoder and the decoder may form an encoder-decoder based model and wherein the data storage may store computer program instructions operable to cause the processor to: train the encoder-decoder based model using a Denoising Diffusion Probabilistic Model (DDPM) diffusion process.

[0023] The data storage may store computer program instructions operable to cause the processor to: calculate a diffusion loss, Lditfusion, using a diffusion loss function given by ] where x0relates to original data, xtrelates to noisy data at time step t, € represents noise and network egrelates to prediction of noise at each time step.

[0024] The data storage may store computer program instructions operable to cause the processor to: calculate a pose reconstruction loss, Lpose, using a pose reconstruction loss function given by: Lpose - y| |2] where y represents ground truth 2D hand poses and y represents predicted 2D hand poses; and minimise an overall loss function, L, using: L = Ldi^usion+ Lposewhere A is a hyperparameter adapted to balance the diffusion loss and the pose reconstruction loss. The data storage may store computer program instructions operable to cause the processor to: generate a first population and a second population of 2D hand poses; calculate a pose-FID score to capture a similarity between the first population and the second population of 2D hand poses, wherein to calculate the pose-FID score comprises to: calculate pairwise distances between each points to generate a descriptor vector for each 2D hand pose of the first population and the second population of 2D hand poses, a pairwise distance between point / and point y is defined as: dtj = | | pt - p7|| where Pi and Pj are positions of points / and j, respectively; calculate a mean, and a covariance, a1, of descriptor vectors of the first population of 2D hand poses and a mean, p2, and a covariance, cr2, of descriptor vectors of the second population of 2D hand poses; and calculate the pose-FID score as: pose - FID

[0025] It should be appreciated that features relating to one aspect may be applicable to the other aspects. Embodiments therefore provide a computer-implemented method and system for hand pose generation in hand-object interaction (HOI). By providing a 2D image of an object and an input mask adapted to specify at least a location for HOI with the object, the present method allows generation of a 2D hand pose at a user-specified location for affordance hand object interaction. This allows the method to be better adapted at capturing the natural variations and subtleties of human hand poses, leading to more realistic and diverse hand pose generations. The ability of the present method to generate hand poses at user-specified locations by using only a 2D image of the object of interest also avoids the need to reconstruct hand poses from hand-object interaction images, thereby improving accuracy and computing efficiency.

[0026] Further, a computer-implemented method for evaluating a similarity between a first population and a second population of hand poses is also provided. By calculating and obtaining the pose-FID score, the quality of generated 2D and / or 3D hand poses can be evaluated. The present evaluation method therefore provides versatility for evaluating the quality of generated hand poses for various applications, such as robotic manipulation training and interactive training simulations. The present evaluation method also evaluates the generated hand poses more holistically by considering both a central tendency (i.e. using the mean) and a spread (i.e. using the covariance) of the joint positions within the populations. The pose-FID score generated provides a comprehensive measure of the quality and realism of the generated hand poses. Brief description of the drawings

[0027] Embodiments will now be described, by way of example only, with reference to the following drawings, in which:

[0028] Figure 1 shows a block diagram of a system for hand pose generation in hand-object interaction (HOI) in accordance with an embodiment;

[0029] Figure 2 is a flowchart showing steps of a method for hand pose generation in handobject interaction (HOI) in accordance with an embodiment;

[0030] Figure 3 is a flowchart showing steps of a method for training a model in accordance with an embodiment;

[0031] Figure 4 is a flowchart showing steps of a method for evaluating a similarity between a first population and a second population of hand poses in accordance with an embodiment;

[0032] Figure 5 shows a block diagram of an overview of the process flow for hand pose generation including training of an encoder-decoder based model in accordance with an embodiment; and

[0033] Figure 6 shows a series of photographs to illustrate qualitative comparison of two variations of the methods of the present disclosure with a baseline method in accordance with an embodiment.

[0034] Detailed description

[0035] Exemplary embodiments relate to a computer-implemented method and system for hand pose generation in hand-object interaction (HOI).

[0036] Current hand grasp generation methods rely on 3D information for both a hand and an object but these methods lack controllability concerning the hand's location and orientation. In the present disclosure, to overcome this limitation, a hand pose is treated as discrete graph structure and geometric priors are exploited. Higher order contextual dependency among points forming a 2D hand pose improves a quality of the generated hand poses. In the present disclosure, a framework relating to higher order geometric representations (HOR's) inspired by spectral graph theory and vector algebra to improve the quality of generated hand poses is provided. In embodiments as discussed below, an encoder-decoder based diffusion model (based on 2D information) for controllable hand pose generation is demonstrated which outperforms the current state-of-the-art. The present method overcomes the limitations of existing methods such as a lack of controllability and dependency on 3D information. Further, popular metrics like the FID method and the MMD method are biased and inefficient for evaluating the generated hand poses. In the present disclosure, a computer-implemented method for evaluating a similarity between a first population and a second population of hand poses is also provided. This method provides an efficient and stable evaluation framework which addresses inefficiencies and biases in current metrics like the FID and the MMD. This is discussed in relation to Figure 4 below.

[0037] Figure 1 shows a block diagram of a system 100 for hand pose generation in hand-object interaction (HOI) in accordance with an embodiment.

[0038] The computer system 100 has memory that stores computer program modules which implement computer-implemented methods for hand pose generation in hand-object interaction (HOI) in accordance with an embodiment. The computer system 100 comprises a processor 102, a working memory 104, an input module 106, an output module 108, a user interface 1 10, program storage 1 12 and data storage 1 14. The processor 102 may be implemented as one or more central processing unit (CPU) chips. The program storage 1 12 is a non-volatile storage device such as a hard disk drive which stores computer program modules such as an encoder 1 16, a positional encoding module 1 18, a decoder 120, a training module 122 and an evaluation module 124. The computer program modules are loaded into the working memory 104 for execution by the processor 102. The input module 106 is an interface which allows data, for example input data or images to be received by the encoder 1 16, to be received by the computer system 100. The output module 108 is an output device which allows data and results generated by the computer system 100 to be output. The output module 108 may be coupled to a display device or a printer. The user interface 110 allows a user of the computer system 100 to input selections and commands and may be implemented as a graphical user interface.

[0039] The program storage 1 12 stores the encoder 116, the positional encoding module 1 18, the decoder 120, the training module 122 and the evaluation module 124. These computer program modules cause the processor 102 to execute various analytical processes which are described in more detail below. For example, the encoder 1 16 can be executed by the processor 102 to receive a two-dimensional (2D) image of an object and an input mask adapted to specify at least a location for HOI with the object, and to process the 2D image and the input mask to generate encoder outputs. In the present embodiment, the decoder 120 can then be executed by the processor 102 to generate a 2D hand pose based on the encoder outputs. In an embodiment, the positional encoding module 1 18 can be executed by the processor 102 to generate positional encodings for providing spatial relationships between the joints, where the positional encoding includes learnable positional encoding defined using a learnable parameter vector p, associated with an / -th joint’s position in the form of coordinates (x„ y). In an embodiment, the training module 122 can be executed by the processor 102 to train the architecture used in generating 2D hand poses, for example, an encoder-decoder based model. In an embodiment, the decoder 120 includes a transformer-based cross attention decoder. In an embodiment, at least the encoder 116 and the decoder 120 form the encoder-decoder based model and the encoder-decoder based model is trained using a Denoising Diffusion Probabilistic Model (DDPM) diffusion process. Details of the training method is discussed in detail in relation to Figure 3. The evaluation module 124 can be executed by the processor 102 to evaluate a similarity between a first population and a second population of hand poses, for example, to provide a measure of a quality and a realism of the generated hand poses. In an embodiment, the evaluation module 124 uses a pose- FID score for the evaluation process and this is described in detail in relation to Figure 4.

[0040] The program storage 1 12 may be referred to in some contexts as computer readable storage media and / or non-transitory computer readable media. In an embodiment, the computer program modules such as the encoder 1 16, the positional encoding module 1 18, the decoder 120, the training module 122 and the evaluation module 124 can be stored in a cloud storage. As depicted in Figure 1 , the computer program modules 116, 1 18, 120, 122, 124 are distinct modules which perform respective functions implemented by the computer system 100. It will be appreciated that the boundaries between these modules are exemplary only, and that alternative embodiments may merge modules or impose an alternative decomposition of functionality of modules. For example, the modules discussed herein may be decomposed into sub-modules to be executed as multiple computer processes, and, optionally, on multiple computers. Moreover, alternative embodiments may combine multiple instances of a particular module or submodule. It will also be appreciated that, while a software implementation of the computer program modules is described herein, these may alternatively be implemented as one or more hardware modules (such as field-programmable gate array(s) or applicationspecific integrated circuit(s)) comprising circuitry which implements equivalent functionality to that implemented in software.

[0041] The data storage 1 14 stores various data and parameters. As shown in Figure 1 , the data storage 1 14 has storage for encoder data 126, positional encoding module data 128, decoder data 130, training module data 132 and evaluation module data 134 for use with their corresponding modules 1 16, 1 18, 120, 122, 124. In the present embodiment, the encoder data 126 comprises data used by the encoder 1 16, such as data related to a U-Net encoder for sematic image segmentation. The positional encoding module data 128 comprises positional encoding data which can be provided to the decoder 120 for providing spatial relationships between joints in a hand pose. The decoder data 130 comprises at least data for use in generating a 2D hand pose based on the encoder outputs. In an embodiment where the decoder 120 includes a transformer-based cross attention decoder, the decoder data 130 can include trained decoder parameters or weights of the transformer-based cross attention decoder. The training module data 132 comprises data used for training at least the encoder-decoder based model in an embodiment. In an embodiment, the training module data 132 includes data relating to the DDPM diffusion process and / or data associated with a diffusion loss and a pose reconstruction loss for training the encoder-decoder based model. The evaluation module data 134 comprises data associated with the evaluation module 124 for evaluating a similarity between a first population and a second population of hand poses. In an embodiment, the evaluation module data 134 includes data for use in generating a pose-FID score, such as pairwise distances between two points of a hand pose for generating a descriptor vector, a mean and a covariance of descriptor vectors of each of the first population and the second population of hand poses, and the pose- FID score. Similar to the computer program modules as discussed above, it will be appreciated that the boundaries between the data 126, 128, 130, 132 and 134 of the data storage 114 as presented in Figure 1 are exemplary only, and that alternative embodiments may impose an alternative decomposition.

[0042] Although the technical architecture is described with reference to a computer system 100, it should be appreciated that the technical architecture may be formed by two or more computers in communication with each other that collaborate to perform a task. For example, but not by way of limitation, an application may be partitioned in such a way as to permit concurrent and / or parallel processing of the instructions of the application. Alternatively, the data processed by a computer program module may be partitioned in such a way as to permit concurrent and / or parallel processing of different portions of a data set by the two or more computers. In an embodiment, virtualization software may be employed by the technical architecture to provide the functionality of a number of servers that is not directly bound to the number of computers in the technical architecture. In an embodiment, the functionality disclosed above may be provided by executing a computer program module or computer program modules in a cloud computing environment. Cloud computing may comprise providing computing services via a system connection using dynamically scalable computing resources. A cloud computing environment may be established by an enterprise and / or may be hired on an as-needed basis from a third-party provider.

[0043] In the present disclosure, the system 100 of Figure 1 is used in a computer-implemented method for hand pose generation in hand-objection interaction (HOI). Computer- implemented methods in relation to hand pose generation are described in relation to Figures 2 to 4 below.

[0044] Figure 2 is a flowchart showing steps of a method 200 for hand pose generation in handobject interaction (HOI) in accordance with an embodiment.

[0045] In a step 202, the system 100 is adapted to receive a two-dimensional (2D) image of an object and an input mask adapted to specify at least a location for HOI with the object. In an embodiment, the input mask includes a lollipop-shaped mask having parameters including a palm size, a location and an approaching direction. In an embodiment, the input mask includes five parameters defined using the palm size in the form of a2, the location in the form of coordinates (x, y), and the approaching direction in the form of arctan(bi, te). Providing an input mask adapted to specify at least a location for HOI with the object allows generation of hand pose at a user-specified location for affordance hand object interaction. This is in contrast to existing works for which the locations of the hand-object-interaction may be randomly selected.

[0046] In a step 204, the encoder 1 16 is executed by the processor 102 to process the 2D image and the input mask to generate encoder outputs. In an embodiment, the encoder 1 16 includes a Unet-Encoder. In a step 206, the decoder 120 is executed by the processor 102 to generate a two- dimensional (2D) hand pose based on the encoder outputs. In an embodiment, the decoder 120 includes a transformer-based cross attention decoder which is trained to generate 2D hand pose in response to the 2D image of the object and the input mask received. In the present embodiment, a 2D hand pose is represented by points relating to joints of the 2D hand pose.

[0047] In an embodiment, the transformer-based cross-attention decoder can be coupled with learnable positional encodings as inputs into the decoder 120 and is adapted to attend to both global hand structure and local finger articulation simultaneously for accurate affordance hand pose generation.

[0048] Figure 3 is a flowchart showing steps of a method 300 for training an encoder-decoder based model using a Denoising Diffusion Probabilistic Model (DDPM) diffusion process in accordance with an embodiment.

[0049] In a step 302, the training module 122 is executed by the processor 102 to calculate a diffusion loss, Ldtffusion. In the present embodiment, the diffusion loss Ldiffusionis calculated using a diffusion loss function given by: where0relates to original data, xtrelates to noisy data at time step t, e represents noise and network egrelates to prediction of noise at each time step.

[0050] In a step 304, the training module 122 is executed by the processor 102 to calculate a pose reconstruction loss, Lpose, using a pose reconstruction loss function given by:

[0051] Lpose= WHy - yi i2] (2) where y represents ground truth 2D hand poses and y represents predicted 2D hand poses

[0052] In a step 306, the training module 122 is executed by the processor 102 to minimise an overall loss function, L, using: where is a hyperparameter adapted to balance the diffusion loss and the pose reconstruction loss. It is important to evaluate a quality of the generated hand poses for improving and / or maintaining an accuracy of the hand pose generation. In the present disclosure, an evaluation metric, called the pose-FID score (or pose-FID metric), is introduced for evaluating the generated hand poses. This evaluation metric pose-FID score is built upon the principles of the Frechet Inception Distance (FID) metric for use in capturing a similarity between two sets of hand pose points. The pose-FID score can evaluate the quality of both 2D and 3D hand poses, making it versatile to evaluate the quality of generated hand poses for various applications such as robotic manipulation training and interactive training simulations. In other words, although the present hand pose generation method is described in relation to 2D hand poses, it should be appreciated that this evaluation metric can also be applied to other hand pose generation methods, such as for the generation of three-dimensional (3D) hand poses.

[0053] Figure 4 is a flowchart showing steps of a method 400 for evaluating a similarity between a first population and a second population of hand poses in accordance with an embodiment. In the present embodiment, the evaluation metric pose-FID score is used to evaluate the similarity between the first population and the second population of hand poses for assessing the hand pose generation performance. For example, the pose-FID score can be used to compare distribution of the generated hand poses with the distribution of real hand poses, where a lower pose-FID score indicates a better generation quality.

[0054] In a step 402, the evaluation module 124 is executed by the processor 102 to calculate pairwise distances between each points to generate a descriptor vector for each 2D hand pose of the first population and the second population of 2D hand poses. In the present embodiment, a pairwise distance between point / and point / is defined as: (4) wher 7are positions of points / and / , respectively.

[0055] In a step 404, the evaluation module 124 is executed by the processor 102 to calculate a mean, and a covariance, oi, of descriptor vectors of the first population of 2D hand poses and a mean descriptor vectors of the second population of 2D hand poses. In a step 406, the evaluation module 124 is executed by the processor 102 to calculate the pose-FID score as: pose where Trfa + a2- 2(<71cr2)1 / 2) is a trace of the matrix (X + <J2- 2(o1o2)1 / 2).

[0056] This method evaluates the generated hand poses more holistically by considering both the central tendency and the spread of the joint positions within the populations. The pose-FID score provides a comprehensive measure of the quality and realism of the generated hand poses.

[0057] Figures 5 and 6 provide detailed examples of the computer-implemented method for hand pose generation in hand-object interaction (HOI).

[0058] Figure 5 shows a block diagram 500 of an overview of the process flow or architecture for hand pose generation including training of an architecture model in accordance with an embodiment. In the present embodiment, the architecture model comprises an encoder-decoder based model designed for generating 2D hand poses at specified locations. In the present example, the architecture employs a 21 -hand joint model (i.e. a hand model or a hand pose comprising 21 joints or hand joints) which allows for detailed and accurate representations of hand poses.

[0059] As shown in the block diagram 500, a two-dimensional (2D) image 502 of an object and an input mask 504 are provided as inputs to an encoder 506. In the present example, a five-parameter input mask 504 is used to specify a position and an orientation of a hand in the 2D image frame 502. As shown in Figure 5, the input mask 504 is represented as a lollipop shape mask with five (5) parameters. These five parameters include a hand palm size a2, a location x, y, and an approaching direction arctan(bi, b2). Unlike previous approaches that rely on depth sensors, motion capture systems, or 3D hand models, the present method requires only 2D hand poses that are coupled with the target object, enabling accurate simulations of hand-object interactions.

[0060] In the present example, the encoder 506 includes a Unet-encoder which is capable of capturing both global and local features efficiently for providing accurate hand pose generation. The Unet-encoder 506 is adapted to receive the input image 502 and the input mask 504 associated with the input image 502 and to process the input image 502 and the input mask 504 to generate encoder outputs. In the present example, learnable positional encodings 508 are also included in the architecture to capture spatial relationships between joints of the hand pose. In the present embodiment, learnable positional encodings are applied to each joint. Unlike static positional encodings that use predefined functions, learnable positional encodings allow a model, for example an encoder-decoder based model, to optimise positional information during training. This approach enables the encoder-decoder based model to better capture the spatial dependencies and interactions between joints. A learnable parameter vector p, is given by: (6)

[0061] For a joint at position (x, y), the learnable positional encoding is defined as a vector p, for an / -th joint, which is updated during training to capture an optimal positional information. In the present embodiment, these positional encodings are learned during the training process (e.g. using a diffusion training method as described below), allowing the encoder-decoder based model to dynamically adjust the positional information for each joint to improve performance. By incorporating learnable positional encodings in the present embodiment, the encoder-decoder based model can more effectively understand and generate the spatial relationships between the 21 hand joints, leading to more accurate and realistic hand pose generation. In the present embodiment, an output from the encoder 506 is used as an input for training and optimising to form the learnable positional encodings 508, as shown in relation to Figure 5.

[0062] The architecture of the present embodiment also includes a transformer-based cross attention decoder 510. The decoder 510 is based on a transformer model having cross attention mechanism and is adapted to enhance an ability of the decoder 510 to focus on relevant features from the encoded inputs (i.e. encoder outputs from the encoder 506 and the positional encodings 508).

[0063] The cross attention mechanism can be described using the following equation. Given an input quer d value V , the attention output is calculated as: (7) where dkis the dimension of the key vectors. The output states of the encoder 506 are used to initialise the key K and the value V. The input query Q is a learnable weight matrix and T is a transpose operation. The cross attention mechanism deployed is adapted to allow the decoder 510 to attend to different parts of the encoded inputs dynamically, leading to more precise and accurate hand pose generation.

[0064] In the present embodiment, a Denoising Diffusion Probabilistic Model (DDPM) diffusion process is used to train the encoder-decoder based model for hand pose generation which is adapted to synthesize hand poses at user specified locations for affordance hand object interaction (HOI) generation. Diffusion models have become a powerful generative technique, particularly for image synthesis tasks. Denoising Diffusion Probabilistic Models (DDPMs) iteratively denoise a noise distribution to generate high- quality images. Diffusion models can also be used for conditional image generation, enabling control over the generated outputs. In the context of hand pose generation, a diffusion-based method can leverage on depth images and point clouds as multi-modal input conditions to estimate hand poses, such as 3D hand poses. These diffusion methods employ a joint-wise local feature-aware denoising module and a kinematic correspondence-aware layer to capture local details and kinematic dependencies, respectively.

[0065] In the present disclosure, the DDPM diffusion process used has a forward phase that adds noise to data and a reverse phase that denoises it. The loss function for the DDPM can be described using the Equation (1 ) as follows: Oil ] where x0is the original data, xtis the noisy data at time step t, and e represents the noise. The network egpredicts the noise at each step.

[0066] To ensure the generated poses are accurate, in the present embodiment, a pose reconstruction loss which measures the difference between the predicted hand poses and the ground truth poses is used. The pose reconstruction loss is given by the Equation (2) as: where y represents the ground truth hand poses, and y represents the predicted hand poses.

[0067] An overall loss function (also referred to as combined loss) 512 which combines the diffusion loss and the pose reconstruction loss is used in the present example for training the encoder-decoder based model. The overall loss function is given by the Equation (3) as: where -A is a hyperparameter that balances the two loss terms.

[0068] Finally, the generated 2D hand pose 513 for the object 502 is shown in the image 514.

[0069] As described in relation to Figure 4, in the present embodiment, the pose-FID score can also be used for evaluating a similarity between a first population and a second population of hand poses, such as a similarity between a distribution of the generated hand poses and a distribution of real hand poses, for assessing the hand pose generation performance. To reiterate, to compute the pose-FID score, pairwise distances between each hand joint can be calculated to generate a descriptor vector for each hand pose, akin to the step 402. In the present example, given the set of 21 joints in a hand pose, J, the pairwise distance between joint / and joint y ean be calculated using the Equation (4) as: where and pj are the positions of joints / and j, respectively. The descriptor vector for a hand pose is constructed by computing dtj for all pairs ( / ', j) in J.

[0070] Once the descriptor vectors are calculated for each of the two population of hand poses, a mean / j and a covariance <J of the descriptor vectors over a given population of hand poses are computed. In the present example having two populations of hand poses, with means and p? and covariances and o2, the pose-FID score can be calculated using the Equation (5) as:

[0071] Using the aforementioned hand pose generation method based on the architecture as shown in relation to Figure 5 and the evaluation metric pose-FID score as discussed above, a quantitative evaluation and qualitative comparison of generated hand poses using three models are provided in the following example. The three models used are: (1 ) the Vanilla model, (2) the PoseNet model, and (3) the PoseNetDeep model. The Vanilla model serves as the baseline method, utilizing a 3-layer fully connected Multilayer Perceptron (MLP) network as the decoder for hand pose generation. The PoseNet model uses the methods as described in the present embodiment and leverages a transformerbased cross attention decoder to enhance an accuracy and realism of the generated poses. The PoseNetDeep model is an extension of the PoseNet model which incorporates two additional cross attention layers in the transformer-based cross attention decoder to further refine the hand pose generation process.

[0072] It is noted that in the present example, the encoder used for the Vanilla model (i.e. the baseline model) is similar to that used in the PoseNet model and the PoseNetDeep model, utilizing a Unet-based encoder to process the input data in a similar manner as shown and described in relation to the architecture of Figure 5. However, differing from the PoseNet model and the PoseNetDeep model, the Vanilla model used in the present example does not include positional encodings.

[0073] In the present example, a HOI4D dataset is utilised, specifically selecting a subset that includes the ’Grasp’ action performed with a water cup object. The HOI4D dataset is a four dimensional (4D) egocentric dataset for category-level human-object interaction and details of this can be found in “Y. Liu et al., "HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction," 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 20981 -20990, doi: 10.1 109 / CVPR52688.2022.02034” which is incorporated herein for completeness. The subset of the HOI4D dataset used comprises 2200 images, which are used to train these encoder-decoder based models by the diffusion method. The specific dataset used is to ensure that training is limited to these specific conditions to ensure a controlled environment for evaluating the performance and quality of hand pose generation across the three different models. The comparisons aim to highlight the improvements and strengths of the present methods over the baseline method / model, demonstrating an effectiveness of utilising cross-attention mechanisms in generating high-quality hand poses.

[0074] Figure 6 shows a series of photographs 600 to illustrate qualitative comparison of two variations of the present method with a baseline method in accordance with an embodiment.

[0075] The various stages of the hand pose generation method / process is shown by the different columns of the series of photographs 600. As shown in relation to Figure 6, the first column 602 shows an input RGB image of an object, the second column 604 shows an input mask used, the third column 606 shows the resultant output (i.e. a generated hand pose) based on the Vanilla model, the fourth column 608 shows the resultant output based on the PoseNet model, and the fifth column 610 shows the resultant output based on the PoseNetDeep model.

[0076] The three rows 612, 614, 616 relate to different input masks used as shown in relation to the column 604. It is clear from the series of photographs 600 that by using different input masks to specify a location and / or an approaching direction of the hand pose, the affordance of the Hand-Object Interaction (HOI) can be controlled as illustrated by the various outputs of the models as shown with respect to the third, fourth and fifth columns 606, 608, 610 for the rows 612, 614, 616.

[0077] In the present embodiment, the hand pose is represented by 21 points, where finger joints and fingertips are represented as 21 points, and fingers are represented as lines. The points and lines as shown in relation to the outputs of the Vanilla model, the PoseNet model and the PoseNetDeep model are for visualisation of the generated hand pose.

[0078] Further, all the three models were trained using the DDPM diffusion process with 1000 steps as the training diffusion hyperparameters, and using 0.2 as the loss hyperparameter A This extensive training process ensures that these models learned to generate accurate and realistic 2D hand poses.

[0079] In the present example, pose-FID values for the outputs of the different models were calculated and these are tabulated in Table 1 below.

[0080] Table 1 : Comparison of pose-FID values for three models

[0081] Table 1 is used to illustrate a comparison of the pose-FID values between the PoseNet model, the PoseNetDeep model and the baseline Vanilla model. As shown in Table 1 , the Vanilla model achieved a pose-FID value of 87.7, indicating lower similarity to the reference set of hand poses. In contrast, the PoseNet model achieved a much lower pose-FID value of 9.49, showcasing its superior performance in generating realistic hand poses. The PoseNetDeep model further improved the pose-FID value to 8.09, highlighting the benefits of incorporating additional cross attention layers in the transformer-based cross attention decoder. The results shown in Table 1 demonstrate a notable improvement in the quality of the generated hand poses by using the methods of the present disclosure (i.e. the PoseNet model or the PoseNetDeep model) as compared to the baseline Vanilla model.

[0082] Understanding human hand-object interactions is a longstanding challenge in computer vision. Automatically synthesizing affordance hand pose on interested objects has many applications in virtual reality, human-computer interaction and robotic manipulation training. Studies have shown that affordance hand pose generation can achieve impressive results.

[0083] The systems and methods for hand pose generation as described in the present disclosure serve to address problems in the existing art and / or provide a useful alternative. Existing methods for generating plausible hand poses given an interested object lack control over the placement of the generated hand poses (i.e., the locations of the hand-object-interaction are randomly selected) which limits their applicability greatly. Further, existing methods for reconstructing hand poses from hand-object interaction images relying on depth sensors or motion capture systems can be affected by occlusions, lighting conditions, and surface properties, or in the case of 3D hand models, can be affected by the inaccuracy in capturing the nuances of individual hand shapes and poses. In contrast, the systems and methods of the present disclosure are capable of generating hand poses at user-specified locations using only a 2D image of the object of interest as an input, improving both an accuracy and an efficiency of hand pose generation.

[0084] Further, with the advent of deep learning, data-driven methods leveraging large-scale datasets have become prevalent for supervised training. Convolutional neural networks (CNNs) can be employed to directly predict 3D hand joint locations from RGB images. Generative adversarial networks (GANs) can also be used to synthesize depth hand images from 3D hand poses, enabling data augmentation and semi-supervised learning. In the methods of the present disclosure, a conditional affordance 2D hand pose generation method based on a transformer-based cross attention decoder is provided. A Denoising Diffusion Probabilistic Model (DDPM) diffusion process is used to train the encoder-decoder based model which provides superior performance in generating realistic hand poses as compared to a baseline Vanilla model. The present methods can be applied to generate synthetic but yet realistic hand poses for robotic manipulation training and interactive training simulations. With the ability to synthesize realistic hand poses in a controllable manner, the approach as described in the present disclosure enables training of robotic systems to better understand and replicate human-like manipulation of objects, improving their functionality and reliability. As illustrated above, the methods of the present disclosure can also be applied to create interactive training simulations, providing users with realistic environments to practice complex tasks, such as medical procedures, assembly line operations, or maintenance work. These simulations enhance training effectiveness and skill acquisition by allowing users to interact with virtual objects in a lifelike manner. Additionally, in the realm of ergonomics and design, the present methods and systems offer valuable insights into optimising tools and equipment by simulating and analysing hand-object interactions at specific locations. This application not only improves user comfort and efficiency but also aids in designing more ergonomic and user-friendly products. Collectively, these applications highlight the transformative potential of the present hand pose generation methods across various industries.

[0085] Moreover, it should be appreciated that the hand pose generation methods as described in the present disclosure can be used in other wide-ranging applications and have potential to revolutionise various fields. One significant application is in synthetic dataset generation, where the methods of the present disclosure excel in creating affordance hand poses necessary for diverse tasks. This capability is pivotal for developing large- scale, high-quality datasets that can enhance machine learning models, particularly in areas requiring precise hand-object interaction.

[0086] Overall, the present hand pose generation methods offer a practical and efficient way for simulating hand-object interactions, with potential applications in areas such as robotics, virtual reality, ergonomic design, and synthetic data generation for machine learning models.

[0087] Alternative embodiments

[0088] It should be appreciated that in an embodiment, the learning positional encodings are also trained using the diffusion method as afore-described, in conjunction with the encoder and the decoder in the encoder-decoder based model. In an embodiment, the encoder-decoder architecture forms part of a machine learning model, such as a transformer model. In other embodiments, the decoder uses a transformer-based model (e.g. the transformer-based cross attention model), while the encoder uses a separate machine learning model such as a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0089] Alternative embodiments may include: (i) the positional encoding module 1 18 being not part of the system 100 for generating positional encoding for providing spatial relationships between joints in a hand pose, and the positional encoding is provided to the decoder 120 from an external source; (ii) the positional encoding is not provided to the decoder 120 for generating the 2D hand pose, for example, if other decoder model such as a convolutional neural network (CNN) or a recurrent neural network (RNN) is used; (iii) a hand pose comprising other numbers of hand joints, e.g. 15, 20, 25 and 30 etc.; (iv) other form of positional encoding, other than a learnable positional encoding as defined using a learnable parameter vector p, can be used; (v) an input mask having other shapes or combination of shapes such as a triangular shape, a trapezium shape or a rectangular shape with rounded comers to represent a palm of the hand in combination of a rectangular shape to present an extension from the palm of the hand (e.g. a forearm) etc.; (vi) other types or forms of decoder model for use in the decoder 120 such as a Multilayer Perceptron (MLP) network based decoder or a transformerbased self attention decoder etc.; (vii) the encoder 1 16 and / or the decoder 120 being trained using another training process other than the DDPM diffusion process; (viii) training the encoder 1 16 and / or the decoder 120 using only the diffusion loss based on the DDPM diffusion process; (ix) incorporating one or more additional cross attention layers in the transformer-based decoder 120; (x) using other form of evaluation technique (e.g. FID) to evaluate a quality of the generated 2D hand poses; and (xi) using other encoder besides the U-net encoder as used in the afore-described examples.

[0090] Although only certain embodiments of the present invention have been described in detail, many variations are possible in accordance with the appended claims. For example, features described in relation to one embodiment may be incorporated into one or more other embodiments and vice versa.

Claims

Claims1 . A computer-implemented method for hand pose generation in hand-object interaction (HOI), the computer-implemented method comprising: receiving a two-dimensional (2D) image of an object and an input mask adapted to specify at least a location for HOI with the object; processing, using an encoder, the 2D image and the input mask to generate encoder outputs; and generating, using a decoder, a two-dimensional (2D) hand pose based on the encoder outputs, the 2D hand pose being represented by points relating to joints of the 2D hand pose.

2. The computer-implemented method of claim 1 , wherein the input mask includes a lollipop-shaped mask having parameters including a palm size, a location and an approaching direction.

3. The computer-implemented method of claim 1 or claim 2, further comprising: receiving, at the decoder, positional encoding for providing spatial relationships between the joints, wherein the positional encoding includes learnable positional encoding defined using a learnable parameter vector p, associated with an / -th joint’s position in the form of coordinates (x,, yi).

4. The computer-implemented method of claim 3, wherein the 2D hand pose includes twenty-one joints.

5. The computer-implemented method of claim 3 or claim 4, wherein the decoder includes a transformer-based cross attention decoder.

6. The computer-implemented method of claim 5, further comprising incorporating one or more additional cross attention layers in the transformer-based cross attention decoder.

7. The computer-implemented method of any one of the preceding claims, wherein the encoder and the decoder form an encoder-decoder based model, the computer-implemented method further comprising training the encoder-decoder based model using a Denoising Diffusion Probabilistic Model (DDPM) diffusion process.

8. The computer-implemented method of claim 7, wherein training the encoder-decoder based model using the DDPM diffusion process comprises: calculating a diffusion loss, Lditf usion, using a diffusion loss function given by:where x0relates to original data, xtrelates to noisy data at time step t, e represents noise and network eerelates to prediction of noise at each time step.

9. The computer-implemented method of claim 8, wherein training the encoder-decoder based model using the DDPM diffusion process further comprises: calculating a pose reconstruction loss, Lpose, using a pose reconstruction loss function given by:where y represents ground truth 2D hand poses and y represents predicted 2D hand poses; and minimising an overall loss function, L, using:L [‘diffusion T A Lpose whereis a hyperparameter adapted to balance the diffusion loss and the pose reconstruction loss.

10. The computer-implemented method of any one of the preceding claims, the computer-implemented method further comprising: generating a first population and a second population of 2D hand poses; and calculating a pose-FID score to capture a similarity between the first population and the second population of 2D hand poses, wherein calculating the pose-FID score comprises: calculating pairwise distances between each points to generate a descriptor vector for each 2D hand pose of the first population and the second population of 2D hand poses, a pairwise distancebetween point / and point / is defined as: dtj = I I Pi - P / l l where and pj are positions of points / and j, respectively;calculating a mean,and a covariance, a1, of descriptor vectors of the first population of 2D hand poses and a mean, p2, and a covariance, cr2, of descriptor vectors of the second population of 2D hand poses; and calculating the pose-FID score as: pose1 1 . A computer-implemented method for evaluating a similarity between a first population and a second population of hand poses, wherein a hand pose is represented by points relating to joints of the hand pose, the computer-implemented method comprising: calculating a pose-FID score to capture the similarity between the first population and the second population of hand poses, wherein calculating the pose-FID score comprising: calculating pairwise distances between each points to generate a descriptor vector for each hand pose of the first population and the second population of hand poses, a pairwise distancebetween point Z and point / is defined as:d I I l l where and pj are positions of points / and j, respectively; calculating a mean,and a covariance,of descriptor vectors of the first population of hand poses and a mean, / z2, and a covariance, <J2, of descriptor vectors of the second population of hand poses; and calculating the pose-FID score as: pose12. A computer readable medium storing processor executable instructions which when executed on a processor cause the processor to carry out a computer-implemented method according to any one of claims 1 to 1 1 .

13. A system for hand pose generation in hand-object interaction (HOI), the system comprises a processor and a data storage storing computer program instructions operable to cause the processor to: receive a two-dimensional (2D) image of an object and an input mask adapted to specify at least a location for HOI with the object; process, using an encoder, the 2D image and the input mask to generate encoder outputs; andgenerate, using a decoder, a two-dimensional (2D) hand pose based on the encoder outputs, the 2D hand pose being represented by points relating to joints of the 2D hand pose.

14. The system of claim 13, wherein the input mask includes a lollipop-shaped mask having parameters including a palm size, a location and an approaching direction.

15. The system of claim 13 or claim 14, wherein the data storage further stores computer program instructions operable to cause the processor to: receive, at the decoder, positional encoding for providing spatial relationships between the joints, wherein the positional encoding includes learnable positional encoding being defined using a learnable parameter vector p, associated with an / -th joint’s position in the form of coordinates (x„ y).

16. The system of claim 15, wherein the decoder includes a transformer-based cross attention decoder.

17. The system of any one of claims 13 to 16, wherein the encoder and the decoder form an encoder-decoder based model and wherein the data storage further stores computer program instructions operable to cause the processor to: train the encoder-decoder based model using a Denoising Diffusion Probabilistic Model (DDPM) diffusion process.

18. The system of claim 17, wherein the data storage further stores computer program instructions operable to cause the processor to: calculate a diffusion lossusing a diffusion loss function given by:where x0relates to original data, xtrelates to noisy data at time step t, e represents noise and network eerelates to prediction of noise at each time step.

19. The system of claim 18, wherein the data storage further stores computer program instructions operable to cause the processor to: calculate a pose reconstruction loss, Lpose, using a pose reconstruction loss function given by:where y represents ground truth 2D hand poses and y represents predicted 2D hand poses; and minimise an overall loss function, L, using:where / t is a hyperparameter adapted to balance the diffusion loss and the pose reconstruction loss.

20. The system of any one of claims 13 to 19, wherein the data storage further stores computer program instructions operable to cause the processor to: generate a first population and a second population of 2D hand poses; calculate a pose-FID score to capture a similarity between the first population and the second population of 2D hand poses, wherein to calculate the pose-FID score comprises to: calculate pairwise distances between each points to generate a descriptor vector for each 2D hand pose of the first population and the second population of 2D hand poses, a pairwise distance dtj between point / and point / ' is defined as: where pLand pj are positionsof points / and j, respectively; calculate a mean, JU-L, and a covariance,of descriptor vectors of the first population of 2D hand poses and a mean, p2, and a covariance, <J2, of descriptor vectors of the second population of 2D hand poses; and calculate the pose-FID score as:

Images