Methods for generating modified avatars

Abstract

A method for generating a personalised avatar for a user of an extended reality (XR) device comprises obtaining S100 data on a head position and a position of at least one hand of the user and estimat

Claims

1. A computer-implemented method for generating a modified avatar for a user of an extended reality, XR, device, the method comprising:5 obtaining data from a plurality of sensors of the XR device, the plurality of sensorsproviding data on a head position and one or both hand positions of the user while using the XR device;estimating, using the obtained data:a body shape of the user, and10 at least one skeletal joint position of the user;fine-tuning, using the estimated body shape and at least one skeletal joint position, a first pre-trained machine learning, ML, model that is pre-trained to generate avatars for users of the XR device, to:generate a prediction of body motion; and15 generate, using the generated prediction, a modified avatar for the user fordisplay on the XR device, the modified avatar having a body shape and at least one skeletal joint position similar to those estimated for the user.

2. The method as claimed in claim 1 wherein estimating a body shape of the user and at 20 least one skeletal joint position of the user comprises:using a second trained machine learning, ML, model for estimating human body shapes to process the obtained data.

3. The method as claimed in claim 2 wherein obtaining data from a plurality of sensors25 comprises:requesting, via the XR device, a user to perform a personalisation process during which the data is obtained.

4. The method as claimed in claim 3 wherein when the request is declined:30 obtaining data from a plurality of sensors comprises obtaining data during ordinary useof the XR device by the user; andestimating a body shape of the user and at least one skeletal joint position of the user comprises using the second trained ML model to estimate at least a height and limb lengths of the user.3504 02 265. The method as claimed in claim 3 wherein when the request is accepted, obtaining data from one or more sensors comprises:instructing the user to perform at least one pre-defined pose; andobtaining data from the plurality of sensors while the user performs the at least one 5 pre-defined pose.

6. The method as claimed in claim 5 wherein instructing the user to perform at least one pre-defined pose comprises instructing the player to perform any one or more of: a T-Pose; a touching shoulders pose; a touching knees pose; a hand around waist pose; and a hands 10 around lower torso pose.

7. The method as claimed in claim 5 or 6 wherein instructing the user to perform at leastone pre-defined pose comprises displaying, on a display of the XR device, an image or video of the at least one pre-defined pose to aid the user to perform the at least one pre-defined 15 pose.

8. The method as claimed in any of claims 5 to 7 wherein estimating a body shape of the user and at least one skeletal joint position of the user comprises:generating an initial estimate of at least a height and limb lengths of the user by using 20 the second trained ML model to process a subset of the obtained data;iteratively modifying the initial estimate, using the second trained ML model, to identify optimum values for at least the height and limb lengths of the user that best fit all of the obtained data; andoutputting the estimate that best fits all of the obtained data as a final estimate for at 25 least the height and limb lengths of the user.

9. The method of any one of the preceding claims wherein estimating the body shape of the user and at least one skeletal joint position comprises estimating one or more of: height of the user; hand length; arm length; leg length; torso length; arm span; and torso width.3010. The method as claimed in any preceding claim, further comprising:receiving, from a user device that is communicatively coupled to the XR device, motion information indicative of at least one user-specific body motion; andfine-tuning, using the received motion information, the first pre-trained ML model to 35 generate the personalised avatar of the user to move in a similar way to the at least one userspecific body motion.04 02 2611. The method as claimed in claim 10 wherein receiving motion information comprises: receiving a plurality of frames or images that include at least a position of at least one skeletal joint of the user during the at least one user-specific body motion.

512. The method as claimed in any preceding claim wherein obtaining data from a plurality of sensors comprises:extracting, from the data obtained from each sensor, any one or more of: position; velocity; orientation; and angular velocity of the sensor.1013. The method as claimed in claim 12 wherein fine-tuning the first pre-trained machine learning, ML, model comprises:training the first pre-trained ML model to predict angular orientations of a plurality of skeletal joints using the extracted plurality of dimensional features.1514. A system for generating a modified avatar for a user, the system comprising:an extended reality, XR, device comprising a plurality of sensors, and at least one processor coupled to memory, for:obtaining data from a plurality of sensors of the XR device, the plurality of20 sensors providing data on a head position and one or both hand positions of the userwhile using the XR device;estimating, using the obtained data: a body shape of the user, and at least one skeletal joint position of the user;25 fine-tuning, using the estimated body shape and at least one skeletal jointposition, a first pre-trained machine learning, ML, model that is pre-trained to generate avatars for users of the XR device, to:generate a prediction of body motion; andgenerate, using the generated prediction, a modified avatar for the user, 30 the modified avatar having a body shape and at least one skeletal joint positionsimilar to those estimated for the user.

15. The system as claimed in claim 14 further comprising a user device, the user device comprising:35 at least one camera for capturing a sequence of images of the user while using the XRdevice; and04 02 26at least one processor coupled to memory for:receiving the captured sequence of images; andgenerating, using the captured sequence of images and a third trained machine learning, ML, model for three-dimensional pose sequence generation, motion 5 information indicative of at least one user-specific body motion of the user.

16. The system as claimed in claim 15 wherein generating motion information comprises: receiving, from the XR device, the estimated body shape of the user and at least one skeletal joint position of the user;10 inputting the received estimated body shape of the user and at least one skeletal jointposition of the user into the third trained ML model; andoutputting motion information from the third trained ML model.

17. The system as claimed in claim 14, 15 or 16 wherein the at least one processor of the 15 XR device estimates a body shape of the user and at least one skeletal joint position of the user by:using a second trained machine learning, ML, model for estimating human body shapes, to process the obtained data; andoutputting an estimated body shape and at least one skeletal joint position from the 20 second trained ML model.

18. The system as claimed in claim 16 or 17 wherein generating motion information comprises:generating, using the third trained ML model, an initial estimate of a plurality of three-25 dimensional poses from a subset of images from the sequence of images;iteratively modifying the initial estimate to identify optimum three-dimensional poses to fit all images in the sequence of images; andoutputting the estimate that best fits all of the image images in the sequence of images as the motion information.3019. The system as claimed in claim 16, 17 or 18 wherein fine-tuning the first pre-trained ML model comprises:receiving the generated motion information from the user device; andfine-tuning, using the received motion information, the first pre-trained ML model to 35 generate the personalised avatar of the user to move in a similar way to the at least one userspecific body motion.04 02 2620. The system as claimed in any of claims 16 to 19 wherein the XR device prompts the user to use their user device to capture a sequence of images when the XR device is used to generate a new XR environment.

521. The system as claimed in any of claims 15 to 20 wherein the at least one processor of the XR device fine-tunes the pre-trained ML model while the XR device is being used by the user.10 22. A user device for use with the system of claim 14, the user device comprising:at least one camera for capturing a sequence of images of the user while using the XR device; andat least one processor coupled to memory for:receiving the captured sequence of images; and15 generating, using the captured sequence of images and a third trained machinelearning, ML, model for three-dimensional pose sequence generation, motion information indicative of at least one user-specific body motion of the user.

23. The user device as claimed in claim 22 wherein generating motion information 20 comprises:receiving, from the XR device, the estimated body shape of the user and at least one skeletal joint position of the user; andinputting the received estimated body shape of the user and at least one skeletal joint position of the user into the third trained ML model.2524. The user device as claimed in claim 23 wherein generating motion information comprises:generating, using the third trained ML model, an initial estimate of a plurality of threedimensional poses from a subset of images from the sequence of images;30 iteratively modifying the initial estimate to identify optimum three-dimensional poses tofit all images in the sequence of images; andoutputting the estimate that best fits all of the image images in the sequence of images as the motion information.35 25. A computer-readable storage medium comprising instructions which, when executedby a processor, causes the processor to carry out the method of any of claims 1 to 13.IntellectualPropertyOfficeApplication GB2500512.5Search report under Section 17 of the Patents Act 1977Date search completed: 23 July 2025Claims searched: 1-21 and 25International classificationSubclass and subgroup Valid from A63F13 / 65 01 / 01 / 2014 A63F13 / 655 01 / 01 / 2014 G06T13 / 40 01 / 01 / 2011 G06T19 / 00 01 / 01 / 2011Field of searchWorldwide search of patent documents classified in the following areas of the IPC:G06T, A63F, G06FDatabases used in the preparation of this search report:SEARCH-NPL; SEARCH-PATENTDocuments considered to be relevantPatent literatureCategory Relevant claims Document of relevanceIntellectual Property Office is an operating name of the Patent Officewww.gov.uk / ipoA - CN 119494943 A LUXSHARE PRECISION TECH NANJING CO LTD, See whole document A - US 2022 / 0319041 A1 FACEBOOK TECH LLC, See whole document A - US 2021 / 0275925 A1 ELECTRONIC ARTS INC, See whole documentNon-patent literatureCategory Relevant claims Document of relevance A Computer Graphics Forum, Volume 41, Issue 8, Eurographics - The European Association for Computer Graphics and John Wiley &Sons Ltd., 20 / 03 / 2023, J. L. Ponton e.t aL, Combining Motion Matching and Orientation Prediction to Animate Avatars for Consumer-Grade VR Devices, 107-118, https: / / onlinelibrary.wiley.eom / doi / 10.1111 / cgf.14628, See whole documentCategoriesLetter or symbolDescriptionDocument indicating lack of novelty or inventive step.Document indicating lack of inventive step, if combined with another document of the same category.Member of the same patent family.Letter or symbol Description A Document indicating technological background. P Document published on or after the priority date but before the fling date of the present application. E Earlier application published on or after the filing date of the present application.