Computing system and method for rendering avatars

Abstract

A computing system is provided that includes a processor configured to execute an augmented reality avatar program to receive a second user avatar data object and second user gesture information via a computer network, receive an image from a real-time image feed captured by a camera of a computing device, and transmit the received image to the computer network. Identifying a virtual surface corresponding to the physical surface appearing in the image; a position and gesture of a second user avatar relative to the virtual surface is calculated based on the second user gesture information, and a second user avatar data object is rendered, thereby displaying the second user avatar of the second user superimposed on the image. The second user avatar is anchored to the virtual surface according to the calculated position and gesture of the second user avatar. The virtual surface is world-locked to a physical surface in an image in a real-world 3-D environment.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Application No. 17 / 806,701, filed on June 13, 2022, entitled “COMPUTING SYSTEM AND METHOD FORRENDERING AVATARS,” the disclosure of which is incorporated herein by reference in its entirety. Background Art

[0003] With the proliferation of online platforms for human-to-human interaction, including social media services, messaging services, and online gaming services, people are increasingly using avatars to represent themselves in the online world. Broadly speaking, an avatar is an image that represents a user in a virtual world, which is usually in the shape of a person, a cartoon character, or an arbitrary object. Some avatars are two-dimensional, while others are three-dimensional. Some avatars are configured to move or change facial expressions as well as the user's movements and facial expressions captured by a camera. However, even with such animated avatars, it can be said that, for example, the interaction with another person's avatar in a video chat lacks a sense of belonging between users. Therefore, there is an opportunity to improve the user experience when using avatars in computer-based interactions with others. Summary of the invention

[0004] In view of the above, a computing system is provided, which includes a processor and a memory of a computing device of a first user. The processor is configured to use a portion of the memory to execute an augmented reality avatar program to receive a second user avatar data object and second user posture information of a second user via a computer network. The processor is also configured to receive an image from a real-time image feed captured by a camera of the computing device, identify a virtual surface corresponding to a physical surface appearing in the image, calculate a position and posture of the second user avatar relative to the virtual surface based on the second user posture information, and draw the second user avatar data object, thereby displaying the second user avatar of the second user superimposed on the image. The second user avatar is anchored to the virtual surface according to the calculated position and posture of the second user avatar. The virtual surface is world-locked to a physical surface in the image in a real-world three-dimensional environment.

[0005] This summary is provided to introduce a selection of concepts in a simplified form, which are further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages noted in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A schematic diagram of a computing system according to an example of the present disclosure is shown.

[0007] Figure 2 Shows Figure 1 Another schematic diagram of a computing system.

[0008] Figure 3 An example according to the present disclosure is shown. Figure 1 An example use case of a computing system of schematically illustrates a real user viewing three avatars displayed on a virtual surface that is world-locked to the top of a real-world table.

[0009] Figure 4 According to the example of the present disclosure Figure 3 Additional illustration of an example use case showing a close-up of a real-world user viewing a computing device displaying an augmented reality image, the augmented reality image including Figure 3 Three avatars and the real world table.

[0010] Figure 5 According to an example of the present disclosure Figure 3 An illustration of an image displayed in an example use case showing movement of each avatar in response to user input.

[0011] Figure 6 is a flowchart of a method for performing an initialization process for drawing an avatar according to an example embodiment of the present disclosure.

[0012] Figure 7 is a flow chart of a method for performing a runtime process for drawing an avatar according to an example embodiment of the present disclosure.

[0013] Figure 8 is a flowchart of a method for drawing an avatar according to an example embodiment of the present disclosure.

[0014] Fig. 9 An example computing environment for the present disclosure is shown. DETAILED DESCRIPTION

[0015] refer to Figure 1 and Figure 2, the computing system 10 includes a first computing device 12A of a first user, a second computing device 12B of a second user, and a third computing device 12C of a third user. It should be understood that the number of users and their computing devices is not particularly limited, and the number of computing devices in the computing system 10 can be more than three, or only two. The first computing device 12A of the first user includes a processor 28, a memory 30, a front camera 24A configured to capture a first user image 25A of the first user, a rear camera 26A configured to capture a real-time image feed and an image 27A, and a display 22A configured to display the image 27A. It should be understood that the other computing devices in the computing system 10 are also provided with hardware similar to the first computing device 12A, so that the second computing device 12B also includes a processor, a memory, a front camera 24B, a rear camera 26B and a display 22B, and the third computing device 12C also includes a processor, a memory, a front camera 24C, a rear camera 26C and a display 22C.

[0016] The processor 28 is configured to use a portion of the memory 30 to execute the augmented reality program 14 to receive a second user avatar data object 106b and second user pose information 120 of a second user via the computer network 136. The augmented reality program 14 receives the image 27A from the real-time image feed captured by the rear camera 26A, identifies a virtual surface 34 corresponding to a physical surface appearing in the image 27A, calculates a position and pose of a second user avatar 132b relative to the virtual surface 34 based on the second user pose information 120, and draws the second user avatar data object 106b, thereby displaying the second user avatar 132b of the second user superimposed on the image 27A. The second user avatar 132b is anchored to the virtual surface 34 according to the calculated position and pose of the second user avatar 132b. The virtual surface 34 is world-locked to the physical surface in the image 27A in the real world three-dimensional environment. The second user pose information 120 and the image 27A may be the initial second user pose information and the initial image, respectively, for a session of the augmented reality program 14.

[0017] The processor 28 also executes a posture tracking module 16, which is configured to receive a first user image 25A captured by the front camera 24A, identify tracking features in the first user image 25A of the first user, generate first user posture information 108a of the first user based on the identified tracking features, and send the first user posture information 108a to the second computing device 12B of the second user via the computer network 136, and also to the three-dimensional engine 20 of the first computing device 12A for drawing and displaying the first user avatar 132a on the display 22A.

[0018] An image 27A from a real-time image feed captured by a rear-facing camera 26A is received by an augmented reality module 18, which identifies a virtual surface 34 that corresponds to a physical surface appearing in the image 27A by identifying surface features 36 of the virtual surface 34. The augmented reality module 18 may optionally perform simultaneous localization and mapping (SLAM) within the physical environment, thereby generating a map of the physical environment that includes a mesh of reconstructed surfaces, including the virtual surface 34. The identified virtual surface 34 may be included in the SLAM data 32 that contains the map generated by the augmented reality module 18.

[0019] In order to draw the avatars 132a, 132b, 132c to be superimposed on the image 27A, the three-dimensional engine 20 can receive the SLAM data 32 of the first computing device 12A, the posture information 108a-108c of all users in the image 27A (the first user posture information 108a of the first user, the second user posture information 108b of the second user, and the third user posture information 108c of the third user), and the avatar data objects 106a-106c of all users in the image 27A (the first user avatar data object 106a of the first user, the second user avatar data object 106b of the second user, and the third user avatar data object 106c of the third user), and then calculate the position and posture of some or all of the users' avatars 132a-132c (the first user avatar 132a, the second user avatar 132b, the third user avatar 132c) relative to the identified virtual surface 34. The three-dimensional engine 20 then renders some or all of the user's avatars 132a-132c, thereby displaying the user's rendered avatars 132a-132c superimposed on the image 27A on the display 22A of the first computing device 12A. The three-dimensional engine 20 may be, for example, UNITY3D, UNREALENGINE, or EffectSDK.

[0020] Next, the augmented reality program 14 is executed to loop through the steps of receiving updated second user pose information 126 of the second user via the computer network 136, receiving an updated image 27A from a real-time image feed captured by the rear camera 26A of the first computing device 12A, calculating a position and pose of a second user avatar 132b relative to the virtual surface 34 based on the updated second user pose information 126, and drawing the second user avatar data object 106b, thereby displaying the second user avatar 132b of the second user superimposed on the updated image 27A. The second user avatar 132b poses according to the updated second user pose information 126 at each loop run, while the second user avatar 132b remains anchored to the virtual surface 34, which remains world-locked to the physical surface in the image 27A in the real-world three-dimensional environment.

[0021] A similar loop feedback process is implemented for the other avatars 132a, 132c, so that while the first user avatar 132a remains anchored to the virtual surface 34, the first user avatar 132a poses according to the first user posture information 108a updated each time the loop runs, and while the third user avatar 132c remains anchored to the virtual surface 34, the third user avatar 132c poses according to the third user posture information 108c updated each time the loop runs.

[0022] The second user pose information 120 and the updated second user pose information 126 may include second facial tracking data 122, 128, respectively. The processor 28 may be configured to execute the three-dimensional engine 20 to render a second user avatar data object 132b of the second user based on the second facial tracking data 122, 128, thereby displaying the second user avatar 132b of the second user posing with a face corresponding to the facial expression of the second user. The face of the second user avatar 132b may be rendered so as to change appearance as the second facial tracking data 128 changes while the continuous loop is running. A similar loop feedback process may also be implemented for the third user avatar 132c, such that the third user avatar 132c is rendered so as to change appearance as the facial tracking data of the third user pose information 108c changes while the continuous loop is running.

[0023] A similar cyclic feedback process may also be implemented for the first user avatar 132a. The first user pose information 108a may include the first facial tracking data 138. The processor 28 may be configured to execute the three-dimensional engine 20 to render the first user avatar data object 106a of the first user based on the first facial tracking data 138 of the first user pose information 108a, thereby displaying the first user avatar 132a of the first user posing with a face corresponding to the facial expression of the first user. The face of the first user avatar 132a may be drawn so as to change appearance as the first facial tracking data 138 changes while the continuous loop is running.

[0024] The second user pose information 120 and the updated second user pose information 126 may include the second skeletal tracking data 124, 130, respectively. The processor 28 is configured to execute the three-dimensional engine 20 to draw the body of the second user avatar 132b based on the second skeletal tracking data 124, 130 so as to change the appearance as the second skeletal tracking data 124, 130 changes during the continuous loop operation. A similar loop feedback process may also be implemented for the third user avatar 132c, so that the body of the third user avatar 132c is drawn so as to change the appearance as the skeletal tracking data of the third user pose information 108c changes during the continuous loop operation.

[0025] A similar loop feedback process may also be implemented for the first user avatar 132a. The first user gesture information 108a may include the first skeleton tracking data 140. The processor 28 may be configured to execute the three-dimensional engine 20 to draw the body of the first user avatar 132a based on the first skeleton tracking data 140 so as to change the appearance as the first skeleton tracking data 140 changes when the continuous loop is run.

[0026] In response to generating the first user gesture information 108a, the processor 28 is configured to render the first user avatar data object 106a of the first user, thereby displaying the first user avatar 132a of the first user superimposed on the updated image 27A such that the first user avatar 132a poses according to the first user gesture information 108a.

[0027] Although the gesture tracking module 16 is depicted in this example as receiving the front camera image 25A from the front camera 24A, alternatively or additionally, the gesture tracking module 16 may be configured to receive the rear camera image 27A from the rear camera 26A, such that the processor 28 is configured to execute the gesture tracking module 16 to identify tracking features in the image 27A and the updated image 27A captured by the rear camera 26A and the first user image 25A of the first user, generate first user gesture information 108a of the first user based on the identified tracking features, and send the first user gesture information 108a to the second user computing device 12B of the second user via the computer network 136. In this example, the front camera 24A and the rear camera 26A would be operated to capture the first user image 25A and the updated image 27A, respectively, in real time, with the timestamps synchronizing the identified tracking features in the image 27A and the updated image 27A with the identified tracking features in the first user image 25A of the first user.

[0028] In one scenario, users participate in a real-time augmented reality session 134 hosted by a server computing system 100 of a social media platform, where each user has their own associated account. Figure 1In the example of FIG. 1 , a first computing device 12A of a first user, a second computing device 12B of a second user, and a third computing device 12C of a third user each execute an instance of an augmented reality avatar program 15 configured to communicate with a server computing system 100 of a social media platform, and also execute instances of an augmented reality program 14, a gesture tracking module 16, an augmented reality module 18, and a three-dimensional engine 20. First user gesture information 108a of the first user, first user avatar data object 106a of the first user, second user gesture information 120, updated second user gesture information 126, and second user avatar data object 106b of the first user are exchanged between the augmented reality avatar program 15 executed on the first computing device 12A and the augmented reality avatar program executed on the second computing device 12B via the server computing system 100. It will be appreciated that the gesture information of each user is continuously sampled in a control loop for a real-time augmented reality session 134. Therefore, to distinguish the first capture of the second user gesture information 120, the second user gesture information 120 may be referred to as initial second user gesture information.

[0029] like Figure 1 and Figure 2 As shown, the avatar data objects and gesture information may be packaged into a data container or configuration file by server computing device 100, which sends the data container to a corresponding computing device of computing system 10. The avatar data objects may be sent to server computing device 100 by computing devices 12A-12C in a configuration file, which is a descriptive file that specifies a list of characteristics and attributes of the avatar. For example, the configuration file may specify hair color, eye color, clothing, accessories, virtual skeleton structure, body skin shape, facial expression control points, etc.

[0030] exist Figure 1 In the example of FIG. 1 , server computing device 100 sends first avatar data container 118a including second user avatar data object 106b and third user avatar data object 106c to first computing device 12A, and also sends first gesture data container 116a including second user gesture information 108b and third user gesture information 108c to first computing device 12A. Second computing device 12B sends second configuration file 104b including second user avatar data object 106b to server computing device 100, and third computing device 12C sends third configuration file 104c including third user avatar data object 106c to server computing device 100.

[0031] Figure 2 The server computing device 100 and Figure 1Schematic diagram of three computing devices 12A-12C exchanging gesture information and avatar data objects. The server computing device 100 includes a processor 114, a non-volatile storage device 102 operatively coupled to the processor 114, and an avatar hosting program 110 stored in the non-volatile storage device 102, which can be loaded into the volatile memory 112 and executed by the processor 114 of the server computing device 100.

[0032] During the initialization process, when the avatars 132a-132c are initially drawn on the displays 22A-22C of the computing devices 12A-12C, the avatar hosting program 110 is configured to host the augmented reality session 134 to receive second user pose information 108b and a second configuration file 104b including a second user avatar data object 106b from the second computing device 12B of the second user. The server computing device also receives initial first user pose information 108a and a first configuration file 104a including the first user avatar data object 106a from the first computing device 12A of the first user, and receives initial third user pose information 108c and a third configuration file 104c including the third user avatar data object 106c from the third computing device 12C of the third user. Then, the avatar hosting program 110 stores the first configuration file 104a, the second configuration file 104b and the third configuration file 104c in the avatar configuration database 104, and stores the initial first user gesture information 108a, the second user gesture information 108b and the initial third user gesture information 108c in the avatar gesture information database 108.

[0033] In response to receiving the first request from the first computing device 12A, the avatar hosting program 110 sends the second user avatar data object 106b and the second user pose information 108b to the first computing device 12A. The first request may be the first profile 104a and the first user pose information 108a of the first user. In this example, where the avatars of three users are to be drawn on the display 22A of the first computing device 12A, the avatar hosting program 110 sends the first avatar data container 118a containing the second user avatar data object 106b and the third user avatar data object 106c, and the first pose data container 116a containing the second user pose information 108b and the third user pose information 108c to the first computing device 12A. This causes the first computing device 12A to draw the second user avatar data object 106b and the third user avatar data object 106c to display the second user avatar and the third user avatar on the display 22A.

[0034] In response to receiving the second request from the second computing device 12B, the avatar hosting program 110 sends the first user avatar data object 106a and the initial first user pose information 108a to the second computing device 12B. The second request may be the second profile 104b and the second user pose information 108b of the second user. In this example, where the avatars of three users are to be drawn on the display 22B of the second computing device 12B, the avatar hosting program 110 sends the second computing device 12B a second avatar data container 118b containing the first user avatar data object 106a and the third user avatar data object 106c, and a second pose data container 116b containing the first user pose information 108a and the third user pose information 108c. This causes the second computing device 12B to draw the first user avatar data object 106a and the third user avatar data object 106c to display the first user avatar and the third user avatar on the display 22B.

[0035] In response to receiving the third request from the third computing device 12C, the avatar hosting program 110 sends the third user avatar data object 106c and the initial third user pose information 108c to the third computing device 12C. The third request may be the third profile 104c and the third user pose information 108c of the third user. In this example, where the avatars of three users are to be drawn on the display 22C of the third computing device 12C, the avatar hosting program 110 sends the third avatar data container 118c containing the first user avatar data object 106a and the second user avatar data object 106b, and the third pose data container 116c containing the first user pose information 108a and the second user pose information 108b to the third computing device 12C. This causes the third computing device 12C to draw the first user avatar data object 106a and the second user avatar data object 106b to display the second user avatar and the third user avatar on the display 22C.

[0036] After the initialization process, a runtime process is performed in which the avatar is continuously redrawn by the display of the computing device using the continuously updated posture information in a loop. In the loop, the processor 114 of the server computing device 100 is configured to receive updated second user posture information 108b of the second user from the second computing device 12B. In response to receiving the updated second user posture information 108b, the processor 114 sends the updated second user posture information 108b to the first computing device 12A and the third computing device 12C, thereby causing the first computing device 12A to use the updated second user posture information 108b to draw the second user avatar data object 106b and display the second user avatar on the display 22A, and causing the third computing device 12C to use the updated second user posture information 108b to draw the second user avatar data object 106b and display the second user avatar on the display 22C. Then, updated first user pose information 108a of the first user is received from first computing device 12A, and updated third user pose information 108c of the third user is received from third computing device 12C. In response to receiving updated first user pose information 108a and updated third user pose information 108c, updated first user pose information 108a and updated third user pose information 108c are sent to second computing device 12B, thereby causing second computing device 12B to draw first user avatar data object 106a using updated first user pose information 108a, draw third user avatar data object 106c using updated third user pose information 108c, and display the first user avatar and the third user avatar on display 22B.

[0037] It should be understood that a similar runtime process may be performed on the first computing device 12A to draw the second user avatar data object 106b using the updated second user pose information 108b, draw the third user avatar data object 106c using the updated third user pose information 108c, and display the second user avatar and the third user avatar on the display 22A. Similarly, a similar runtime process may be performed on the third computing device 12C to draw the second user avatar data object 106b using the updated second user pose information 108b, draw the first user avatar data object 106a using the updated first user pose information 108a, and display the first user avatar and the second user avatar on the display 22C.

[0038] Initial first user pose information 108a and updated first user pose information 108a may include first facial tracking data, second user pose information 108b and updated second user pose information 108b may include second facial tracking data, and third user pose information 108c may include third facial tracking data. For example, second computing device 12B may be caused to draw first user avatar data object 106a of the first user based on the first facial tracking data, thereby displaying the first user avatar of the first user posing with a face corresponding to the facial expression of the first user. First computing device 12A may be caused to draw second user avatar data object 106b of the second user based on the second facial tracking data, thereby displaying the second user avatar of the second user posing with a face corresponding to the facial expression of the second user on display 22A.

[0039] refer to Figure 3 and Figure 4 , an example use case illustrating various aspects of the present disclosure will now be presented. Figure 3 As shown, the first user 202 may be standing in a restaurant 212 in the real-world three-dimensional environment 200, holding a first computing device 12A, which in this example may be in the form of a smartphone. However, it should be understood that the computing device is not particularly limited to a smartphone, but may be in any form that allows a user to view an augmented reality image overlaid on the real-world three-dimensional environment 200, such as a dedicated smartphone and tablet device, a mixed reality head mounted device (HMD), etc.

[0040] The first computing device 12A performs SLAM mapping within the physical environment 200 of the restaurant 212, thereby generating a map of the restaurant 212, which includes a mesh of reconstructed virtual surfaces and a virtual surface 234 corresponding to the physical surface 210, which in this example use case is the surface of the table. The virtual surface 234 can be a plane (as shown) or a curved surface, and is typically identified by tracking visual features in the rear camera image using an optical flow or depth camera. Object recognition techniques can also be used to identify virtual surfaces. As an alternative to virtual surfaces, suitable anchor points can be programmatically identified in the rear camera image for anchoring each avatar. The first computing device 12A receives second user posture information of the second user from the second computing device and third user posture information of the third user from the third computing device.

[0041] First computing device 12A calculates the position and pose of first user avatar 132a, second user avatar 132b, and third user avatar 132c relative to the identified virtual surface, which is world-locked to physical surface 210 in real-world three-dimensional environment 200 of restaurant 212. First computing device 12A then renders first user avatar 132a, second user avatar 132b, and third user avatar 132c to display rendered avatars 132a-132c superimposed on image 27A on display 22A of first computing device 12A. First user 202 viewing display 22A of first computing device 12A then sees full avatars 132a-132c rendered on display 22A at appropriate screen positions and in appropriate poses, which may give first user 202 the illusion that avatars 132a-132c are standing in real-world three-dimensional environment 200 of restaurant 212, as shown in FIG. Figure 3 By drawing other user avatars in the same room as the user's avatar, a sense of belonging can be added to the interaction.

[0042] refer to Figure 5 , Figure 3 and Figure 4 The example use case shown is for demonstrating changes in the appearance of avatars 132a-132c on display 22A of first computing device 12A during runtime such that one or more portions of avatars 132a-132c are associated with motion. First computing device 12A is configured to draw a body of first user avatar 132a based on initial first user pose information 120a, a body of second user avatar 132b based on second user pose information 120b, and a body and face of third user avatar 132c based on initial third user pose information 120c during initialization. During runtime, as indicated by the dashed lines, the first computing device 12A is configured to draw the body of the first user avatar 132a based on the updated first user pose information 126a (the first user turns her head), draw the body of the second user avatar 132b based on the updated second user pose information 126b (the second user gives a thumbs up), and draw the body and face of the third user avatar 132c based on the updated third user pose information 126c (the third user raises his hand and frowns).

[0043] Thus, the faces and bodies of the first user avatar 132a, the second user avatar 132b, and the third user avatar 132c are drawn so as to change appearance during runtime as the updated first user pose information 126a, the updated second user pose information 126b, and the updated third user pose information 126c change during the continuous loop runtime. Thus, the drawing of the avatars 132a-132c achieves an enhanced sense of reality through the realistic drawing of the faces and bodies of the avatars 132a-132c, so that users using their respective computing devices have a more intimate and shared experience in their online interactions with each other during the augmented reality session, where facial expressions and body language can be used in addition to voice and text for users to communicate with each other, and a sense of physical proximity and belonging between participants of the augmented reality session is enhanced.

[0044] Figure 6 1 shows a flow chart of a method 300 for the initialization process when an avatar is initially drawn on a display of a computing device. The following description of the method 300 is based on the above and Figure 1-Figure 5 It should be understood that method 300 may also be performed in other contexts using other suitable hardware and software components.

[0045] In step 302, at a first display device, a first configuration file having a first user avatar data object is sent to a server computing device. In step 304, at a second display device, a second configuration file having a second user avatar data object is sent to a server computing device. In step 306, at a server computing device, a first configuration file containing the first user avatar data object and a second configuration file containing the second user avatar data object are received and stored. In step 308, at the server computing device, an avatar data object to be sent to the first display device is selected and sent, and an avatar data object to be sent to the second display device is selected and sent. In step 310, at the first display device, the selected avatar data object is received from the server computing device. In step 314, at the first display device, the position and pose of the selected avatar data objects (including the first user avatar data object and the second user avatar data object) relative to a virtual surface are calculated using the selected avatar data object. In step 318, the selected avatar data objects (including the first user avatar data object and the second user avatar data object) are drawn on a display based on the calculated position and pose.

[0046] At step 312, at the second display device, the selected avatar data object is received from the server computing device. At step 316, at the second display device, the position and pose of the selected avatar data objects (including the first user avatar data object and the second user avatar data object) relative to the virtual surface are calculated using the selected avatar data object. At step 320, the selected avatar data objects (including the first user avatar data object and the second user avatar data object) are drawn on the display based on the calculated position and pose.

[0047] Figure 7 A flow chart of a method 400 for a runtime process is shown, wherein an avatar is continuously redrawn by a display of a computing device using continuously updated posture information in a loop. The following description of method 400 is based on the above and other related descriptions. Figure 1-Figure 5 It should be understood that the method 400 may also be performed in other contexts using other suitable hardware and software components.

[0048] At step 402, at a first display device, a first user image of a first user is captured with a camera. At step 404, at the first display device, a tracking feature of the first user image is identified. At step 406, at the first display device, first user gesture information is generated based on the identified tracking feature. At step 408, at the first display device, the first user gesture information is sent to a server computing device.

[0049] At step 410, a second user image of a second user is captured with a camera at the second display device. At step 412, tracking features of the second user image are identified at the second display device. At step 414, second user gesture information is generated at the second display device based on the identified tracking features. At step 416, the second user gesture information is sent to a server computing device at the second display device.

[0050] At step 418, the first user gesture information and the second user gesture information are received and stored at the server computing device as updated gesture information for the first user and the second user, respectively. At step 420, at the server computing device, the updated gesture information is sent to the first display device and the second display device, respectively.

[0051] At the first display device, steps 422, 426, and 428 are performed. In step 422, the updated second user pose information is received. In step 426, the positions and poses of the first user avatar and the second user avatar are calculated based on the first user pose information and the second user pose information. In step 428, the first user avatar and the second user avatar are drawn on the display based on the calculated positions and poses.

[0052] At the second display device, steps 424, 430, and 432 are performed. In step 424, the updated first user pose information is received. In step 430, the positions and poses of the first user avatar and the second user avatar are calculated based on the first user pose information and the second user pose information. In step 432, the first user avatar and the second user avatar are drawn on the display based on the calculated positions and poses.

[0053] Figure 8 5 shows a flow chart of method 500. The following description of method 500 is based on the above and Figure 1-Figure 5 It should be understood that method 500 may also be performed in other contexts using other suitable hardware and software components.

[0054] At step 502, a second user avatar data object and second user gesture information of a second user are received via a computer network. At step 504, an image is received from a real-time image feed captured by a camera of a first computing device. At step 506, a virtual surface corresponding to a physical surface appearing in the image is identified.

[0055] At step 508, a position and a pose of the second user avatar relative to the virtual surface are calculated based on the second user pose information. At step 510, the second user avatar data object is rendered, thereby displaying the second user avatar of the second user superimposed on the image, the second user avatar being anchored to the virtual surface according to the calculated position and pose of the second user avatar, and the virtual surface being world locked to a physical surface in the image in the real world three-dimensional environment.

[0056] The following steps 512, 514, 516 and 518 are performed in a loop. In step 512, updated second user pose information of the second user is received via the computer network. In step 514, an updated image is received from a real-time image feed captured by a camera of the first computing device. In step 516, a position and pose of the second user avatar relative to the virtual surface is calculated based on the updated second user pose information. In step 518, a second user avatar data object is drawn, thereby displaying the second user avatar of the second user superimposed on the updated image, the second user avatar posing according to the updated second user pose information at each run of the loop, while the second user avatar remains anchored to the virtual surface, and the virtual surface remains world-locked to the physical surface in the image in the real-world three-dimensional environment.

[0057] It should be understood that methods 300, 400, and 500 are provided by way of example and are not intended to be limiting. Figure 6-Figure 8In addition and / or alternative steps to those shown. Furthermore, it should be understood that methods 300, 400, and 500 may be performed in any suitable order.

[0058] The above-described system and method allows multiple users in an augmented reality session to use their respective computing devices to see each other's avatars on the displays of their devices, with their positions and appearances changing in real time to correspond to changes in the positions and appearances of the users. The rendering of the avatar's face and body is enhanced so that users participating in the augmented reality session can have a more intimate and shared experience in their online interactions with each other. This enhances the sense of physical proximity and belonging, which is further enhanced when facial expressions and body language are used in addition to voice and text for users to communicate with each other via their respective computing devices.

[0059] In some embodiments, the methods and processes described herein may be bound to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0060] Fig. 9 A non-limiting embodiment of a computing system 600 is schematically shown, which can implement one or more of the above methods and processes. The computing system 600 is shown in simplified form. The computing system 600 can respectively embody the above and Figure 1-Figure 5 The illustrated first computing device 12A, second computing device 12B, third computing device 12C, fourth computing device 12D, and server computing device 100. Computing system 600 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smartphones), and / or other computing devices, as well as wearable computing devices (such as smart watches and head-mounted augmented reality devices).

[0061] The computing system 600 includes a logic processor 602, a volatile memory 604, and a non-volatile storage device 606. The computing system 600 may optionally include a display subsystem 608, an input subsystem 610, a communication subsystem 612, and / or Fig. 9 Other components not shown.

[0062] Logical processor 602 includes one or more physical devices configured to execute instructions. For example, a logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, transform the state of one or more components, implement technical effects, or otherwise achieve desired results.

[0063] The logical processor may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, the logical processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of the logical processor 602 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel and / or distributed processing. Optionally, the individual components of the logical processor may be distributed between two or more independent devices, which may be remotely located and / or configured for coordinated processing. Various aspects of the logical processor may be virtualized and executed by a remotely accessible networked computing device configured with a cloud computing configuration. In this case, these virtualized aspects are run on different physical logical processors of various different machines, which will be understood.

[0064] The non-volatile storage device 606 includes one or more physical devices configured to store instructions executable by a logical processor to implement the methods and processes described herein. When such methods and processes are implemented, the state of the non-volatile storage device 606 can be transformed—for example, to store different data.

[0065] The non-volatile storage device 606 may include a removable and / or built-in physical device. The non-volatile storage device 606 may include an optical memory (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), a semiconductor memory (e.g., ROM, EPROM, EEPROM, FLASH memory, etc.), and / or a magnetic memory (e.g., a hard disk drive, a floppy disk drive, a tape drive, MRAM, etc.) or other mass storage device technology. The non-volatile storage device 606 may include a non-volatile, dynamic, static, read / write, read-only, sequential access, location addressable, file addressable, and / or content addressable device. It will be understood that the non-volatile storage device 606 is configured to save instructions even when the non-volatile storage device 606 is powered off.

[0066] The volatile memory 604 may include physical devices, including random access memory. The volatile memory 604 is typically used by the logical processor 602 to temporarily store information during the processing of software instructions. It should be understood that when the volatile memory 604 is powered off, the volatile memory 604 typically does not continue to store instructions.

[0067] Aspects of the logic processor 602, volatile memory 604, and non-volatile storage device 606 may be integrated together into one or more hardware logic components. For example, such hardware logic components may include field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASIC / AASIC), program and application specific standard products (PSSP / ASP), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

[0068] The terms "module", "program", and "engine" may be used to describe an aspect of the computing system 600 that is typically implemented in software by a processor to use portions of volatile memory to perform a specific function that involves a conversion process that specifically configures the processor to perform the function. Thus, a module, program, or engine may be instantiated via a logical processor 602 that executes instructions stored by a non-volatile storage device 606 using portions of volatile memory 604. It will be appreciated that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module", "program", and "engine" may cover individuals or groups of executable files, data files, libraries, drivers, scripts, database records, and the like.

[0069] When included, the display subsystem 608 can be used to present a visual representation of the data stored by the non-volatile storage device 606. The visual representation can take the form of a graphical user interface (GUI). Since the methods and processes described herein change the data stored by the non-volatile storage device, and thus transform the state of the non-volatile storage device, the state of the display subsystem 608 can also be transformed to visually represent the changes in the underlying data. The display subsystem 608 can include one or more display devices utilizing almost any type of technology. Such a display device can be combined with the logical processor 602, the volatile memory 604, and / or the non-volatile storage device 606 in a shared housing, or such a display device can be a peripheral display device.

[0070] When included, the input subsystem 610 may include or interface with one or more user input devices (such as a keyboard, mouse, touch screen, or game controller). In some embodiments, the input subsystem may include or interface with selected natural user input (NUI) components. Such components may be integrated or peripheral, and the conversion and / or processing of input actions may be handled on-board or off-board. Example NUI components may include microphones for voice and / or sound recognition; infrared, color, stereo, and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers, and / or gyroscopes for motion detection and / or intent recognition; and electric field sensing components for assessing brain activity; and / or any other suitable sensors.

[0071] When included, the communication subsystem 612 can be configured to couple the various computing devices described herein to each other and to other devices. The communication subsystem 612 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem can be configured for communication via a wireless telephone network, or a wired or wireless local area network or wide area network (such as an HDMI connection on Wi-Fi). In some embodiments, the communication subsystem can allow the computing system 600 to send messages to other devices and / or receive messages from other devices via networks such as the Internet.

[0072] The following paragraphs provide additional support for the claims of the subject application. One aspect provides a computing system comprising: a processor and memory of a computing device of a first user, the processor being configured to use a portion of the memory to execute an augmented reality (AR) avatar program to: receive a second user avatar data object and second user pose information of a second user via a computer network; receive an image from a real-time image feed captured by a camera of the computing device; identify a virtual surface corresponding to a physical surface appearing in the image; calculate a position and pose of the second user avatar relative to the virtual surface based on the second user pose information; draw the second user avatar data object, thereby displaying a second user avatar of the second user superimposed on the image, the second user avatar being anchored to the virtual surface and posing according to the calculated position and pose of the second user avatar, the virtual surface being world-locked to a physical surface in the image in a real-world three-dimensional environment. In this aspect, additionally or alternatively, the second user pose information is initial second user pose information for a session of the AR avatar program; and the image is an initial image for a session of the AR avatar program. In this aspect, additionally or alternatively, the AR avatar program is executed to loop: receive updated second user pose information of the second user via a computer network; receive an updated image from a real-time image feed captured by a camera of the computing device; calculate a position and pose of the second user avatar relative to the virtual surface based on the updated second user pose information; and draw the second user avatar data object, thereby displaying the second user avatar superimposed on the updated image and posing according to the updated pose information at each loop run, while the second user avatar remains anchored to the virtual surface, the virtual surface remains world-locked to a physical surface in the image in the real-world three-dimensional environment. In this aspect, additionally or alternatively, the second user pose information and the updated second user pose information include second facial tracking data; the processor is configured to draw the second user avatar data object of the second user based on the second facial tracking data, thereby displaying the second user avatar of the second user posing with a face corresponding to the facial expression of the second user; and the face of the second user avatar is drawn so as to change appearance as the second facial tracking data changes at consecutive loop runs. In this aspect, additionally or alternatively, the second user gesture information includes second facial tracking data; and the processor is configured to draw a second user avatar data object of the second user based on the second facial tracking data, thereby displaying the second user avatar of the second user posing with a face corresponding to the facial expression of the second user. In this aspect, additionally or alternatively, the second user gesture information includes second skeletal tracking data; and the processor is configured to draw the body of the second user avatar so as to change appearance as the second skeletal tracking data changes.In this aspect, additionally or alternatively, the camera is a rear-facing camera, and the computing device is a first computing device, the computing system further comprises a front-facing camera configured to capture a first-user image of a first user, the processor is configured to: identify tracking features in the first-user image of the first user, and generate first-user pose information of the first user based on the identified tracking features; and send the first-user pose information to a second computing device of a second user via a computer network. In this aspect, additionally or alternatively, the first-user pose information comprises first facial tracking data; and the processor is configured to draw a first-user avatar data object of the first user based on the first facial tracking data, thereby displaying the first-user avatar of the first user posing with a face corresponding to the facial expression of the first user. In this aspect, additionally or alternatively, the processor is further configured to draw the first-user avatar data object of the first user, thereby displaying the first-user avatar of the first user superimposed on the image, the first-user avatar posing according to the first-user pose information. In this aspect, additionally or alternatively, the front camera and the rear camera are operated to capture the first user image and the image, respectively, in real time, and the timestamp synchronizes the identified tracking features in the image with the identified tracking features in the first user image of the first user. In this aspect, additionally or alternatively, the processor is configured to: identify the tracking features in the image captured by the rear camera and the first user image of the first user; generate first user gesture information of the first user based on the identified tracking features; and send the first user gesture information to the second computing device of the second user via a computer network. In this aspect, additionally or alternatively, the computing device is a first computing device; the first user and the second user each have an associated account on a social media platform; the first computing device and the second computing device each execute an instance of an AR avatar program, the instance being configured to communicate with a server computing system of the social media platform; and the first user gesture information of the first user, the first user avatar data object of the first user, the second user gesture information, and the second user avatar data object are exchanged between the AR avatar program executed on the first computing device and the AR avatar program executed on the second computing device via the server computing system.

[0073] On the other hand, a method is provided for a client program executed on a computing device operated by a first user, the method comprising the steps of: receiving a second user avatar data object and second user posture information of a second user via a computer network; receiving an image from a real-time image feed captured by a camera of the computing device; identifying a virtual surface corresponding to a physical surface appearing in the image; calculating a position and posture of the second user avatar relative to the virtual surface based on the second user posture information; and drawing the second user avatar data object, thereby displaying the second user avatar of the second user superimposed on the image, the second user avatar being anchored to the virtual surface according to the calculated position and posture of the second user avatar, and the virtual surface being world-locked to a physical surface in the image in a real-world three-dimensional environment. In this aspect, additionally or alternatively, the method further comprises the steps of: cyclically: receiving updated second user pose information of the second user via a computer network; receiving an updated image from a real-time image feed captured by a camera of the computing device; calculating a position and pose of the second user avatar relative to the virtual surface based on the updated second user pose information; and drawing a second user avatar data object, thereby displaying a second user avatar of the second user superimposed on the updated image, the second user avatar posing according to the updated second user pose information at each run of the loop, while the second user avatar remains anchored to the virtual surface, the virtual surface remaining world-locked to a physical surface in the image in the real-world three-dimensional environment. In this aspect, additionally or alternatively, the second user pose information comprises second facial tracking data; and the method further comprises drawing a second user avatar data object of the second user based on the second facial tracking data, thereby displaying the second user avatar of the second user posing with a face corresponding to a facial expression of the second user. In this aspect, additionally or alternatively, the computing device is a first computing device; and the method further comprises the steps of: capturing a first user image of a first user; identifying tracking features in the first user image of the first user; generating first user gesture information of the first user based on the identified tracking features; and sending the first user gesture information to a second computing device of a second user via a computer network. In this aspect, additionally or alternatively, the first user gesture information comprises first facial tracking data; and the method further comprises drawing a first user avatar data object of the first user based on the first facial tracking data, thereby displaying the first user avatar of the first user posing with a face corresponding to a facial expression of the first user.

[0074] On the other hand, a server computing device is provided, which includes: a processor; a non-volatile storage device operably coupled to the processor; and an avatar hosting program stored in the non-volatile storage device and executed by the processor of the server computing device, the avatar hosting program being configured to host an avatar session to: receive and store a second user avatar data object and second user posture information of a second user from a second computing device; in response to receiving a first request for a second user avatar from a first computing device, send the second user avatar data object and the second user posture information to the first computing device, thereby causing the first computing device to draw the second user avatar data object and display the second user avatar; receive and store a first user avatar data object and initial first user posture information of a first user from the first computing device; in response to receiving a second request for a first user avatar from the second computing device, send the second user avatar data object and the second user posture information to the first computing device, thereby causing the first computing device to draw the second user avatar data object and display the second user avatar The device sends a first user avatar data object and initial first user posture information, thereby causing a second computing device to draw the first user avatar data object and display the first user avatar; and cyclically: receiving updated second user posture information of the second user from the second computing device; and in response to receiving the updated second user posture information, sending the updated second user posture information to the first computing device, thereby causing the first computing device to use the updated second user posture information to draw the second user avatar data object and display the second user avatar; receiving updated first user posture information of the first user from the first computing device; and in response to receiving the updated first user posture information, sending the updated first user posture information to the second computing device, thereby causing the second computing device to use the updated first user posture information to draw the first user avatar data object and display the first user avatar. In this aspect, additionally or alternatively, the initial first user pose information and the updated first user pose information include first facial tracking data; the second computing device is caused to draw a first user avatar data object of the first user based on the first facial tracking data, thereby displaying the first user avatar of the first user posing with a face corresponding to the facial expression of the first user; the second user pose information and the updated second user pose information include second facial tracking data; and the first computing device is caused to draw a second user avatar data object of the second user based on the second facial tracking data, thereby displaying the second user avatar of the second user posing with a face corresponding to the facial expression of the second user.In this aspect, additionally or alternatively, the initial first user posture information and the updated first user posture information include first skeletal tracking data; the second computing device is caused to draw a first user avatar data object of the first user based on the first skeletal tracking data, thereby displaying the first user avatar of the first user posing with a body, and the body changes appearance as the first skeletal tracking data changes while the continuous loop is running; the second user posture information and the updated second user posture information include second skeletal tracking data; and the first computing device is caused to draw a second user avatar data object based on the second skeletal tracking data of the second user, thereby displaying the second user avatar of the second user posing with a body, and the body changes appearance as the second skeletal tracking data changes while the continuous loop is running.

[0075] It should be understood that the configuration and / or method described herein are exemplary in nature, and these specific embodiments or examples should not be considered restrictive, because many variations are possible. The specific routines or methods described herein can represent one or more of any number of processing strategies. Therefore, the various actions shown and / or described can be performed in the order shown and / or described, performed in other orders, performed in parallel, or omitted. Similarly, the order of the above-mentioned processing can be changed.

[0076] The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.

[0077] To the extent the terms "includes," "including," "has," "contains" and variations thereof are used herein, such terms are intended to be inclusive in a manner similar to the term "comprises" as an open transitional word without excluding any additional or other elements.

[0078] It should be understood that the configuration and / or method described herein are exemplary in nature, and these specific embodiments or examples should not be considered restrictive, because many variations are possible. The specific routines or methods described herein can represent one or more of any number of processing strategies. Therefore, the various actions shown and / or described can be performed in the order shown and / or described, performed in other orders, performed in parallel, or omitted. Similarly, the order of the above-mentioned processing can be changed.

[0079] The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.

Claims

1. A computing system comprising: A processor and memory of a computing device of a first user, the processor configured to use a portion of the memory to execute an augmented reality (AR) avatar program to: receiving, via the computer network, a second user avatar data object and second user gesture information of a second user; receiving an image from a real-time image feed captured by a camera of the computing device; identifying a virtual surface corresponding to a physical surface appearing in the image; calculating a position and a posture of a second user avatar relative to the virtual surface based on the second user posture information; and rendering the second user avatar data object to display the second user avatar of the second user superimposed on the image, the second user avatar being anchored and posed to the virtual surface according to the calculated position and pose of the second user avatar, the virtual surface being world locked to the physical surface in the image in the real-world three-dimensional environment.

2. The computing system of claim 1, wherein The second user gesture information is initial second user gesture information of the session of the AR avatar program; and The image is an initial image of the session of the AR avatar program.

3. The computing system of claim 1 , wherein the AR avatar program is executed to cyclically: receiving updated second user gesture information of the second user via the computer network; receiving an updated image from the real-time image feed captured by the camera of the computing device; calculating a position and a posture of the second user avatar relative to the virtual surface based on the updated second user posture information; as well as The second user avatar data object is drawn to display the second user avatar superimposed on the updated image and posed according to the updated pose information on each loop run, while the second user avatar remains anchored to the virtual surface, and the virtual surface remains world locked to the physical surface in the image in the real-world three-dimensional environment.

4. The computing system of claim 3, wherein the second user pose information and the updated second user pose information include second facial tracking data; the processor being configured to render the second user avatar data object of the second user based on the second facial tracking data, thereby displaying the second user avatar of the second user posing with a face corresponding to a facial expression of the second user; and The face of the second user avatar is rendered to change appearance as the second facial tracking data changes while running in a continuous loop.

5. The computing system of claim 1, wherein The second user gesture information includes second facial tracking data; and The processor is configured to render the second user avatar data object of the second user based on the second facial tracking data, thereby displaying the second user avatar of the second user posing with a face corresponding to a facial expression of the second user.

6. The computing system of claim 1, wherein The second user gesture information includes second skeletal tracking data; and The processor is configured to render a body of the second user avatar to change appearance as the second skeletal tracking data changes.

7. The computing system of claim 1, wherein the camera is a rear-facing camera and the computing device is a first computing device, the computing system further comprising: a front camera configured to capture a first user image of the first user, The processor is configured to: identifying tracking features in the first user image of the first user, and generating first user gesture information of the first user based on the identified tracking feature; as well as The first user gesture information is sent to a second computing device of the second user via the computer network.

8. The computing system of claim 7, wherein The first user gesture information includes first facial tracking data; and The processor is configured to render a first user avatar data object of the first user based on the first facial tracking data, thereby displaying the first user avatar of the first user posing with a face corresponding to a facial expression of the first user.

9. The computing system of claim 7, wherein the processor is further configured to render a first user avatar data object of the first user, thereby displaying the first user avatar of the first user superimposed on the image, the first user avatar posing according to the first user pose information.

10. The computing system of claim 9, wherein the front-facing camera and the rear-facing camera are operated to capture the first user image and the image, respectively, in real time, and a timestamp synchronizes the identified tracking features in the image with the identified tracking features in the first user image of the first user.

11. The computing system of claim 7, wherein The processor is configured to: identifying tracking features in the image captured by the rear-facing camera and the first user image of the first user; generating first user gesture information of the first user based on the identified tracking feature; as well as The first user gesture information is sent to the second computing device of the second user via the computer network.

12. The computing system of claim 1, wherein The computing device is a first computing device; The first user and the second user each have an associated account on a social media platform; The first computing device and the second computing device each execute an instance of the AR avatar program, the instance being configured to communicate with a server computing system of the social media platform; and The first user posture information of the first user, the first user avatar data object of the first user, the second user posture information and the second user avatar data object are exchanged between the AR avatar program executed on the first computing device and the AR avatar program executed on the second computing device via the server computing system.

13. A method for a client program executed on a computing device operated by a first user, the method comprising the steps of: receiving, via the computer network, a second user avatar data object and second user gesture information of a second user; receiving an image from a real-time image feed captured by a camera of the computing device; identifying a virtual surface corresponding to a physical surface appearing in the image; calculating a position and a posture of a second user avatar relative to the virtual surface based on the second user posture information; as well as The second user avatar data object is drawn to display the second user avatar of the second user superimposed on the image, the second user avatar being anchored to the virtual surface based on the calculated position and pose of the second user avatar, the virtual surface being world locked to the physical surface in the image in the real-world three-dimensional environment.

14. The method according to claim 13, further comprising: The following steps are involved: Cyclicly: receiving updated second user gesture information of the second user via the computer network; receiving an updated image from the real-time image feed captured by the camera of the computing device; calculating a position and a posture of the second user avatar relative to the virtual surface based on the updated second user posture information; as well as The second user avatar data object is drawn to display the second user avatar of the second user superimposed on the updated image, the second user avatar posing according to the updated second user pose information for each loop run, while the second user avatar remains anchored to the virtual surface, and the virtual surface remains world locked to the physical surface in the image in the real-world three-dimensional environment.

15. The method according to claim 13, wherein The second user gesture information includes second facial tracking data; and The method also includes rendering the second user avatar data object of the second user based on the second facial tracking data, thereby displaying the second user avatar of the second user posing with a face corresponding to a facial expression of the second user.

16. The method according to claim 13, wherein The computing device is a first computing device; and The method further comprises the following steps: capturing a first user image of the first user; identifying tracking features in the first user image of the first user; generating first user gesture information of the first user based on the identified tracking feature; as well as The first user gesture information is sent to a second computing device of the second user via the computer network.

17. The method according to claim 16, wherein The first user gesture information includes first facial tracking data; and The method also includes rendering a first user avatar data object of the first user based on the first facial tracking data, thereby displaying the first user avatar of the first user posing with a face corresponding to a facial expression of the first user.

18. A server computing device comprising: processor; a non-volatile storage device operably coupled to the processor; as well as an avatar hosting program stored in the non-volatile storage device and executed by the processor of the server computing device, wherein The avatar hosting program is configured to host an avatar session to: receiving and storing, from a second computing device, a second user avatar data object and second user gesture information of a second user; In response to receiving a first request for a second user avatar from a first computing device, sending the second user avatar data object and the second user gesture information to the first computing device, thereby causing the first computing device to draw the second user avatar data object and display the second user avatar; receiving and storing a first user avatar data object and initial first user gesture information of a first user from the first computing device; In response to receiving a second request for the first user avatar from the second computing device, sending the first user avatar data object and the initial first user pose information to the second computing device, thereby causing the second computing device to draw the first user avatar data object and display the first user avatar; as well as Cyclicly: receiving updated second user gesture information of the second user from the second computing device; as well as In response to receiving the updated second user gesture information, sending the updated second user gesture information to the first computing device, so that the first computing device uses the updated second user gesture information to draw the second user avatar data object and display the second user avatar; receiving updated first user gesture information of the first user from the first computing device; as well as In response to receiving the updated first user gesture information, the updated first user gesture information is sent to the second computing device, so that the second computing device draws the first user avatar data object using the updated first user gesture information and displays the first user avatar.

19. The server computing device of claim 18, wherein The initial first user pose information and the updated first user pose information include first facial tracking data; the second computing device being caused to render the first user avatar data object of the first user based on the first facial tracking data to display the first user avatar of the first user posing with a face corresponding to a facial expression of the first user; The second user pose information and the updated second user pose information include second facial tracking data; and The first computing device is caused to render the second user avatar data object of the second user based on the second facial tracking data, thereby displaying the second user avatar of the second user posing with a face corresponding to a facial expression of the second user.

20. The server computing device of claim 18, wherein The initial first user pose information and the updated first user pose information include first skeletal tracking data; the second computing device being caused to render the first user avatar data object of the first user based on the first skeletal tracking data to thereby display the first user avatar of the first user posing with a body that changes appearance as the first skeletal tracking data changes over successive run cycles; The second user gesture information and the updated second user gesture information include second skeletal tracking data; and The first computing device is caused to render the second user avatar data object based on the second skeletal tracking data of the second user, thereby displaying the second user avatar of the second user posing with a body that changes appearance as the second skeletal tracking data changes while running in a continuous loop.

Images