Computing system, computing method, server computing device, and program for rendering avatars

Abstract

A computing system comprising a processor is provided, the processor being configured to execute an augmented reality avatar program to receive, via a computer network, second user avatar data object and second user pose information, receive an image from a live image feed obtained by a camera of the computing device, identify a virtual plane corresponding to a physical plane appearing in the image, calculate a position and pose of a second user avatar with respect to the virtual plane based on the second user pose information, and display the second user avatar of the second user superimposed on the image by rendering the second user avatar data object. The second user avatar is anchored to the virtual plane according to the calculated position and pose of the second user avatar. The virtual plane is world-locked to the physical plane in the image within a real-world 3D environment.

Description

Technical Field

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS The present application claims priority from U.S. Patent Application No. 17 / 806,701 filed on June 13, 2022 (title of invention: computing system and computing method for rendering an avatar), the disclosure of which is incorporated herein by reference in its entirety.

Background Art

[0002] With the rapid proliferation of online platforms for human interaction, such as social media services, messaging services, and online game services, people increasingly use avatars to represent themselves in the online world. Broadly speaking, an avatar is an image representing a user in a virtual world, and is often shaped as a human, a cartoon character, or any object. Some avatars are two-dimensional, while others are three-dimensional. Some avatars are configured to move or change expressions in accordance with the user's movements and expressions captured by a camera. However, even for such dynamic avatars, it can be said that interaction with other people's avatars in video chat lacks, for example, a sense of solidarity between users. Therefore, there exists an opportunity to improve user experience when using avatars in computer-based interaction with other people.

Summary of the Invention

Means for Solving the Problems

[0003] In view of the above, a computing system is provided comprising a processor and memory of a first user's computing device. The processor is configured to receive a second user avatar data object and second user pose information of a second user via a computer network by executing an augmented reality avatar program using a portion of the memory. The processor is further configured to receive images from a live image feed acquired by the computing device's camera, identify virtual surfaces corresponding to physical surfaces appearing in the images, calculate the position and pose of the second user avatar relative to the virtual surface based on the second user pose information, and render the second user avatar data object to display the second user avatar of the second user superimposed on the image. The second user avatar is anchored to the virtual surface according to the calculated position and pose of the second user avatar. The virtual surface is world-locked to the physical surface in the image within a three-dimensional real-world environment.

[0004] This summary is provided to present a simplified excerpt of the concept, which will be further described in the embodiments for carrying out the invention described below. This summary is not intended to identify the main or basic features of the claimed subject matter, nor to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to an implementation that solves any or all of the defects described in any part of this disclosure. [Brief explanation of the drawing]

[0005] [Figure 1] This is a schematic diagram showing a computing system as an example of this disclosure.

[0006] [Figure 2] This is another schematic diagram of the computing system shown in Figure 1.

[0007] [Figure 3] Figure 1 illustrates an exemplary use case of the computing system according to an example of this disclosure, schematically showing that a real user is viewing three avatars displayed on a world-locked virtual surface at the top of a real-world table.

[0008] [Figure 4] An additional illustration of an exemplary use case relating to an example of the present disclosure, showing a close-up of a real-world user looking at a computing device displaying an augmented reality image including the three avatars and the real-world table in Figure 3.

[0009] [Figure 5] This is an illustration of an image displayed in an exemplary use case, Figure 3, which shows the movement of each avatar in response to user input, as an example of this disclosure.

[0010] [Figure 6] This flowchart shows a method for performing an initialization process for rendering an avatar, according to an exemplary embodiment of the present disclosure.

[0011] [Figure 7] This flowchart shows a method for performing a runtime process for rendering an avatar, according to an exemplary embodiment of the present disclosure.

[0012] [Figure 8] This is a flowchart illustrating a method for rendering an avatar according to an exemplary embodiment of the present disclosure.

[0013] [Figure 9] This figure shows an exemplary computing environment in this disclosure. [Modes for carrying out the invention]

[0014] Referring to Figures 1 and 2, the computing system 10 comprises a first computing device 12A for a first user, a second computing device 12B for a second user, and a third computing device 12C for a third user. The number of users and their computing devices is not particularly limited, and it will be understood that the number of computing devices in the computing system 10 may be three or more, or only two. The first computing device 12A for the first user comprises a processor 28, memory 30, a front camera 24A configured to acquire a first user image 25A of the first user, a rear camera 26A configured to acquire a live image feed and image 27A, and a display 22A configured to display image 27A. Since other computing devices within the computing system 10 also have hardware similar to that of the first computing device 12A, it will be understood that the second computing device 12B also has a processor, memory, a front camera 24B, a rear camera 26B, and a display 22B, and the third computing device 12C also has a processor, memory, a front camera 24C, a rear camera 26C, and a display 22C.

[0015] The processor 28 is configured to receive the second user avatar data object 106b and second user pose information 120 of a second user via the computer network 136 by executing an augmented reality program 14 using a portion of the memory 30. The augmented reality program 14 receives image 27A from a live image feed acquired by the rear camera 26A, identifies a virtual plane 34 corresponding to a physical plane appearing in image 27A, calculates the position and pose of the second user avatar 132b relative to the virtual plane 34 based on the second user pose information 120, and displays the second user avatar 132b of the second user superimposed on image 27A by rendering the second user avatar data object. The second user avatar 132b is anchored to the virtual plane 34 according to the calculated position and pose of the second user avatar 132b. The virtual plane 34 is world-locked to the physical plane of image 27A in the three-dimensional environment of the real world. The second user pose information 120 and image 27A may be the initial second user pose information and initial image of the augmented reality program 14 session, respectively.

[0016] The processor 28 also runs the pose tracking module 16, which receives a first user image 25A acquired by the front camera 24A, identifies tracking features in the first user image 25A of the first user, generates first user pose information 108a of the first user based on the identified tracking features, and transmits the first user pose information 108a to the second computing device 12B of the second user and to the three-dimensional engine 20 of the first computing device 12A via the computer network 136 in order to render and display the first user avatar 132a on the display 22A.

[0017] Image 27A from a live image feed acquired by the rear camera 26A is received by the augmented reality module 18, and the augmented reality module 18 identifies the virtual surface 34 corresponding to the physical surface appearing in image 27A by identifying the surface features 36 of the virtual surface 34. The augmented reality module 18 may optionally perform simultaneous location and mapping (SLAM) in the physical environment to create a map of the physical environment including a mesh of reconstructed surfaces containing the virtual surface 34. The identified virtual surface 34 may be included in the SLAM data 32, which includes the map generated by the augmented reality module 18.

[0018] In order to render avatars 132a, 132b, and 132c so that they can be overlaid on image 27A, the 3D engine 20 uses the SLAM data 32 from the first computing device 12A, the pose information 108a~c of all users in image 27A (first user pose information 108a for the first user, second user pose information 108b for the second user, third user pose information 108c for the third user), and the avatar data of all users in image 27A. The 3D engine 20 receives objects 106a to c (the first user avatar data object 106a of the first user, the second user avatar data object 106b of the second user, and the third user avatar data object 106c of the third user) and calculates the position and pose of some or all of the user avatars 132a to c (the first user avatar 132a, the second user avatar 132b, and the third user avatar 132c) relative to the identified virtual surface 34. The 3D engine 20 then renders some or all of the user avatars 132a to c, displaying the rendered user avatars 132a to c superimposed on image 27A on the display 22A of the first computing device 12A. The 3D engine 20 may be, for example, UNITY3D, UNREAL ENGINE, or EffectSDK.

[0019] Next, the augmented reality program 14 is executed, and in the loop, updated second user pose information 126 of a second user is received via the computer network 136, an updated image 27A from the live image feed acquired by the rear camera 26A of the first computing device 12A is received, the position and pose of the second user avatar 132b relative to the virtual plane 34 are calculated based on the updated second user pose information 126, and the second user avatar 132b of the second user superimposed on the updated image 27A is displayed by rendering the second user avatar data object 106b. The second user avatar 132b remains anchored to the virtual plane 34 which remains world-locked to the physical plane in the image 27A within the real-world three-dimensional environment, and assumes a pose in accordance with the updated second user pose information 126 each time the loop is traversed.

[0020] To implement a similar loop feedback process for the other avatars 132a and 132c, the first user avatar 132a, while the first user avatar 132a remains anchored to the virtual plane 34, assumes a pose in accordance with first user pose information 108a that is updated each time the loop is traversed, and the third user avatar 132c, while the third user avatar 132c remains anchored to the virtual plane 34, assumes a pose in accordance with third user pose information 108c that is updated each time the loop is traversed.

[0021] The second user pose information 120 and the updated second user pose information 126 may each include second face tracking data 122 and 128. The processor 28, by executing the three-dimensional engine 20, renders the second user avatar data object 132b of the second user based on the second face tracking data 122 and 128, thereby may be configured to display the second user avatar 132b of the second user that is posed with a face corresponding to the facial expression of the second user. The face of the second user avatar 132b may be rendered such that its appearance changes as the second face tracking data 128 changes during successive passes through the loop. By implementing a similar loop feedback process for the third user avatar 132c, the third user avatar 132c may be rendered such that its appearance changes as the face tracking data of the third user pose information 108c changes during successive passes through the loop.

[0022] A similar loop feedback process may also be implemented for the first user avatar 132a. The first user pose information 108a may include first face tracking data 138. The processor 28, by executing the three-dimensional engine 20, renders the first user avatar data object 106a of the first user based on the first face tracking data 138 of the first user pose information 108a, thereby may be configured to display the first user avatar 132a of the first user that is posed with a face corresponding to the facial expression of the first user. The face of the first user avatar 132a may be rendered such that its appearance changes as the first face tracking data 138 changes during successive passes through the loop.

[0023] The second user pose information 120 and the updated second user pose information 126 may each include second skeletal tracking data 124 and 130, respectively. The processor 28 is configured to run the three-dimensional engine 20 and render the body of the second user avatar 132b based on the second skeletal tracking data 124 and 130, so that its appearance changes as the second skeletal tracking data 124 and 130 changes while passing through the loop sequentially. A similar loop feedback process may also be implemented for the third user avatar 132c so that the body of the third user avatar 132c changes as the skeletal tracking data of the third user pose information 108c changes while passing through the loop sequentially.

[0024] A similar loop feedback process may also be implemented for the first user avatar 132a. The first user pose information 108a may include first skeletal tracking data 140. The processor 28 may be configured to run the three-dimensional engine 20 and render the body of the first user avatar 132a based on the first skeletal tracking data 140 such that its appearance changes as the first skeletal tracking data 140 changes while passing through the loop sequentially.

[0025] In response to generating the first user pose information 108a, the processor 28 is configured to render the first user avatar data object 106a of the first user to display the first user avatar 132a of the first user superimposed on the updated image 27A, so that the first user avatar 132a takes a pose according to the first user pose information 108a.

[0026] In this example, the pose tracking module 16 is shown to receive a front camera image 25A from the front camera 24A, but alternatively or additionally, the pose tracking module 16 may be configured to receive a rear camera image 27A from the rear camera 26A, thereby enabling the processor 28 to execute the pose tracking module 16 to identify tracking features in the image 27A acquired by the rear camera 26A and the updated image 27A, and in the first user image 25A of the first user, generate first user pose information 108a of the first user based on the identified tracking features, and transmit the first user pose information 108a to the second computing device 12B of the second user via the computer network 136. In this example, the front camera 24A and the rear camera 26A are operated to acquire the first user image 25A and the updated image 27A in real time, each with a timestamp that synchronizes the identified tracking features in image 27A and the updated image 27A with the identified tracking features in the first user image 25A of the first user.

[0027] In one scenario, users participate in a live augmented reality session 134 hosted by a social media platform's server computing system 100, where each user has their own associated account. In the example in Figure 1, the first computing device 12A of the first user, the second computing device 12B of the second user, and the third computing device 12C of the third user each run instances of the augmented reality avatar program 15 configured to communicate with the social media platform's server computing system 100, as well as instances of the augmented reality program 14, pose tracking module 16, augmented reality module 18, and 3D engine 20. The first user pose information 108a, the first user avatar data object 106a, the second user pose information 120, the updated second user pose information 126, and the second user avatar data object 106b of the first user are exchanged between the augmented reality avatar program 15 running on the first computing device 12A and the augmented reality avatar program 15 running on the second computing device 12B via the server computing system 100. It will be understood that each user's pose information is continuously sampled within the control loop for the live augmented reality session 134. Thus, to clarify the first time the second user pose information 120 is acquired, this second user pose information 120 may be referred to as the initial second user pose information.

[0028] As shown in Figure 1 and in more detail in Figure 2, the avatar data object and pose information may be packaged by the server computing device 100 into a data container or a configuration file, and the server computing device 100 transmits the data container to each computing device of the computing system 10. The avatar data object may be transmitted to the server computing device 100 by computing devices 12A to C in a configuration file, which is a description file that defines a list of the avatar's properties and attributes. For example, the configuration file may define hair color, eye color, clothing, accessories, virtual skeletal structure, skin shape, facial expression control points, etc.

[0029] In the example shown in Figure 1, the server computing device 100 transmits a first avatar data container 118a containing a second user avatar data object 106b and a third user avatar data object 106c to the first computing device 12A, and also transmits a first pause data container 116a containing second user pause information 108b and third user pause information 108c to the first computing device 12A. The second computing device 12B transmits a second configuration file 104b containing a second user avatar data object 106b to the server computing device 100, and the third computing device 12C transmits a third configuration file 104c containing a third user avatar data object 106c to the server computing device 100.

[0030] Figure 2 is a schematic diagram of a server computing device 100 that exchanges pause information and avatar data objects with the three computing devices 12A to C in Figure 1. The server computing device 100 comprises a processor 114, a non-volatile storage device 102 operationally coupled to the processor 114, and an avatar hosting program 110 stored in the non-volatile storage device 102, which may be loaded into volatile memory 112 and executed by the processor 114 of the server computing device 100.

[0031] During the initialization process, when avatars 132a~c are first rendered on the displays 22A~C of the computing devices 12A~C, the avatar hosting program 110 is configured to receive a second user pose information 108b and a second configuration file 104b containing a second user avatar data object 106b from the second user's second computing device 12B by hosting an augmented reality session 134. The server computing device also receives initial first user pose information 108a and a first configuration file 104a containing a first user avatar data object 106a from the first user's first computing device 12A, and receives initial third user pose information 108c and a third configuration file 104c containing a third user avatar data object 106c from the third user's third computing device 12C. The avatar hosting program 110 then stores the first configuration file 104a, the second configuration file 104b, and the third configuration file 104c in the avatar configuration database 104, and stores the initial first user pose information 108a, the second user pose information 108b, and the initial third user pose information 108c in the avatar pose information database 108.

[0032] In response to receiving a first request from the first computing device 12A, the avatar hosting program 110 sends a second user avatar data object 106b and second user pose information 108b to the first computing device 12A. The first request may be a first user's first configuration file 104a and first user pose information 108a. In this example, the avatars of the three users are rendered on the display 22A of the first computing device 12A, and the avatar hosting program 110 sends to the first computing device 12A a first avatar data container 118a containing the second user avatar data object 106b and the third user avatar data object 106c, and a first pose data container 116a containing the second user pose information 108b and the third user pose information 108c. This causes the first computing device 12A to render the second user avatar data object 106b and the third user avatar data object 106c, thereby displaying the second and third user avatars on the display 22A.

[0033] In response to receiving a second request from the second computing device 12B, the avatar hosting program 110 sends the first user avatar data object 106a and the initial first user pose information 108a to the second computing device 12B. The second request may be the second user's second configuration file 104b and the second user pose information 108b. In this example, the avatars of the three users are rendered on the display 22B of the second computing device 12B, and the avatar hosting program 110 sends to the second computing device 12B a second avatar data container 118b containing the first user avatar data object 106a and the third user avatar data object 106c, and a second pose data container 116b containing the first user pose information 108a and the third user pose information 108c. This causes the second computing device 12B to render the first user avatar data object 106a and the third user avatar data object 106c, thereby displaying the first user avatar and the third user avatar on the display 22B.

[0034] In response to receiving a third request from the third computing device 12C, the avatar hosting program 110 sends a third user avatar data object 106c and initial third user pose information 108a to the third computing device 12C. The third request may be a third user configuration file 104c and third user pose information 108c. In this example, the avatars of the three users are rendered on the display 22C of the third computing device 12C, and the avatar hosting program 110 sends to the third computing device 12C a third avatar data container 118c containing the first user avatar data object 106a and the second user avatar data object 106b, and a third pose data container 116c containing the first user pose information 108a and the second user pose information 108b. This causes the third computing device 12C to render the first user avatar data object 106a and the second user avatar data object 106b, thereby displaying the second user avatar and the third user avatar on the display 22C.

[0035] A runtime process in which the avatar is continuously re-rendered using pose information that is continuously updated in a loop by the display of the computing device is executed following the initialization process. In this loop, the processor 114 of the server computing device 100 is configured to receive updated second user pose information 108b for the second user from the second computing device 12B. Upon receiving the updated second user pose information 108b, the processor 114 transmits the updated second user pose information 108b to the first computing device 12A and the third computing device 12C, causing the first computing device 12A to render the second user avatar data object 106b using the updated second user pose information 108b and display the second user avatar on the display 22A, and the third computing device 12C to render the second user avatar data object 106b using the updated second user pose information 108b and display the second user avatar on the display 22C. Then, the updated first user pose information 108a for the first user is received from the first computing device 12A, and the updated third user pose information 108c for the third user is received from the third computing device 12C. Upon receiving the updated first user pose information 108a and the updated third user pose information 108c, the updated first user pose information 108a and the updated third user pose information 108c are transmitted to the second computing device 12B, causing the second computing device 12B to render the first user avatar data object 106a using the updated first user pose information 108a, render the third user avatar data object 106c using the updated third user pose information 108c, and display the first user avatar and the third user avatar on the display 22B.

[0036] It will be understood that by executing a similar runtime process on the first computing device 12A, the second user avatar data object 106b may be rendered using the updated second user pose information 108b, the third user avatar data object 106c may be rendered using the updated third user pose information 108c, and the second and third user avatars may be displayed on the display 22A. Similarly, by executing a similar runtime process on the third computing device 12C, the second user avatar data object 106b may be rendered using the updated second user pose information 108b, the first user avatar data object 106a may be rendered using the updated first user pose information 108a, and the first and second user avatars may be displayed on the display 22C.

[0037] The initial first user pose information 108a and the updated first user pose information 108a may include first face tracking data, the second user pose information 108b and the updated second user pose information 108b may include second face tracking data, and the third user pose information 108c may include third face tracking data. For example, the second computing device 12B may be instructed to render the first user avatar data object 106a of the first user based on the first face tracking data, thereby displaying the first user avatar of the first user posed with a face corresponding to the first user's facial expression. The first computing device 12A may be instructed to render the second user avatar data object 106b of the second user based on the second face tracking data, thereby displaying the second user avatar of the second user posed with a face corresponding to the second user's facial expression on the display 22A.

[0038] Here, with reference to Figures 3 and 4, an exemplary use case illustrating an aspect of the present disclosure is presented. As schematically shown in Figure 3, a first user 202 is standing in a dining room 212 in a real-world three-dimensional environment 200 and may be holding a first computing device 12A, which in this example may take the form of a smartphone. However, the computing device is not particularly limited to a smartphone and may take any form, such as a dedicated smartphone and tablet device or a mixed reality head-mounted device (HMD), which allows the user to view augmented reality images overlaid on the real-world three-dimensional environment 200.

[0039] The first computing device 12A performs SLAM mapping within the physical environment 200 of the dining room 212 to create a map of the dining room 212 that includes a mesh of reconstructed virtual surfaces and a virtual surface 234 corresponding to the physical surface 210, which is the table surface in this exemplary use case. The virtual surface 234 may be a plane or a curved surface (as shown in the figure) and is usually identified by tracking visual features in the rear camera image using an optical flow camera or depth camera. Object recognition techniques may be used to identify the virtual surface. Instead of virtual surfaces, appropriate anchor points for anchoring each avatar may be programmed to be identified in the rear camera image. The first computing device 12A receives second user pose information of the second user from the second computing device and third user pose information of the third user from the third computing device.

[0040] The first computing device 12A calculates the positions and poses of the first user avatar 132a, the second user avatar 132b, and the third user avatar 132c relative to an identified virtual plane that is world-locked to the physical plane 210 in the real-world three-dimensional environment 200 of the dining room 212. The first computing device 12A then renders the first user avatar 132a, the second user avatar 132b, and the third user avatar 132c and displays the rendered avatars 132a-c, superimposed on image 27A, on the display 22A of the first computing device 12A. Then, the first user 202, looking at the display 22A of the first computing device 12A, sees complete avatars 132a-c rendered on the display 22A in the appropriate screen position and pose, which may give the first user 202 the illusion that the avatars 132a-c are standing in the real-world three-dimensional environment 200 of the dining room 212, as shown in Figure 3. The sense of solidarity in the interaction may be enhanced by presenting other user avatars in the same room along with the user's avatar.

[0041] Referring to Figure 5, the exemplary use cases shown in Figures 3 and 4 are used to demonstrate changes in the appearance of avatars 132a to c on the display 22A of the first computing device 12A during the runtime process, so that one or more parts of avatars 132a to c are associated with movement. The first computing device 12A is configured during the initialization process to render the body of the first user avatar 132a based on initial first user pose information 120a, the body of the second user avatar 132b based on second user pose information 120b, and the body and face of the third user avatar 132c based on initial third user pose information 120c. During the runtime process, as shown by the dotted line, the first computing device 12A is configured to render the body of the first user avatar 132a based on the updated first user pose information 126a (the first user turns their head), the body of the second user avatar 132b based on the updated second user pose information 126b (the second user gives a thumbs-up), and the body and face of the third user avatar 132c based on the updated third user pose information 126c (the third user raises their hand and frowns).

[0042] Therefore, during the runtime process, the faces and bodies of the first user avatar 132a, the second user avatar 132b, and the third user avatar 132c are rendered so that their appearance changes as the updated first user pose information 126a, the updated second user pose information 126b, and the updated third user pose information 126c change as they pass through the loop sequentially. Thus, the rendering of avatars 132a-c achieves a reality enhanced by the realistic rendering of the faces and bodies of avatars 132a-c, allowing users using each computing device to have a more intimate collaborative experience in their online interactions with each other during an augmented reality session. In online interactions, they can communicate with each other using facial expressions and body language in addition to voice and text, thereby enhancing the sense of physical proximity and solidarity among participants in the augmented reality session.

[0043] Figure 6 shows a flowchart of Method 300 for the initialization process when the avatar is first rendered on the computing device's display. The following description of Method 300 is provided with reference to the software and hardware components shown in Figures 1-5, as described above. It will be understood that Method 300 may also be performed in other contexts using other appropriate hardware and software components.

[0044] In step 302, the first display device transmits a first configuration file containing a first user avatar data object to the server computing device. In step 304, the second display device transmits a second configuration file containing a second user avatar data object to the server computing device. In step 306, the server computing device receives and stores the first configuration file containing the first user avatar data object and the second configuration file containing the second user avatar data object. In step 308, the server computing device selects and transmits the avatar data object to be transmitted to the first display device, and selects and transmits the avatar data object to be transmitted to the second display device. In step 310, the first display device receives the selected avatar data object from the server computing device. In step 314, the first display device uses the selected avatar data object to calculate the position and pose of the selected avatar data object, including the first and second user avatar data objects, relative to the virtual plane. In step 318, the selected avatar data object, including the first user avatar data object and the second user avatar data object, is rendered on the display based on the calculated position and pose.

[0045] In step 312, the selected avatar data object is received from the server computing device in the second display device. In step 316, the second display device uses the selected avatar data object to calculate the position and pose of the selected avatar data object, including the first user avatar data object and the second user avatar data object, relative to the virtual plane. In step 320, the selected avatar data object, including the first user avatar data object and the second user avatar data object, is rendered on the display based on the calculated position and pose.

[0046] Figure 7 shows a flowchart of Method 400 for a runtime process in which the avatar is continuously re-rendered using pause information that is continuously updated in a loop by the display of a computing device. As described above, the following description of Method 400 is provided with reference to the software and hardware components shown in Figures 1-5. It will be understood that Method 400 may also be executed in other contexts using other suitable hardware and software components.

[0047] In step 402, a first user image of the first user is acquired in the first display device using the camera. In step 404, tracking features of the first user image are identified in the first display device. In step 406, first user pose information is generated in the first display device based on the identified tracking features. In step 408, the first user pose information is transmitted to the server computing device in the first display device.

[0048] In step 410, the second user image of the second user is acquired in the second display device using the camera. In step 412, tracking features of the second user image are identified in the second display device. In step 414, second user pose information is generated in the second display device based on the identified tracking features. In step 416, the second user pose information is transmitted to the server computing device in the second display device.

[0049] In step 418, the first user pause information and the second user pause information are received as updated pause information for the first user and the second user, respectively, and stored in the server computing device. In step 420, the updated pause information is transmitted from the server computing device to the first display device and the second display device, respectively.

[0050] Steps 422, 426, and 428 are performed in the first display device. In step 422, updated second user pose information is received. In step 426, the positions and poses of the first and second user avatars are calculated based on the first and second user pose information. In step 428, the first and second user avatars are rendered on the display based on the calculated positions and poses.

[0051] Steps 424, 430, and 432 are performed in the second display device. In step 424, the updated first user pose information is received. In step 430, the positions and poses of the first and second user avatars are calculated based on the first and second user pose information. In step 432, the first and second user avatars are rendered on the display based on the calculated positions and poses.

[0052] Figure 8 shows a flowchart of Method 500. As described above, the following explanation of Method 500 is provided with reference to the software and hardware components shown in Figures 1-5. It will be understood that Method 500 may also be implemented in other contexts using other appropriate hardware and software components.

[0053] In step 502, the second user avatar data object and second user pose information of the second user are received via the computer network. In step 504, an image is received from a live image feed acquired by the camera of the first computing device. In step 506, a virtual surface corresponding to a physical surface appearing in the image is identified.

[0054] In step 508, the position and pose of the second user avatar relative to the virtual plane are calculated based on the second user pose information. In step 510, the second user avatar of the second user is displayed superimposed on the image by rendering the second user avatar data object. The second user avatar is anchored to the virtual plane according to the calculated position and pose of the second user avatar, and the virtual plane is world-locked to the physical plane of the image in the real-world three-dimensional environment.

[0055] Steps 512, 514, 516, and 518 are performed in a loop. In step 512, updated second user pose information for the second user is received via the computer network. In step 514, the updated image is received from the live image feed acquired by the camera of the first computing device. In step 516, the position and pose of the second user avatar relative to the virtual plane are calculated based on the updated second user pose information. In step 518, the second user avatar of the second user is displayed overlaid on the updated image by rendering the second user avatar data object, with the second user avatar anchored to a virtual plane that remains world-locked to a physical plane in the image within the real-world three-dimensional environment, and posing according to the updated second user pose information each time the loop is passed.

[0056] It will be understood that methods 300, 400, and 500 are provided as examples and are not intended to be limiting. Therefore, methods 300, 400, and 500 may include additional and / or alternative steps compared to those shown in Figures 6–8. Furthermore, it should be understood that methods 300, 400, and 500 may be performed in any suitable order.

[0057] The above system and method allows multiple users, each using their own computing device, to see each other's avatars on their devices during an augmented reality session, so that their positions and appearances change in real time in response to changes in the users' positions and appearances. Because the rendering of the avatars' faces and bodies has been enhanced, users participating in an augmented reality session can have a more intimate collaborative experience in their online interactions with each other. This increases the sense of physical proximity and solidarity, which is further enhanced when users communicate with each other via their respective computing devices using facial expressions and body language in addition to voice and text.

[0058] In some embodiments, the methods and processes described herein may be linked to a computing system of one or more computing devices. Specifically, such methods and processes may be implemented as computer application programs or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0059] Figure 9 schematically illustrates a non-limiting embodiment of a computing system 600 capable of implementing one or more of the methods and processes described above. The computing system 600 is shown in a simplified form. The computing system 600 may comprise the first computing device 12A, the second computing device 12B, the third computing device 12C, the fourth computing device 12D, and the server computing device 100, as described above, as shown in Figures 1 to 5, respectively. The computing system 600 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing, game devices, mobile computing devices, mobile communication devices (e.g., smartphones) and / or other computing devices, and wearable computing devices, such as smartwatches and head-mounted augmented reality devices.

[0060] The computing system 600 comprises a logical processor 602, a volatile memory 604, and a non-volatile storage device 606. The computing system 600 may optionally include a display subsystem 608, an input subsystem 610, a communication subsystem 612, and / or other components not shown in Figure 9.

[0061] The logical processor 602 includes one or more physical devices configured to execute instructions. For example, the logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical structures. Such instructions may be implemented to perform tasks, realize data types, transform the state of one or more components, achieve technical effects, or achieve desired results.

[0062] A logical processor may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, a logical processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processors of the logical processor 602 may be single-core or multi-core, and the instructions executed on them may be configured for sequential, parallel, and / or distributed processing. Individual components of the logical processor may optionally be distributed across two or more separate devices located remotely and / or configured for coordinated processing. Embodiments of the logical processor may be virtualized and executed by computing devices connected to a remotely accessible network configured in a cloud computing setup. In such cases, it should be understood that these virtualized embodiments run on different physical logical processors on various different machines.

[0063] The non-volatile storage device 606 includes one or more physical devices configured to hold instructions executable by a logical processor in order to implement the methods and processes described herein. When such methods and processes are implemented, the state of the non-volatile storage device 606 may be transformed, for example, to hold different data.

[0064] The non-volatile storage device 606 may include removable and / or built-in physical devices. The non-volatile storage device 606 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor memory (e.g., ROM, EPROM, EEPROM, flash memory, etc.), and / or magnetic memory (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), or other mass storage technology. The non-volatile storage device 606 may include non-volatile, dynamic, static, read / write, read-only, sequential access, position-addressable, file-addressable, and / or content-addressable devices. It will be understood that the non-volatile storage device 606 is configured to retain instructions even when power to the non-volatile storage device 606 is cut off.

[0065] The volatile memory 604 may include a physical device that has random access memory. The volatile memory 604 is typically used by the logical processor 602 to temporarily store information during the processing of software instructions. If power to the volatile memory 604 is cut off, it is understood that the volatile memory 604 will not continue to store instructions.

[0066] The logical processor 602, the volatile memory 604, and the non-volatile storage device 606 may be integrated together in one or more hardware logic components. Such hardware logic components may include, for example, field-programmable gate arrays (FPGAs), integrated circuits for specific programs and applications (PASICs / ASICs), standard products for specific programs and applications (PSSPs / ASSPs), systems on a chip (SOCs), and complex programmable logic devices (CPLDs).

[0067] The terms “module,” “program,” and “engine” may be used to describe a form of computing system 600 that is typically implemented in software, to execute a specific function, which involves a processor specially configuring a program to perform a certain function using a portion of volatile memory. Thus, a module, program, or engine may be instantiated using a portion of volatile memory 604 via a logic processor 602 that executes instructions held by a non-volatile memory 606. It will be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Similarly, the same module, program, and / or engine may be instantiated from different applications, services, code block, object, routine, API, function, etc. The terms “module,” “program,” and “engine” may include individuals or groups such as executable files, data files, libraries, drivers, scripts, database records, etc.

[0068] The display subsystem 608, if included, may be used to present a visual representation of the data held by the non-volatile memory 606. This visual representation may take the form of a graphical user interface (GUI). Since the methods and processes described herein modify the data held by the non-volatile memory and transform the state of the non-volatile memory, the state of the display subsystem 608 may also be transformed to visually represent the changes in the underlying data. The display subsystem 608 may include one or more display devices utilizing substantially any type of technology. Such display devices may be combined with the logical processor 602, volatile memory 604 and / or non-volatile memory 606 within a shared enclosure, or such display devices may be peripheral display devices.

[0069] The input subsystem 610 may include, if included, one or more user input devices, such as a keyboard, mouse, touchscreen, or game controller, and may interact with them. In some embodiments, the input subsystem may include, and interact with, selected natural user input (NUI) components. These components may be integrated or peripheral, and the transmission and / or processing of input actions may be handled onboard or offboard. Exemplary NUI components may include a microphone for speech and / or voice recognition, an infrared camera, color camera, stereo camera, and / or depth camera for machine vision and / or gesture recognition, a head tracker, eye tracker, accelerometer, and / or gyroscope for motion detection and / or intent recognition, an electric field sensing component for evaluating brain activity, and / or any other suitable sensors.

[0070] The communication subsystem 612, if included, may be configured to enable communication between the various computing devices described herein and other devices. The communication subsystem 612 may include wired and / or wireless communication devices compatible with one or more different communication protocols. In a non-limiting example, the communication subsystem may be configured to communicate via a wireless telephone network, or via a wired or wireless local or wide area network, such as HDMI® over a Wi-Fi connection. In some embodiments, the communication subsystem may enable the computing system 600 to send and receive messages with other devices over a network such as the Internet.

[0071] The following paragraphs provide additional support to the claims of this application. One embodiment provides a computing system comprising a processor and memory of a computing device of a first user, wherein the processor is configured to run an augmented reality (AR) avatar program using a portion of the memory to receive a second user avatar data object and second user pose information of a second user via a computer network, receive an image from a live image feed acquired by a camera of the computing device, identify a virtual plane corresponding to a physical plane appearing in the image, calculate the position and pose of the second user avatar relative to the virtual plane based on the second user pose information, and render the second user avatar data object to display the second user avatar of the second user superimposed on the image. The second user avatar is anchored to the virtual plane and posed according to the calculated position and pose of the second user avatar, and the virtual plane is world-locked to the physical plane of the image in a real-world three-dimensional environment. In this embodiment, additionally or alternatively, the second user pose information is the initial second user pose information for the session of the AR avatar program, and the image is the initial image for the session of the AR avatar program. In this embodiment, additionally or alternatively, by running the AR avatar program, the loop includes receiving updated second user pose information for the second user via the computer network, receiving updated images from the live image feed acquired by the camera of the computing device, calculating the position and pose of the second user avatar relative to the virtual plane based on the updated second user pose information, and rendering the second user avatar data object to display the second user avatar, which remains anchored to a virtual plane that remains world-locked to the physical plane in the image within the three-dimensional environment of the real world, overlaid on the updated image, and which poses according to the updated pose information each time the loop is passed.In this embodiment, additionally or alternatively, the second user pose information and the updated second user pose information include second face tracking data, and the processor is configured to display the second user's second user avatar posed with a face corresponding to the second user's facial expressions by rendering the second user's second user avatar data object for the second user based on the second face tracking data, and the face of the second user avatar is rendered to change in appearance as the second face tracking data changes while passing through the loop sequentially. In this embodiment, additionally or alternatively, the second user pose information includes second face tracking data, and the processor is configured to display the second user's second user avatar posed with a face corresponding to the second user's facial expressions by rendering the second user's second user avatar data object for the second user based on the second face tracking data. In this embodiment, additionally or alternatively, the second user pose information includes second skeletal tracking data, and the processor is configured to render the body of the second user avatar such that its appearance changes as the second skeletal tracking data changes. In this embodiment, additionally or alternatively, the camera is a rear camera, the computing device is a first computing device, and the computing system further comprises a front camera configured to acquire a first user image of the first user, and the processor is configured to identify tracking features in the first user image of the first user, generate first user pose information of the first user based on the identified tracking features, and transmit the first user pose information to the second computing device of the second user via the computer network.In this embodiment, additionally or alternatively, the first user pose information includes first face tracking data, and the processor is configured to display the first user's first user avatar posed with a face corresponding to the first user's facial expression by rendering the first user's first user avatar data object based on the first face tracking data. In this embodiment, additionally or alternatively, the processor is further configured to display the first user's first user avatar superimposed on the image, posed according to the first user pose information, by rendering the first user's first user avatar data object. In this embodiment, additionally or alternatively, the front camera and rear camera are operated to acquire the first user image and the image in real time, with timestamps that synchronize the identified tracking features in the image with the identified tracking features in the first user image of the first user. In this embodiment, the processor is configured to identify tracking features in the image acquired by the rear camera and the first user image of the first user, generate first user pose information of the first user based on the identified tracking features, and transmit the first user pose information to the second computing device of the second user via the computer network.In this embodiment, additionally or alternatively, the computing device is a first computing device, the first user and the second user each have associated accounts on a social media platform, the first computing device and the second computing device each run instances of the AR avatar program configured to communicate with the server computing system of the social media platform, and the first user pose information of the first user, the first user avatar data object of the first user, the second user pose information, and the second user avatar data object are exchanged between the AR avatar program running on the first computing device and the AR avatar program running on the second computing device via the server computing system.

[0072] Another embodiment provides a method for a client program running on a computing device operated by a first user, the method comprising the steps of receiving a second user's second user avatar data object and second user pose information via a computer network, receiving an image from a live image feed acquired by a camera on the computing device, identifying a virtual plane corresponding to a physical plane appearing in the image, calculating the position and pose of the second user avatar relative to the virtual plane based on the second user pose information, and rendering the second user avatar data object to display the second user's second user avatar superimposed on the image, wherein the second user avatar is anchored to the virtual plane according to the calculated position and pose of the second user avatar, and the virtual plane is world-locked to the physical plane of the image in a real-world three-dimensional environment. In this embodiment, additionally or alternatively, the method further includes the steps of: the loop receiving updated second user pose information of the second user via the computer network; receiving updated images from the live image feed acquired by the camera of the computing device; calculating the position and pose of the second user avatar relative to the virtual plane based on the updated second user pose information; and rendering the second user avatar data object to display the second user avatar of the second user superimposed on the updated image, wherein the second user avatar remains anchored to the virtual plane, which remains world-locked to the physical plane in the image within the three-dimensional real-world environment, and poses according to the updated second user pose information each time it passes through the loop.In this embodiment, additionally or alternatively, the second user pose information includes second face tracking data, and the method further includes displaying the second user's second user avatar posed with a face corresponding to the second user's facial expressions by rendering the second user's second user avatar data object based on the second face tracking data. In this embodiment, additionally or alternatively, the computing device is a first computing device, and the method further includes acquiring a first user image of the first user, identifying tracking features in the first user image of the first user, generating first user pose information of the first user based on the identified tracking features, and transmitting the first user pose information to the second computing device of the second user via the computer network. In this embodiment, the first user pose information may include first face tracking data, and the method further includes rendering the first user avatar data object of the first user based on the first face tracking data to display the first user avatar of the first user posed with a face corresponding to the facial expression of the first user.

[0073] Another embodiment includes a processor, a non-volatile memory device operationally coupled to the processor, and an avatar hosting program stored in the non-volatile memory device and executed by the processor of the server computing device, wherein the avatar hosting program hosts an avatar session, receiving and storing a second user avatar data object and second user pose information of a second user from a second computing device, and, upon receiving a first request for a second user avatar from the first computing device, transmitting the second user avatar data object and the second user pose information to the first computing device, thereby causing the first computing device to render the second user avatar data object and display the second user avatar, and receiving and storing a first user avatar data object and initial first user pose information of a first user from the first computing device, and transmitting a second request for a first user avatar from the second computing device In response to receiving the first user avatar data object and the initial first user pose information, the second computing device is instructed to render the first user avatar data object and display the first user avatar. The loop includes receiving updated second user pose information for the second user from the second computing device, and in response to receiving the updated second user pose information, transmitting the updated second user pose information to the first computing device, instructing the first computing device to render the second user avatar data object using the updated second user pose information and display the second user avatar. The loop also includes receiving updated first user pose information for the first user from the first computing device, and in response to receiving the updated first user pose information, transmitting the updated first user pose information to the second computing device.The second computing device is configured to render the first user avatar data object using the updated first user pose information and display the first user avatar. In this embodiment, additionally or alternatively, the initial first user pose information and the updated first user pose information include first face tracking data, causing the second computing device to render the first user avatar data object of the first user based on the first face tracking data to display the first user avatar of the first user posed with a face corresponding to the first user's facial expression; and the second user pose information and the updated second user pose information include second face tracking data, causing the first computing device to render the second user avatar data object of the second user based on the second face tracking data to display the second user avatar of the second user posed with a face corresponding to the second user's facial expression. In this embodiment, the initial first user pose information and the updated first user pose information include first skeleton tracking data, causing the second computing device to render the first user's first user avatar data object based on the first skeleton tracking data, thereby displaying the first user's first user avatar posing with a body whose appearance changes as the first skeleton tracking data changes while passing through the loop sequentially; the second user pose information and the updated second user pose information include second skeleton tracking data, causing the first computing device to render the second user avatar data object based on the second user's second skeleton tracking data, thereby displaying the second user's second user avatar posing with a body whose appearance changes as the second skeleton tracking data changes while passing through the loop sequentially;

[0074] It will be understood that the configurations and / or approaches described herein are illustrative in nature and are subject to numerous modifications; therefore, these specific embodiments or examples should not be considered restrictively. The specific routines or methods described herein may represent one or more of any number of processing strategies. For this reason, the illustrated and / or described operations may be performed in parallel, in any other order, or omitted, in the order illustrated and / or described. Similarly, the order of the processes described above may be changed.

[0075] The subject matter of this disclosure includes novel and non-obvious combinations and subcombinations of the various processes, systems, configurations, and other features, functions, operations, and / or characteristics disclosed herein, as well as all equivalents thereof.

[0076] To the extent that the terms “include,” “include,” “possess,” “encompass,” and their variations are used herein, such terms are intended to be inclusive in a manner similar to the term “equip” as an open transposition, without the exclusion of any addition or other element.

[0077] It will be understood that the configurations and / or approaches described herein are illustrative in nature and are subject to numerous modifications; therefore, these specific embodiments or examples should not be considered restrictively. The specific routines or methods described herein may represent one or more of any number of processing strategies. For this reason, the illustrated and / or described operations may be performed in parallel, in any other order, or omitted, in the order illustrated and / or described. Similarly, the order of the processes described above may be changed.

[0078] The subject matter of this disclosure includes novel and non-obvious combinations and subcombinations of the various processes, systems, configurations, and other features, functions, operations, and / or characteristics disclosed herein, as well as all equivalents thereof.

Claims

1. A computing system comprising a processor and memory of a first user's computing device, The aforementioned processor executes an augmented reality (AR) avatar program using a portion of the memory, The system receives the second user avatar data object and second user pose information of a second user via a computer network. The computer device receives images from a live image feed acquired by the camera of the aforementioned computing device. Identify the virtual surface corresponding to the physical surface that appears in the aforementioned image, Based on the second user pose information, the position and pose of the second user avatar relative to the virtual plane are calculated. The system is configured to display the second user avatar of the second user superimposed on the image by rendering the second user avatar data object, the second user avatar being anchored to a virtual plane and posed according to the calculated position and pose of the second user avatar, and the virtual plane being world-locked to the physical plane in the image in the real-world three-dimensional environment. The computing device is a first computing device, The first user and the second user each have associated accounts on a social media platform, The first computing device and the second computing device each execute an instance of the AR avatar program, which is configured to communicate with the server computing system of the social media platform. The first user pose information of the first user, the first user avatar data object of the first user, the second user pose information, and the second user avatar data object are exchanged between the AR avatar program running on the first computing device and the AR avatar program running on the second computing device via the server computing system. Computing system.

2. The second user pose information is the second user pose information at the beginning of the session of the AR avatar program, The aforementioned image is the initial image of the session of the AR avatar program. The computing system according to claim 1.

3. By executing the aforementioned AR avatar program, within the loop, The updated second user pause information of the second user is received via the aforementioned computer network. The computing device receives updated images from the live image feed acquired by the camera, Based on the updated second user pose information, the position and pose of the second user avatar relative to the virtual plane are calculated. By rendering the second user avatar data object, a second user avatar is displayed that is superimposed on the updated image and poses according to pose information updated each time the loop is passed, while the second user avatar remains anchored to the virtual surface, and the virtual surface remains world-locked to the physical surface in the image within the real-world three-dimensional environment. The computing system according to claim 1.

4. The second user pose information and the updated second user pose information include second face tracking data, The processor is configured to render the second user's second user avatar data object based on the second face tracking data, thereby displaying the second user's second user avatar posed with a face corresponding to the second user's facial expression. The face of the second user avatar is rendered to change its appearance as the second face tracking data changes while passing through the loop sequentially. The computing system according to claim 3.

5. The second user pose information includes second face tracking data, The processor is configured to render the second user's second user avatar data object based on the second face tracking data, thereby displaying the second user's second user avatar posed with a face corresponding to the second user's facial expression. The computing system according to claim 1.

6. The second user pose information includes second skeletal tracking data, The processor is configured to render the body of the second user avatar so that its appearance changes as the second skeletal tracking data changes. The computing system according to claim 1.

7. The camera is a rear camera, the computing device is a first computing device, and the computing system is The system further comprises a front camera configured to acquire a first user image of the first user, The aforementioned processor, Identify the tracking features in the first user image of the first user, Based on the identified tracking features, first user pose information for the first user is generated. The first user pause information is configured to be transmitted to the second user's second computing device via the computer network. The computing system according to claim 1.

8. The first user pose information includes first face tracking data, The processor is configured to display the first user avatar of the first user posed with a face corresponding to the facial expression of the first user by rendering a first user avatar data object of the first user based on the first face tracking data. The computing system according to claim 7.

9. The processor is further configured to render the first user avatar data object of the first user to display the first user avatar of the first user superimposed on the image, which is posed according to the first user pose information. The computing system according to claim 7.

10. The front camera and rear camera are operated to acquire the first user image and the image in real time, each with a timestamp that synchronizes the identified tracking features in the image with the identified tracking features in the first user image of the first user. The computing system according to claim 9.

11. The aforementioned processor, The tracking features in the image acquired by the rear camera and the first user image of the first user are identified. Based on the identified tracking features, first user pose information for the first user is generated. The first user pause information is configured to be transmitted to the second user's second computing device via the computer network. The computing system according to claim 7.

12. A method for a client program to run on a computing device operated by a first user, The system receives the second user avatar data object and second user pose information of a second user via a computer network. The computer device receives images from a live image feed acquired by the camera of the aforementioned computing device. Identify the virtual surface corresponding to the physical surface that appears in the aforementioned image, Based on the second user pose information, the position and pose of the second user avatar relative to the virtual plane are calculated. The process includes the step of rendering the second user avatar data object to display the second user avatar of the second user superimposed on the image, The second user avatar is anchored to the virtual plane according to the calculated position and pose of the second user avatar, and the virtual plane is world-locked to the physical plane in the image in the real-world three-dimensional environment. The computing device is a first computing device, The first user and the second user each have associated accounts on a social media platform, The first computing device and the second computing device each execute instances of an augmented reality (AR) avatar program configured to communicate with the server computing system of the social media platform. The first user pose information of the first user, the first user avatar data object of the first user, the second user pose information, and the second user avatar data object are exchanged between the AR avatar program running on the first computing device and the AR avatar program running on the second computing device via the server computing system. method.

13. In the loop, The updated second user pause information of the second user is received via the aforementioned computer network. The computing device receives updated images from the live image feed acquired by the camera, Based on the updated second user pose information, the position and pose of the second user avatar relative to the virtual plane are calculated. The process further includes rendering the second user avatar data object to display the second user avatar of the second user superimposed on the updated image, The second user avatar poses according to the updated second user pose information each time it passes through the loop, while the second user avatar remains anchored to the virtual plane, and the virtual plane remains world-locked to the physical plane in the image in the real-world three-dimensional environment. The method according to claim 12.

14. The second user pose information includes second face tracking data, The method further includes rendering the second user's second user avatar data object based on the second face tracking data to display the second user's second user avatar posed with a face corresponding to the second user's facial expression. The method according to claim 12.

15. The computing device is the first computing device, The aforementioned method, The first user image of the first user is obtained, Identify the tracking features in the first user image of the first user, Based on the identified tracking features, first user pose information for the first user is generated. The process further includes transmitting the first user pause information to the second user's second computing device via the computer network. The method according to claim 12.

16. The first user pose information includes first face tracking data, The method further includes rendering a first user avatar data object of the first user based on the first face tracking data to display the first user avatar of the first user posed with a face corresponding to the facial expression of the first user. The method according to claim 15.

17. Processor and A non-volatile memory device operationally coupled to the aforementioned processor, The system comprises an avatar hosting program stored in the non-volatile memory device and executed by the processor of the server computing device, The aforementioned avatar hosting program hosts avatar sessions, The second user's second user avatar data object and second user pose information are received and stored from the second computing device. Upon receiving a first request for a second user avatar from the first computing device, the second user avatar data object and the second user pose information are transmitted to the first computing device, causing the first computing device to render the second user avatar data object and display the second user avatar. The first user avatar data object and initial first user pose information of the first user are received from the first computing device and stored. Upon receiving a second request for a first user avatar from the second computing device, the system is configured to transmit the first user avatar data object and the initial first user pose information to the second computing device, thereby causing the second computing device to render the first user avatar data object and display the first user avatar. In the loop, The second user receives updated second user pause information from the second computing device. Upon receiving the updated second user pose information, the updated second user pose information is transmitted to the first computing device, causing the first computing device to render the second user avatar data object using the updated second user pose information and display the second user avatar. The updated first user pause information of the first user is received from the first computing device. Upon receiving the updated first user pose information, the updated first user pose information is transmitted to the second computing device, causing the second computing device to render the first user avatar data object using the updated first user pose information and display the first user avatar. Server computing device.

18. The initial first user pose information and the updated first user pose information include first face tracking data, The second computing device is instructed to render the first user avatar data object of the first user based on the first face tracking data, thereby displaying the first user avatar of the first user posed with a face corresponding to the first user's facial expression. The second user pose information and the updated second user pose information include second face tracking data, The first computing device is instructed to render the second user's second user avatar data object based on the second face tracking data, thereby displaying the second user's second user avatar posed with a face corresponding to the second user's facial expression. The server computing device according to claim 17.

19. The initial first user pose information and the updated first user pose information include first skeletal tracking data. The second computing device is instructed to render the first user avatar data object of the first user based on the first skeletal tracking data, thereby displaying the first user avatar of the first user posing with a body whose appearance changes as the first skeletal tracking data changes while passing through the loop sequentially. The second user pose information and the updated second user pose information include second skeletal tracking data, The first computing device is caused to render the second user avatar data object based on the second skeleton tracking data of the second user, thereby displaying the second user avatar of the second user posing with a body whose appearance changes as the second skeleton tracking data changes while passing through the loop sequentially. The server computing device according to claim 17.

20. A program that causes a processor to perform the method described in any one of claims 12 to 16.

Images