Gaze-adjusted avatar for immersive reality applications

Abstract

A method for updating a gaze direction of a sender avatar in a receiver head-mounted device is provided. The method includes verifying, in a recipient device, that a visual tracking of a sender avatar is active in the sender device; and in the receiver device, adjusting the gaze direction of the sender avatar to the gaze point. Adjusting a gaze direction of a sender avatar includes estimating coordinates of a gaze point in a receiver frame at a later time; and in the receiver device, rotating the two eyeballs of the sender avatar to a point in the direction of the gaze point. Also provided are a head-mounted device, a memory storing instructions in the head-mounted device, and a processor configured to execute the instructions to perform the above method.

Description

Background Art Technical Field

[0001] The present disclosure relates to real-time adjustment of three-dimensional human representations for immersive reality applications. More specifically, the present disclosure relates to real-time gaze locking of avatars of social presence in virtual reality applications and augmented reality applications.

[0002] Related technologies

[0003] Virtual reality applications, augmented reality applications, mixed reality applications, and other immersive reality applications are becoming more and more popular with users. Typically, realistic representations of human faces are computationally intensive operations, typically relayed to a remote server that is directly coupled to an augmented reality headset or coupled to an augmented reality headset via a paired mobile device. In addition to reducing the signal-to-noise ratio and thus increasing errors, these strategies may also result in time delays (e.g., latency) that can produce embarrassing artifacts in the virtual world. One of the most notorious artifacts caused by this latency is the dull eye gaze on the sender / receiver avatar. This can cause misunderstandings, expectations, and anxiety to two (or more) people trying to maintain a virtual conversation. Summary of the invention

[0004] In a first embodiment, a computer-implemented method includes: verifying in a receiver device that visual tracking of a sender avatar is active in the sender device; and in the receiver device, adjusting a gaze direction of the sender avatar to a fixation point. Adjusting the gaze direction of the sender avatar includes: estimating coordinates of the fixation point in a receiver frame at a later time; and in the receiver device, rotating two eyeballs of the sender avatar to a point in the direction of the fixation point.

[0005] In a second embodiment, a system includes: a memory storing a plurality of instructions; and one or more processors configured to execute the plurality of instructions to cause the system to perform a process. The process includes: verifying that visual tracking of a sender avatar is active; and in a recipient device, adjusting a gaze direction of the sender avatar to a gaze point, wherein to adjust the gaze direction of the sender avatar, the one or more processors execute instructions for: estimating coordinates of the gaze point in a recipient frame at a later time; and in the recipient device, rotating two gaze directions from the sender device to track the gaze point.

[0006] In other embodiments, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the computer to perform a method. The method includes: verifying in a receiver device that visual tracking of a sender avatar is active in the sender device; and in the receiver device, adjusting a gaze direction of the sender avatar to a gaze point. Adjusting the gaze direction of the sender avatar includes: estimating coordinates of the gaze point in a receiver frame at a later time; and in the receiver device, rotating two eyeballs of the sender avatar to a point in the direction of the gaze point.

[0007] In yet another embodiment, a system includes: a first device for storing instructions; and a second device for executing the instructions to cause the system to perform a method. The method includes: verifying in a receiver device that visual tracking of a sender avatar is active in the sender device; and in the receiver device, adjusting the gaze direction of the sender avatar to a fixation point. Adjusting the gaze direction of the sender avatar includes: estimating the coordinates of the fixation point in a receiver frame at a later time; and in the receiver device, rotating two eyeballs of the sender avatar to a point in the direction of the fixation point.

[0008] These and other embodiments will be apparent to those of ordinary skill in the art from the following.

[0009] According to a first aspect of the present disclosure, a computer-implemented method is provided, comprising: verifying in a receiving device that visual tracking of a sender's avatar is active in the sending device; and in the receiving device, adjusting the gaze direction of the sender's avatar to a gaze point, wherein: adjusting the gaze direction of the sender's avatar comprises: estimating the coordinates of the gaze point in a receiving frame at a later time; and in the receiving device, rotating two eyeballs of the sender's avatar to a point in the direction of the gaze point.

[0010] In some embodiments, verifying that the visual tracking of the sender avatar is currently in place includes identifying a conversation between the sender avatar and the recipient avatar.

[0011] In some embodiments, verifying that the visual tracking of the sender avatar is currently in place includes querying the sender device for a visual tracking signal.

[0012] In some embodiments, verifying that the visual tracking of the sender avatar is currently in place includes verifying that a function value of a virtual distance between the sender avatar and the recipient avatar is greater than a preselected threshold.

[0013] In some embodiments, verifying that the visual tracking of the sender avatar is currently in place includes verifying that a function value of an angular difference between a gaze direction and a gaze point of the sender avatar is greater than a preselected threshold.

[0014] In some embodiments, the computer-implemented method further includes selecting a time period for the later time that is shorter than a network delay for transmission of a signal between the sender device and the receiver device.

[0015] In some embodiments, the visual tracking of the sender avatar is a gaze lock between the sender avatar and the recipient avatar, and estimating the coordinates of the gaze point includes estimating the relative position between the sender avatar and the recipient avatar at a later time.

[0016] In some embodiments, the computer-implemented method further includes updating the gaze direction at the later time to the new gaze direction when a distance between the gaze direction and the new gaze direction received from the sender device is greater than a preselected threshold.

[0017] In some embodiments, the computer-implemented method further includes adjusting a head orientation of the sending avatar based on the gaze point at the later time.

[0018] In some embodiments, the computer-implemented method further comprises updating the world state at a later time based on a dynamic condition of the world state at the current time.

[0019] According to a second aspect of the present disclosure, a system is provided, comprising: a memory storing a plurality of instructions; and one or more processors configured to execute the plurality of instructions so that the system: verifies that visual tracking of a sender avatar is active; and in a receiver device, adjusts a gaze direction of the sender avatar to a gaze point, wherein, in order to adjust the gaze direction of the sender avatar, the one or more processors execute instructions for the following operations: estimates the coordinates of the gaze point in a receiver frame at a later time; and in the receiver device, rotates two gaze directions from the sender device to track the gaze point.

[0020] In some embodiments, to verify that visual tracking of the sender avatar is currently in place, the one or more processors execute instructions for: identifying a conversation between the sender avatar and the recipient avatar.

[0021] In some embodiments, to verify that visual tracking of the sender avatar is currently in place, the one or more processors execute instructions for querying the sender device for a visual tracking signal.

[0022] In some embodiments, to verify that the visual tracking of the sender avatar is currently in place, the one or more processors execute instructions for verifying that a function value of a virtual distance between the sender avatar and the recipient avatar is greater than a preselected threshold.

[0023] In some embodiments, to verify that the visual tracking of the sender avatar is currently in place, the one or more processors execute instructions for the following operations: verify that a function value of an angular difference between the sender avatar's gaze direction and the gaze point is greater than a preselected threshold.

[0024] In some embodiments, the one or more processors further execute instructions for selecting a time period for the later time that is shorter than a network delay for transmission of a signal between the sender device and the receiver device.

[0025] In some embodiments, the one or more processors execute instructions for visually tracking a sending avatar using gaze lock between the sending avatar and the receiving avatar, and in order to estimate the coordinates of the gaze point, the one or more processors execute instructions for estimating the relative position between the sending avatar and the receiving avatar at a later time.

[0026] In some embodiments, the one or more processors execute instructions for: updating the gaze direction at a later time to the new gaze direction when a distance between the gaze direction and the new gaze direction received from the sender device is greater than a preselected threshold.

[0027] In some embodiments, the one or more processors further execute instructions for adjusting the head direction of the sending avatar based on the gaze point at the later time.

[0028] In some embodiments, the one or more processors further execute instructions for updating a world state at a later time based on a dynamic condition of the world state at a current time. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Shown is a recipient and sender participating in a virtual conversation using an augmented reality head mounted device configured for real-time gaze lock in accordance with some embodiments.

[0030] Figure 2 is a block diagram 200 illustrating an example server 130 , client device 110 , and database 152 from architecture 100 , according to some embodiments.

[0031] Figure 3 Gaze adjustment of a sender's gaze in a recipient's head mounted device is shown in accordance with some embodiments.

[0032] Figure 4 Shown are world state parameters of a function for verifying the gaze lock state of a sender avatar in accordance with some embodiments.

[0033] Figure 5is a flow chart illustrating steps in a method for real-time gaze locking of a sender avatar for immersive reality applications in accordance with some embodiments.

[0034] Figure 6 is a block diagram showing an exemplary computer system with which a head mounted device and other client devices, and Figure 5 method.

[0035] In the various drawings, unless explicitly stated otherwise, elements with the same or similar reference numbers are associated with the same or similar features or attributes. DETAILED DESCRIPTION

[0036] In the following detailed description, many specific details are set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to one of ordinary skill in the art that various embodiments of the present disclosure may be practiced without some of these specific details. In other instances, well-known structures and techniques are not shown in detail to avoid obscuring the present disclosure.

[0037] General Overview

[0038] In current augmented reality applications or virtual reality applications, it is crucial that two avatars immersed in a virtual world can interact with each other, where mutual gaze keeps real-time updates about the so-called objects of interest for each avatar. For example, when two avatars have gaze lock with each other, it is expected that as the receiving avatar moves, the sending avatar maintains gaze lock on the new position of the receiving avatar. Traditionally, this would mean that the network latency (which may be as high as 200 milliseconds (ms) or so) for the new position of the gaze of the sending avatar to be received and processed by the receiver's head-mounted device will produce artifacts and gaps in the gaze lock between the two avatars. These gaps become obvious errors, greatly reducing the immersion of the application. These gaps break the sense of eye contact between the avatars, or break the feeling of smoothly pursuing an object in the control of the receiver (e.g., an object that the avatar remains motionless), and these gaps are very obvious to the receiver.

[0039] In addition to network latency, error propagation caused by wireless remote interaction of multiple devices can be a source of inaccuracy that will also contribute to the apparent loss of gaze-locked configuration.

[0040] Therefore, the various embodiments as disclosed herein provide a receiving head mounted device having a mechanism and program to locally adjust the gaze of a sending avatar without having to remotely communicate with a mobile device, a server, and ultimately the sending head mounted device.

[0041] In some embodiments, in addition to avoiding issues caused by network latency, techniques as disclosed herein may also overcome noisy eye tracking detection systems by accurately estimating the sender / receiver's gaze fixation during a virtual conversation.

[0042] Example system architecture

[0043] Figure 1 A recipient 101 and a sender 102 are shown participating in a virtual conversation via an architecture 10 configured for real-time gaze lock using augmented reality head mounted devices 100-1 and 100-2 (hereinafter collectively referred to as "head mounted devices 100") according to some embodiments. The recipient 101 uses a first head mounted device 100-1, and the sender 102 uses a second head mounted device 100-2. Each of the head mounted devices 100 can be paired with a mobile device 110-1 (receiver) and a mobile device 110-2 (sender) and exchange data sets 103-1 and 103-2 with a remote server 130 via a network 150. The server 130 can also communicate with a remote database 152 and transfer data sets 103-3 to each other. The data sets 103-1, 103-2, and 103-3 (hereinafter collectively referred to as "data sets 103") can include images, text, audio, and computer-generated 3D renditions of avatars in virtual reality conversations. In the virtual world 170, the recipient avatar 111 is involved in a conversation with the sender avatar 112. Typically, each of the head mounted devices 100 includes at least cameras 125-1 and 125-2 (hereinafter collectively referred to as "cameras 125") and microphones 121-1 and 121-2 (hereinafter collectively referred to as "microphones 121"). The camera 125 may include an inward-looking camera that, as part of an eye tracking device, detects movement of the pupils of the recipient 101 / sender 102 during the virtual conversation. The eye tracking device can determine the gaze direction of the recipient 101 / sender 102. The microphone 121 can record a voice 107-1 from the recipient 101 or a voice 107-2 from the sender 102 (hereinafter collectively referred to as "voice 107"). In some embodiments, the architecture 10 sends the speech 107 to the remote server 130 via the network 150 , where a full capture of the conversation can occur to be fed into the virtual world 170 .

[0044] In an embodiment consistent with the present disclosure, each of the devices shown in architecture 10 may include a memory storing instructions and one or more processors configured to execute the instructions to cause each device to at least partially participate in a method as disclosed herein.

[0045] In virtual world 170, it is desirable to maintain gaze 115-1 of recipient avatar 111 focused on an object of interest of recipient 101 (e.g., a flower that sender avatar 112 is showing to recipient avatar 111). In some embodiments, the object of interest of recipient 101 may be the eyes or face of avatar 112, the flower, or even a distant moving object (e.g., a car, a train, an airplane, another avatar, etc.). Thus, it is desirable to maintain gaze 115-1 focused on the flower as long as the context of the conversation in virtual world 170 requires. Accordingly, in the context of the conversation that recipient 101 and sender 102 are having in virtual world 170, it may be desirable to keep gaze 115-2 of sender avatar 112 on the face (e.g., eyes) of recipient avatar 111.

[0046] In some embodiments, the processing and transmission of the data set 103 causes inherent latency in the network 150. In addition, since the data transmission involves at least three devices (e.g., the head mounted device 100, the mobile device 110, and the remote server 130) communicating over the network 150, errors are expected to occur. Therefore, it is expected that the head mounted device 100-1 of the recipient 101 makes a decision to adjust the gaze lock of such avatar before receiving an accurate reproduction of the sender avatar 112 from the head mounted device 100-2 over the network 150. This adjustment, which is a local change made by the head mounted device 100-1, is a real-time gaze lock that gives a highly realistic impression to the conversation in the virtual world 170. In order to evaluate whether the gaze lock configuration of the sender avatar 112 is expected, the head mounted device 100-1 can be configured to analyze the context of the conversation based on the speech 107 to extract meaning and intent and other more explicit or implicit content from the conversation between the recipient and the sender.

[0047] Figure 22 is a block diagram 200 illustrating an example server 130, client device 110, and database 152 from architecture 100 according to some embodiments. Client device 110 and server 130 are communicatively coupled via network 150 via respective communication modules 218-1 and 218-2 (hereinafter collectively referred to as "communication modules 218"). Communication module 218 is configured to connect to network 150 to send information to other devices and receive information from other devices via network 150, such as data, requests, responses, and commands. Communication module 218 may be, for example, a modem or an Ethernet card, and may include radio hardware and software for wireless communication (e.g., via electromagnetic radiation, such as radio frequency (RF), near field communication (NFC), Wi-Fi, and Bluetooth radio technologies). A user may interact with client device 110 via input device 214 and output device 216. Input device 214 may include a mouse, keyboard, pointer, touch screen, microphone, joystick, virtual joystick, and the like. In some embodiments, the input device 214 may include a camera, a microphone, and sensors, such as a touch sensor, an acoustic sensor, an inertial motion unit (IMU), and other sensors configured to provide input data to a virtual reality (VR) / augmented reality (AR) head-mounted device. For example, in some embodiments, the input device 214 may include an eye tracking device to detect the position of the user's pupil in the VR / AR head-mounted device. The output device 216 may be a screen display, a touch screen, a speaker, etc.

[0048] The client device 110 may include a memory 220-1 and a processor 212-1. The memory 220-1 may include an application 222 and a graphical user interface (GUI) 225, which are configured to run in the client device 110 and are coupled with an input device 214 and an output device 216. The application 222 may be downloaded by a user from the server 130 and may be hosted by the server 130. In some embodiments, the client device 110 is a VR / AR head mounted device and the application 222 is an immersive reality application. In some embodiments, the client device 110 is a mobile phone that is used to collect videos or pictures and upload them to the server 130 using a video or image collection application 222 for storage in the training database 152. The memory 220-1 may also include an eye tracking module 242 that receives input from an eye tracking device in the head mounted device and identifies the gaze direction of the head mounted device user in the virtual world (e.g., the virtual world 170). In addition, in some embodiments, the eye tracking module 242 may also collect the pupil position and gaze direction of the avatar in the virtual world. Thus, the eye tracking module 242 may associate the gaze direction of a "receiving" avatar (e.g., a "local" avatar modeled after the user of the client device 110) with the gaze direction of a "sending" avatar (e.g., an avatar that interacts with the receiving avatar and is modeled after the user of the remote device 110). The gaze direction of the sending avatar may be provided to the client device 110 via the server 130 from a remote client device used by the sender. However, in some cases, network latency may cause the client device 110 to receive outdated signals regarding the gaze direction of the sending avatar. To avoid awkward mismatches in gaze direction between the receiving avatar and the sending avatar, the eye tracking module 242 may use a neural network tool 248 that is trained to predict the correct gaze direction of the sending avatar in the virtual world. In some embodiments, when the mismatch between the two becomes greater than a preselected threshold, the eye tracking module 242 is triggered to replace the current feed of the sender avatar's gaze with the locally predicted gaze.

[0049] In some embodiments, neural network tool 248 may be part of one or more machine learning models stored in database 152. Database 152 includes training archives and other data files that eye tracking module 242 can use to accurately predict the gaze direction of the sender avatar. In addition, in some embodiments, at least one or more training archives or machine learning models may be stored in any of multiple memories 220, and client device 110 may access these training archives or machine learning models through application 222.

[0050] Neural network tools 248 may include algorithms trained for the specific purpose of the engines and tools included therein. The algorithms may include machine learning algorithms or artificial intelligence algorithms using any linear or nonlinear algorithms, such as neural network algorithms or multivariate regression algorithms. In some embodiments, machine learning models may include: neural network (NN); convolutional neural network (CNN); generative adversarial neural network (GAN); deep reinforcement learning (DRL) algorithm; deep recurrent neural network (DRNN); classical machine learning algorithms such as random forest, k-nearest neighbor (KNN) algorithm, k-means clustering algorithm, etc.; or any combination thereof. More generally, machine learning models may include any machine learning model that involves a training step and an optimization step. In some embodiments, training database 152 may include training archives for modifying coefficients based on the expected results of the machine learning model.

[0051] The server 130 includes a memory 220-2, a processor 212-2, and a communication module 218-2. Hereinafter, the processors 212-1 and 212-2 and the memories 220-1 and 220-2 will be collectively referred to as "processor 212" and "memory 220", respectively. The processor 212 is configured to execute instructions stored in the memory 220. In some embodiments, the memory 220-2 includes an avatar engine 232. The avatar engine 232 can share or provide features and resources (including multiple tools associated with stylization, personalization, and animation) to the GUI 225, or design an application (e.g., application 222) that uses an avatar retrieved using the avatar engine 232. The user can access the avatar engine 232 through the application 222 installed in the memory 220-1 of the client device 110. Therefore, the application 222 (including the GUI 225) can be installed by the server 130, and the scripts and other routines provided by the server 130 are executed by any one of the multiple tools. Execution of application 222 may be controlled by processor 212 - 1 . In some embodiments, avatar engine 232 , tools included in avatar engine, and at least a portion of training database 152 may be hosted in a different server accessible by server 130 or client device 110 .

[0052] Figure 3Gaze adjustment 300 of a sender's gaze 301 in a receiver head mounted device from a previous configuration 370-1 to a current configuration (370-2, hereinafter collectively referred to as "virtual world configuration 370") is shown in accordance with some embodiments. At a first time (configuration 370-1), a receiver avatar 311 (having eyes 361 oriented according to pupils 371) and a sender avatar 312 may have gaze locked in virtual world configuration 370. At a later time (configuration 370-2), the receiver avatar 311 may have moved slightly, e.g., moved to maintain gaze lock with the sender avatar 312. The receiver head mounted device may adjust (locally at the receiver head mounted device) the position of the eyes 362 in the sender avatar 312 (e.g., by rotating one or both pupils 372), which is updated to account for the receiver's displacement. Then, a predicted gaze direction 302-1 is generated by a local head mounted device (e.g., a head mounted device of a user associated with the receiving avatar 311). Finally, a current gaze direction 302-2 may be provided by a sending head mounted device (e.g., a head mounted device of a user associated with the sending avatar 312). Thus, the difference between the predicted gaze direction 302-1 and the current gaze direction 302-2 may not be noticeable to the user of the receiving head mounted device, any of the users of the sending head mounted device, and any other participants in the virtual world configuration 370.

[0053] Figure 4 4. World state parameters 400 of a function for verifying the gaze lock state of a sender avatar 412 in a virtual world 470 are shown in accordance with some embodiments. A gaze point 402 forms a direction and distance 407(d) to the sender's avatar eye 462. In some embodiments, the gaze point 402 is the location of the recipient's avatar pupil in the eye 461 of the recipient avatar 411 (e.g., when a gaze lock configuration is selected). In some embodiments, the gaze point 402 may be an object of interest for the sender avatar 412 in the virtual world 470. The sender avatar gaze 401 is currently at an angle 405, α, to the direction of the gaze point 402.

[0054] The angular distance δ can be defined as:

[0055] δ=1-cosα (1.1)

[0056] The distance factor Δf can be defined as:

[0057] Δ f =g(min,max,d)+ε (1.2)

[0058] Where g is a linear step function of the minimum distance, the maximum distance, and the distance 407(d), and ε is a minimum factor. For example, in some embodiments:

[0059] g=(d-min) / (max-min) (1.3)

[0060] The function for verifying the gaze lock state of the sender avatar may include the following mathematical expression:

[0061] f=exp{-γ·δ·△f}(1.4)

[0062] where γ is an ad-hoc factor (e.g., 1×10 -6 ). When the gaze of the sender avatar 412 is close to the gaze point 402, the function, is intended to have a maximum value of one (1), and when the gaze of the sender avatar 412 is away from the contact gaze point 402, the function, is intended to have a minimum value of zero (0). Therefore, in some embodiments, the function, is a probability value that the sender avatar 412 has a gaze lock on the object of interest (e.g., the gaze of the recipient avatar 411 or another object of interest).

[0063] In some embodiments, the gaze point 402 is not a single point. For example, the gaze point 402 may include multiple points (e.g., one point for the left eye and one point for the right eye). The uncertainty function may be used to weigh different options.

[0064] In some embodiments, the gaze point 402 may be maintained as an offset or difference in the function, (see Equation 1). Thus, the receiving head mounted device may track smooth pursuit and convergence of the sending avatar 412 and head movements (e.g., in a virtual operating room (VOR)). A simplified version of local tracking (e.g., at the receiving head mounted device) of the sending avatar gaze 401 may include measurements of the object / subject of eye tracking when the gaze estimate is retrieved from the sending head mounted device (at the receiving head mounted device or at the sending head mounted device, based on the latest information). The state of the virtual world 470 is updated locally (e.g., at the receiving head mounted device) to simulate visual tracking of the object / subject. In some embodiments, a best estimate of the gaze point 402 consistent with the state of the virtual world 470 is available at the receiving head mounted device.

[0065] In some embodiments, the function may include a neural network or nonlinear regression approximation for the sender avatar 412 (e.g., provided by the neural network tool 248), where the predicted user state (t) is obtained from the function f of the user state and stage at N previous times t-1, ..., tN and the local world state (e.g., the receiving head-mounted device) at time t. In some embodiments, the local (receiving) head-mounted device can be configured to predict, estimate, and measure the sender's eye movements, including saccades, corrective saccades, nystagmus (e.g., involuntary eye movements), and time delays. In some embodiments, the local (receiving) head-mounted device can be configured to predict the sender's eye movements based on the conversation context and other social gestures and cues, which can be collected by the receiving head-mounted device and interpreted locally using a neural network, artificial intelligence, or machine learning algorithm.

[0066] Figure 5 is a flowchart showing multiple steps in a method 500 for real-time gaze locking of a sender avatar for an immersive reality application according to some embodiments. At least one or more of the multiple steps in the method 500 may be performed by one or more of the head mounted device, mobile device, server, and database (see architecture 10) disclosed herein. In addition, the memory circuit in any of the above devices may include instructions that, when executed by one or more processor circuits in the device or in any other of the above devices, may cause the device to perform at least some of the steps in the method 500. The memory circuit may include a VR / AR application, a GUI, an eye tracking module configured to receive and process signals from an eye tracking device, from an eye tracking module in a remote head mounted device, or from a sender avatar in a virtual world associated with a user of the remote head mounted device (e.g., application 222, GUI 225, eye tracking module 242, sender avatar 112, 312, and 412, and virtual world 170, 370, and 470). Furthermore, a method consistent with the present disclosure may include at least one of the steps in method 500 being performed in a different order, or simultaneously, or overlapping in time.

[0067] Step 502 includes verifying in the sender device that visual tracking of the sender avatar is currently in place. In some embodiments, step 502 includes identifying a conversation between the sender avatar and the recipient avatar. In some embodiments, step 502 includes querying the sender device for visual tracking signals. In some embodiments, step 502 includes verifying that a function value of a virtual distance between the sender avatar and the recipient avatar is greater than a preselected threshold. In some embodiments, step 502 includes verifying that a function value of an angular difference between a gaze direction of the sender avatar and a gaze point is greater than a preselected threshold.

[0068] Step 504 includes, in the receiver device, adjusting the gaze direction of the sender avatar to the gaze point. In some embodiments, step 504 includes estimating the coordinates of the gaze point in the receiver frame at a later time, and in the receiver device, rotating the two gaze directions from the sender device to track the gaze point. In some embodiments, step 504 includes selecting a time period shorter than the network delay of the signal transmitted between the sender device and the receiver device for a later time. In some embodiments, the visual tracking of the sender avatar is a gaze lock between the sender avatar and the receiver avatar, and step 404 includes estimating the relative position between the sender avatar and the receiver avatar at a later time. In some embodiments, step 504 includes updating the gaze direction at a later time to a new gaze direction when the distance between the gaze direction and the new gaze direction received from the sender device is greater than a preselected threshold. In some embodiments, step 504 includes adjusting the head direction of the sender avatar based on the gaze point at a later time. In some embodiments, step 504 includes updating the world state at a later time based on the dynamic conditions of the world state at the current time.

[0069] Hardware Overview

[0070] Figure 6 is a block diagram illustrating an exemplary computer system 600 with which the head mounted device and other client devices 110, as well as the method 500, may be implemented. In some aspects, the computer system 600 may be implemented using hardware or a combination of software and hardware in a dedicated server, or integrated into another entity, or distributed across multiple entities. The computer system 600 may include a desktop computer, a laptop computer, a tablet computer, a tablet phone, a smartphone, a feature phone, a server computer, or other computer. The server computer may be located remotely in a data center, or stored locally.

[0071] The computer system 600 includes a bus 608 or other communication mechanism for communicating information, and a processor 602 (e.g., processor 212) coupled to the bus 608 for processing information. As an example, the computer system 600 may be implemented with one or more processors 602. The processor 602 may be a general purpose microprocessor, a microcontroller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a state machine, gate logic, a plurality of discrete hardware components, or any other suitable entity that can perform calculations or other manipulations of information.

[0072] In addition to hardware, the computer system 600 may also include code that creates an execution environment for the computer program in question, such as code that constitutes the following stored in the included memory 604 (e.g., memory 220): processor firmware, protocol stack, database management system, operating system, or a combination of one or more of them, such as random access memory (Random Access Memory, RAM), flash memory, read-only memory (Read-Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable PROM (Erasable PROM, EPROM), registers, hard disk, removable disk, compact disc read-only memory (CD-ROM), digital video disc (DVD) or any other suitable storage device, which is coupled to the bus 608 for storing information and instructions to be executed by the processor 602. The processor 602 and the memory 604 may be supplemented by or incorporated in special purpose logic circuitry.

[0073] Instructions may be stored in memory 604 and may be implemented in one or more computer program products, such as one or more modules of multiple computer program instructions encoded on a computer-readable medium for execution by computer system 600 or for controlling the operation of computer system 600, and according to any method known to those skilled in the art, including but not limited to computer languages ​​such as data-oriented languages ​​(e.g., SQL, dBase), system languages ​​(e.g., C, object-oriented programming languages ​​extending C (Objective-C), C++, assembly), structured languages ​​(e.g., Java, .NET), and application languages ​​(e.g., PHP, Ruby, Perl, Python). The instructions may also be implemented in computer languages ​​such as array languages, aspect-oriented languages, assembly languages, authoring languages, command line interface languages, compiled languages, concurrent languages, curly-bracket languages, data flow languages, data structured languages, declarative languages, esoteric languages, extension languages, fourth generation languages, functional languages, interactive mode languages, interpreted languages, iterative languages, list-based languages, little languages, logic-based languages, machine languages, macro languages, metaprogramming languages, multiparadigm languages, numerical analysis, non-English-based languages, class-based object-oriented languages, prototype-based object-oriented languages, off-side rule languages, procedural languages, reflective languages, rule-based languages, scripting languages, stack-based languages, synchronous languages, syntax handling languages, visual languages, wirth languages, and XML-based languages. Memory 604 may also be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 602 .

[0074] Computer programs as discussed herein do not necessarily correspond to files in a file system. A program may be stored in a portion of a file storing other programs or data (e.g., one or more scripts stored in a markup language document), a single file dedicated to the program in question, or multiple collaborative files (e.g., a file storing one or more modules, subroutines, or partial codes). A computer program may be deployed to execute on one or more computers, which are located at a site or distributed across multiple sites and interconnected by a communication network. The processes and logic flows described in this specification may be performed by one or more programmable processors, which execute one or more computer programs to perform functions by operating on input data and generating outputs.

[0075] The computer system 600 also includes a data storage device 606 such as a disk or optical disk, which is coupled to the bus 608 for storing information and instructions. The computer system 600 can be coupled to various devices via an input / output module 610. The input / output module 610 can be any input / output module. An exemplary input / output module 610 includes a data port such as a USB port. The input / output module 610 is configured to be connected to a communication module 612. An exemplary communication module 612 includes a network interface card, such as an Ethernet card and a modem. In some aspects, the input / output module 610 is configured to be connected to multiple devices, such as an input device 614 and / or an output device 616. An exemplary input device 614 includes a keyboard and a pointing device (e.g., a mouse or a tracking ball), and a user (consumer) can provide input to the computer system 600 through the keyboard and the pointing device. Other types of input devices 614 can also be used to provide interaction with the user, such as a tactile input device, a visual input device, an audio input device, or a brain-computer interface device. For example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and the input from the user can be received in any form including sound input, voice input, tactile input, or brain wave input. Exemplary output device 616 includes a display device for displaying information to the user, such as a liquid crystal display (LCD) monitor.

[0076] According to one aspect of the present disclosure, the head mounted device and the client device 110 may be implemented at least in part using a computer system 600 in response to a processor 602 executing one or more sequences of one or more instructions contained in a memory 604. These instructions may be read into the memory 604 from another machine-readable medium (e.g., a data storage device 606). The execution of the sequence of instructions contained in the main memory 604 causes the processor 602 to perform the process steps described herein. One or more processors in a multi-processing configuration may also be employed to execute the sequence of instructions contained in the memory 604. In alternative aspects, hard-wired circuits may be used in place of software instructions, or hard-wired circuits may be used in combination with software instructions to implement various aspects of the present disclosure. Therefore, aspects of the present disclosure are not limited to any specific combination of hardware circuitry and software.

[0077] Aspects of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification); or aspects of the subject matter described in this specification may be implemented in any combination of one or more such back-end components, one or more such middleware components, or one or more such front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). The communication network may, for example, include any one or more of a LAN, a WAN, the Internet, etc. In addition, for example, the communication network may include, but is not limited to, any one or more of the following network topologies, including a bus network, a star network, a ring network, a mesh network, a star bus network, a tree or hierarchical network, etc. The communication module may, for example, be a modem or an Ethernet card.

[0078] Computer system 600 can include client and server.Client and server are usually far away from each other, and usually interact through communication network.The relationship between client and server is generated by means of computer programs running on respective computers and having client-server relationship between each other.Computer system 600 can be but not limited to, for example, desktop computer, laptop computer or tablet computer.Computer system 600 can also be embedded in another device, and this another device is for example but not limited to: mobile phone, personal digital assistant (PDA), mobile audio player, global positioning system (Global Positioning System, GPS) receiver, video game console and / or TV set-top box.

[0079] As used herein, the term "machine-readable storage medium" or "computer-readable medium" refers to any one or more media that participate in providing instructions to processor 602 for execution. Such media can take many forms, including, but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical or magnetic disks, such as data storage device 606. Volatile media include dynamic memory, such as memory 604. Transmission media include coaxial cables, copper wires, and optical fibers, including the wires that form bus 608. Common forms of machine-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs, any other optical media, punch cards, paper tapes, any other physical media with hole patterns, RAM, PROMs, EPROMs, FLASH EPROMs, any other memory chips or cassettes, or any other media that a computer can read. The machine-readable storage medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them.

[0080] For example, the subject technology is illustrated according to the various aspects described below. For convenience, various examples of various aspects of the subject technology are described as numbered claims (claims 1, 2, etc.). These are provided as examples and do not limit the subject technology.

[0081] In one aspect, a method may be an operation, an instruction, or a function, and vice versa. In one aspect, a clause may be modified to include some or all of the words (e.g., instructions, operations, functions, or components) in one or more clauses, one or more words, one or more sentences, one or more phrases, one or more paragraphs, and / or a plurality of words stated in one or more clauses.

[0082] To illustrate the interchangeability of hardware and software, items such as various illustrative blocks, modules, components, methods, operations, instructions, and algorithms are generally described in terms of the functionality of hardware and software. Whether such functionality is implemented as hardware, software, or a combination of hardware and software depends on the specific application and design constraints imposed on the overall system. A skilled person may implement the described functionality in different ways for each specific application.

[0083] As used herein, the phrase "at least one of..." following a list of items, together with the terms "and" or "or" used to separate any of the items, modifies the list as a whole, rather than modifying each member of the list (e.g., each item). The phrase "at least one of..." does not require selection of at least one item; rather, the phrase is meant to include at least one of any of the items, and / or at least one of any combination of the items, and / or at least one of each of the items. As an example, the phrase "at least one of A, B, and C" or "at least one of A, B, or C" both refer to: only A, only B, or only C; any combination of A, B, and C; and / or, at least one of each of A, B, and C.

[0084] The word "exemplary" is used herein to mean "used as an example, instance or illustration". Any embodiment described as "exemplary" herein is not necessarily to be interpreted as being preferred or advantageous over other embodiments. Phrases such as on the one hand, this aspect, on the other hand, some aspects, one or more aspects, an implementation, this implementation, another implementation, some implementations, one or more implementations, an embodiment, this embodiment, another embodiment, some embodiments, one or more embodiments, a configuration, this configuration, another configuration, some configurations, one or more configurations, user technology, the disclosure, the present disclosure, and other variations thereof are only for convenience, and do not imply that the disclosure associated with one or more such phrases is essential to user technology, or that such disclosure is applicable to all configurations of user technology. The disclosure associated with one or more such phrases may be applicable to all configurations, or one or more configurations. The disclosure associated with one or more such phrases may provide one or more examples. Phrases such as on the one hand or some aspects may refer to one or more aspects, and vice versa, and this is similarly applicable to other aforementioned phrases.

[0085] Unless otherwise specified, references to elements in the singular are not intended to mean "one and only one", but "one or more". Positive pronouns (e.g., his) include negative and neuter genders (e.g., her and its), and vice versa. The term "some" refers to one or more. Underlined and / or italicized titles and subtitles are used only for convenience, do not limit user technology, and do not refer to the interpretation of the description of user technology. Relational terms such as first and second can be used to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between these entities or actions. All structural and functional equivalents of the elements of the various configurations described throughout this disclosure that are known or will be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be included by user technology. In addition, nothing disclosed herein is intended to be dedicated to the public, whether or not such disclosure is explicitly recorded in the above description. No clause element shall be construed under the provisions of 35 USC § 112, paragraph 6, unless the element is expressly recited using the phrase “means for” or, in the case of a method clause, the element is recited using the phrase “steps for”.

[0086] Although this specification contains many details, these details should not be interpreted as limitations on the scope of the content that may be described, but should be interpreted as descriptions of specific implementations of user matters (user matter). Certain features described in the context of different embodiments in this specification may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented separately or in any suitable sub-combination in multiple embodiments. In addition, although features may be described above as working in certain combinations and even initially described as such, in some cases, one or more features from the described combination may be removed from the combination, and the described combination may be for a sub-combination or a variation of the sub-combination.

[0087] The user matters of this specification have been described in certain aspects, but other aspects can be implemented and are within the scope of the above clauses. For example, although each operation is depicted in a specific order in the accompanying drawings, this should not be understood as requiring the operations to be performed in the specific order shown or in a continuous order, or requiring the execution of all the operations shown to achieve the desired result. The actions stated in the various clauses can be performed in different orders and still achieve the desired results. As an example, the process depicted in the accompanying drawings does not necessarily require the specific order shown or the continuous order to achieve the desired result. In some cases, multi-task parallel processing may be advantageous. In addition, the separation of the various system components in the above-mentioned multiple aspects should not be understood as requiring such separation in all aspects, but it should be understood that the described program components and systems can usually be integrated together in one software product or encapsulated in multiple software products.

[0088] Names, background technologies, drawings and their descriptions are incorporated into the present disclosure herein and are provided as illustrative examples of the present disclosure rather than as restrictive descriptions. It should be understood that they will not be used to limit the scope or meaning of the various clauses. In addition, in the specific implementation, it can be seen that the present specification provides illustrative examples, and for the purpose of streamlining the present disclosure, various features are grouped together in various implementations. The method of the present disclosure should not be interpreted as reflecting the following intention: the user matters described require more features than the features explicitly recorded in each clause. Instead, as reflected in the various clauses, the creative user matters lie in less than all the features of a single disclosed configuration or operation. The various clauses are incorporated into the specific implementation herein, wherein each clause is independently described as a user matter.

[0089] For example, the subject technology is illustrated according to the various aspects described below. For convenience, various examples of various aspects of the subject technology are described as numbered claims (claims 1, 2, etc.). These are provided as examples and do not limit the subject technology. The claims are not intended to be limited to the various aspects described herein, but should be given the full scope consistent with the language terms and include all legal equivalents. Nevertheless, any claim in the claims is not intended to include user matters that fail to meet the requirements of the applicable patent law, nor should the claims be interpreted in this way.

Claims

1. A computer-implemented method comprising: verifying in the receiving device that visual tracking of the sender's avatar is active in the sending device; as well as In the receiving device, adjusting the gaze direction of the sending avatar to the gaze point, wherein: Adjusting the gaze direction of the sender avatar includes: estimating the coordinates of the gaze point in a receiver frame at a later time; and in the receiver device, rotating two eyeballs of the sender avatar to a point in the direction of the gaze point.

2. The computer-implemented method of claim 1 , wherein: Verifying that the visual tracking of the sender avatar is currently in place includes identifying a conversation between the sender avatar and the recipient avatar.

3. The computer-implemented method of claim 1 or 2, wherein: Verifying that visual tracking of the sender avatar is currently in place includes querying the sender device for a visual tracking signal.

4. A computer-implemented method according to any one of the preceding claims, wherein: Verifying that the visual tracking of the sender avatar is currently in place includes verifying that a function value of a virtual distance between the sender avatar and the recipient avatar is greater than a preselected threshold.

5. A computer-implemented method according to any one of the preceding claims, wherein: Verifying that the visual tracking of the sender avatar is currently in place includes verifying that a function value of an angular difference between the gaze direction of the sender avatar and the gaze point is greater than a preselected threshold.

6. The computer-implemented method of any preceding claim, further comprising: A time period shorter than a network delay for signal transmission between the sender device and the receiver device is selected for the later time.

7. A computer-implemented method according to any one of the preceding claims, wherein: The visual tracking of the sender avatar is a gaze lock between the sender avatar and a recipient avatar, and estimating the coordinates of the gaze point includes estimating a relative position between the sender avatar and the recipient avatar at the later time.

8. The computer-implemented method of any preceding claim, further comprising: When a distance between the gaze direction and a new gaze direction received from the sender device is greater than a preselected threshold, the gaze direction at the later time is updated to the new gaze direction.

9. The computer-implemented method of any preceding claim, further comprising: adjusting a head direction of the sender avatar based on the gaze point at the later time; And / or preferably, the computer-implemented method further comprises: updating the world state at the later time based on a dynamic condition of the world state at the current time.

10. A system comprising: A memory storing a plurality of instructions; as well as one or more processors configured to execute the plurality of instructions so that the system: Verify that the visual tracking of the sender avatar is active; as well as In a receiving device, a gaze direction of the sending avatar is adjusted to a gaze point, wherein, in order to adjust the gaze direction of the sending avatar, the one or more processors execute instructions for: estimating coordinates of the gaze point in a receiving frame at a later time; and in the receiving device, rotating two gaze directions from the sending device to track the gaze point.

11. The system according to claim 10, wherein: To verify that visual tracking of a sender avatar is currently in place, the one or more processors execute instructions for identifying a conversation between the sender avatar and a recipient avatar.

12. The system according to claim 10 or 11, wherein: In order to verify that the visual tracking of the sender avatar is currently in place, the one or more processors execute instructions for the following operations: query the sender device for a visual tracking signal; and / or preferably, wherein, in order to verify that the visual tracking of the sender avatar is currently in place, the one or more processors execute instructions for the following operations: verify that the function value of the virtual distance between the sender avatar and the recipient avatar is greater than a preselected threshold.

13. A system according to any one of claims 10 to 12, wherein: In order to verify that the visual tracking of the sender avatar is currently in place, the one or more processors execute instructions for the following operations: verifying that the function value of the angular difference between the gaze direction of the sender avatar and the gaze point is greater than a preselected threshold; and / or preferably, wherein the one or more processors also execute instructions for the following operations: selecting a time period for the later time that is shorter than the network delay for the signal to be transmitted between the sender device and the recipient device.

14. A system according to any one of claims 10 to 13, wherein: The one or more processors execute instructions for the following operations: visually tracking the sending avatar using gaze lock between the sending avatar and the receiving avatar, and in order to estimate the coordinates of the gaze point, the one or more processors execute instructions for the following operations: estimating the relative position between the sending avatar and the receiving avatar at the later time; and / or preferably, wherein the one or more processors execute instructions for the following operations: when the distance between the gaze direction and a new gaze direction received from the sending device is greater than a preselected threshold, updating the gaze direction at the later time to the new gaze direction.

15. A system according to any one of claims 10 to 14, wherein: The one or more processors also execute instructions for: adjusting the head direction of the sender avatar based on the gaze point at the later time; and / or preferably, wherein the one or more processors also execute instructions for: updating the world state at the later time based on the dynamic conditions of the world state at the current time.

Images