Embedded sensor for realizing social presence in immersive real-time head-mounted device

Abstract

A method for updating an object avatar with facial expressions collected in real time. The method includes receiving a signal from a sensor on a facial interface of the head-mounted device indicative of motion of facial muscles of a user of the head-mounted device; determining a facial expression of the user with a signal from a sensor in a facial interface of the head-mounted device based on a machine learning algorithm trained to associate the facial expression with motion of facial muscles; adjusting an object avatar of a user of the head-mounted device based on the facial expression; and providing the object avatar to an immersive reality application hosted by the remote server. There is also provided a head-mounted device, a computer readable medium storing a plurality of instructions, and a processor, which, when executed by the processor, cause the head-mounted device to perform the above method.

Description

Technical Field

[0001] The present disclosure relates to virtual reality and augmented reality (VR / AR) headsets configured with multiple facial sensors to achieve social presence in immersive reality applications. More specifically, the present disclosure is associated with embedded sensors in VR / AR headsets to provide information and incorporate a user's facial expressions in real time into an object avatar for immersive reality applications. Background Art

[0002] Current VR / AR headsets incorporate an eye-tracking system and an inward-looking camera (e.g., an infrared camera, etc.) and an ultrasonic scanner to collect real-time information about a user's facial expressions. Such information is related to generating an object avatar with a true social presence in an immersive reality environment based on the VR / AR headset user. However, eye-tracking systems, cameras, and scanners tend to be overly complex devices, bulky and heavy, involve heavy data protocols and operations, and consume a large amount of power. Summary of the Invention

[0003] According to a first aspect of the present disclosure, there is provided a computer-implemented method, the computer-implemented method comprising: receiving a signal from a sensor on a facial interface of a headset indicating movement of facial muscles of a user of the headset; determining, based on a machine learning algorithm, a facial expression of the user using signals from sensors in the facial interface of the headset, the machine learning algorithm being trained to associate facial expressions with movements of facial muscles; adjusting an object avatar of the user of the headset based on the facial expression; and providing the object avatar to an immersive reality application hosted by a remote server.

[0004] In some embodiments, receiving a signal from the sensor includes receiving a signal from one of the following: an inertial motion unit, an electronic sensor, a capacitive sensor, a contact microphone, an optical sensor, a tactile sensor, a humidity sensor, and a temperature sensor.

[0005] In some embodiments, the signal is an electrical signal of neural activation from facial muscles, and the method further includes identifying facial muscles by the positions of the sensors on the facial interface.

[0006] In some embodiments, receiving a signal from the sensor includes receiving multiple signals from multiple sensors disposed on a facial interface of the headset, and the method further includes: correlating the multiple signals to evaluate whether and to what extent one or more facial muscles of the user of the headset move.

[0007] In some embodiments, adjusting a user's object avatar includes: shaping a three-dimensional solid model of the face of a user of a head-mounted device based on facial expressions.

[0008] In some embodiments, the facial muscles are the muscles that move the eye pupils of the user of the head-mounted device, and determining the user's facial expression includes determining the gaze direction of the user of the head-mounted device.

[0009] In some embodiments, the method further includes: determining the user's physical condition based on the user's facial expression.

[0010] In some embodiments, the method further includes: determining the user's mental condition based on the user's facial expression and the environment in an immersive reality application.

[0011] In some embodiments, the method further includes: displaying a feedback message to the user in the head-mounted device based on the facial expression.

[0012] In some embodiments, the method further includes: modifying the user's facial expression based on the audience in an immersive reality application.

[0013] According to a second aspect of the present disclosure, there is provided a head-mounted device, the head-mounted device including: a facial interface, the facial interface including one or more sensors and configured to contact the skin around two eyes and the nose in the face of a user of the head-mounted device, wherein the one or more sensors are geometrically arranged on the facial interface to identify the movement of the facial muscles of the user of the head-mounted device; a memory, the memory storing instructions and a chart, the chart including a mapping from the facial expressions of the user of the head-mounted device to the movement of the facial muscles; a processor, the processor configured to at least partially receive a plurality of signals from the one or more sensors and identify the facial pose of the user of the head-mounted device based on the plurality of signals and the chart; a communication module, the communication module configured to send the plurality of signals and the facial pose to a remote server that hosts an immersive reality application including an avatar of the user of the head-mounted device.

[0014] In some embodiments, the one or more sensors include at least one of the following: an inertial motion sensor, an electronic sensor, a capacitance sensor, a contact microphone, an optical sensor, a tactile sensor, a humidity sensor, and a temperature sensor.

[0015] In some embodiments, the one or more sensors include a photoplethysmography sensor for determining cardiovascular activity across the nose of the user of the head-mounted device.

[0016] In some embodiments, one or more sensors include a plurality of electronic sensors that are symmetrically disposed around both eyes of a user of the head-mounted device and configured to evaluate the gaze direction of the user of the head-mounted device.

[0017] In some embodiments, a portion of the facial interface that includes one or more sensors is detachable from the head-mounted device.

[0018] According to a third aspect of the present disclosure, there is provided a computer-implemented method that includes: providing an object avatar of a first participant to one or more head-mounted devices used by a plurality of participants in an immersive reality application from a remote server; receiving a signal from the first head-mounted device of the first participant indicating movement of facial muscles of the first participant; determining a facial pose of the first participant based on the signal; and updating the object avatar of the first participant with the facial pose.

[0019] In some embodiments, receiving a signal indicating movement of facial muscles of the first participant includes receiving a signal from at least one of the following: an inertial motion sensor, a contact microphone, an electronic sensor, a capacitance sensor, an optical sensor, a tactile sensor, a humidity sensor, and a temperature sensor.

[0020] In some embodiments, receiving a signal indicating movement of facial muscles includes: integrating a plurality of signals from one or more sensors in a facial interface of the first head-mounted device.

[0021] In some embodiments, the method further includes: receiving a signal from a head-mounted device of a second participant indicating movement of facial muscles of the second participant; determining a facial pose of the first participant based on the signal; and updating the object avatar of the second participant with the facial pose.

[0022] In some embodiments, determining a facial pose of the first participant includes: conferring with a chart that correlates movement of facial muscles with facial poses.

[0023] In other embodiments, a non-transitory computer-readable medium stores a plurality of instructions that, when executed by a processor, cause a computer to perform a method. The method includes: receiving a signal from a sensor on a facial interface of a head-mounted device indicating movement of facial muscles of a user of the head-mounted device; determining, based on a machine learning algorithm, a facial expression of the user with signals from sensors in the facial interface of the head-mounted device, the machine learning algorithm being trained to correlate facial expressions with movement of facial muscles; adjusting an object avatar of the user of the head-mounted device based on the facial expression; and providing the object avatar to an immersive reality application hosted by a remote server.

[0024] In yet another embodiment, a system includes: a first device for storing instructions, and a second device for executing the instructions and causing the system to perform a method. The method includes: receiving a signal from a sensor on a face interface of a head-mounted device indicating movement of facial muscles of a user of the head-mounted device; determining, based on a machine learning algorithm, a facial expression of the user using signals from sensors in the face interface of the head-mounted device, the machine learning algorithm being trained to correlate facial expressions with movements of facial muscles; adjusting an object avatar of the user of the head-mounted device based on the facial expression; and providing the object avatar to an immersive reality application hosted by a remote server.

[0025] These and other embodiments will be apparent to those of ordinary skill in the art in view of the following disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 An architecture including a head-mounted device coupled to a mobile device, a remote server, and a database is shown in accordance with some embodiments.

[0027] Figure 2 A face interface of a head-mounted device including a plurality of sensors is shown in accordance with some embodiments.

[0028] Figure 3 A chart of a plurality of facial expressions associated with the facial muscles that activate them is shown in accordance with some embodiments.

[0029] Figure 4 A flowchart showing steps in a method for updating an object avatar of an immersive reality application in real time is shown.

[0030] Figure 5 A flowchart showing steps in a method for updating an object avatar using sensor information in a VR / AR head-mounted device is shown.

[0031] Figure 6 A block diagram of a computer system for a method as disclosed herein is shown.

[0032] In the drawings, unless otherwise expressly specified, elements having the same or similar reference numerals are associated with the same or similar attributes and characteristics. DETAILED DESCRIPTION

[0033] In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to one of ordinary skill in the art that embodiments of the present disclosure may be practiced without some of these specific details. In other instances, well-known structures and techniques have not been shown in detail so as not to obscure the present disclosure.

[0034] General Overview

[0035] In current VR / AR head-mounted devices, eye tracking and face tracking enable social presence and are core technologies for VR platforms. State-of-the-art eye tracking and face tracking hardware use imaging systems that require a large amount of system power and computational cycles, are relatively challenging to implement with high precision, and include expensive hardware. To address the above technical problems that arise in the technical field of immersive reality applications, the embodiments disclosed herein use a set of analog sensors to implement facial muscle tracking and achieve social presence at low cost, low power, and without TTL cost to the eye comfort zone.

[0036] In addition, the embodiments disclosed herein utilize a combination of pressure / tactile sensors, temperature sensors, audio, and potentially other sensors (e.g., inertial motion unit (IMU)) in the head-mounted device face interface and other surfaces in contact with the user to implement eye tracking and face tracking features derived from muscle contractions of the user. The sensors can identify which muscle is contracting, and the software can use this input to reconstruct facial muscle movements in an avatar or to understand which direction in the scene the user's gaze is directed. The pressure sensors will be able to identify muscle movements of the face relative to the fixed head-mounted device, and the temperature sensors can detect an increase in blood flow to the muscle, which is a sign of activation. Compared with image processing, low-voltage pressure and temperature signals require a lower computational load, thereby potentially achieving savings in power, cost, and heat. In addition, the embodiments disclosed herein can also assist optical technologies to obtain better performance and lower power.

[0037] Exemplary System Architecture

[0038] Figure 1FIG. 10 shows an architecture 10 of a VR / AR headset device 100 including a user 101 according to some embodiments. The headset device is coupled to another headset device, coupled to a mobile device 110, a remote server 130, and a database 152. The VR / AR headset device 100 may include smart glasses or an augmented reality headset device, and the mobile device 110 may be a smart phone. All these devices may communicate with each other via wireless communication and exchange a first data set 103-1. In some embodiments, the mobile device 110 may also belong to the user 101. The data set 103-1 may include a recorded video, a recorded audio, or some other file or streaming media. The user of the wearable device 100 is also the owner of the mobile device 110 or is associated with the mobile device 110.

[0039] The mobile device 110 may be communicatively coupled to the remote server 130 and the database 152 via a network 150, and transmit / share information and files, etc. (e.g., a data set 103-2 and a data set 103-3) with each other.

[0040] In some embodiments, the VR / AR headset device 100 may include a plurality of sensors 125-1, 125-2, and 125-3 (collectively referred to as "sensors 125" hereinafter) installed within a frame of the VR / AR headset device 100, such as an inertial measurement unit (IMU), a gyroscope, a microphone, and a camera, etc. Other sensors 125 that may be included within the VR / AR headset device 100 may be a magnetometer, a microphone, a photodiode, and a camera, a touch sensor, and other electromagnetic devices (e.g., a capacitance sensor and a pressure sensor, etc.). The smart glasses 100 may include an acoustic microphone and a contact microphone. The acoustic microphone receives an acoustic signal propagated through air as a pressure wave. The contact microphone 125-2 may be mechanically coupled to the skin and bones of the user, for example, in a face interface of the VR / AR headset device 100, and mechanically coupled to the skin and bones of the user when in contact with the user's face, etc.

[0041] In addition, the VR / AR headset 100 or the mobile device 110 may include a memory circuit 120 storing instructions and a processor circuit 112 configured to execute the instructions to cause the smart glasses 100 to at least partially perform some of the multiple steps in the methods consistent with the present disclosure. In some embodiments, the VR / AR headset 100, the mobile device 110, the server 130, and / or the database 152 may further include a communication module 118 that enables the devices to wirelessly communicate with the remote server 130 via the network 150. In some embodiments, the communication module 118 may include, for example, radio frequency hardware (e.g., antennas, filters, and analog-to-digital converters, etc.) and software (e.g., signal processing software). Thus, the VR / AR headset 100 may download multimedia online content (e.g., the data set 103-1) from the remote server 130 to at least partially perform some of the multiple operations in the methods disclosed herein. For example, the network 150 may include any one or more of the following: a local area network (LAN), a wide area network (WAN), and the Internet, etc. In addition, the network may include, but is not limited to, any one or more of the following network topologies, including bus networks, star networks, ring networks, mesh networks, star-bus networks, and tree or hierarchical networks, etc.

[0042] In certain embodiments, privacy settings may allow a user to specify whether current, past, or predicted mood, sentiment, or affect information associated with the user can be determined and whether a particular application or process can access, store, or use such information. The privacy settings may allow the user to select to permit or select not to permit a particular application or process to access, store, or use mood, sentiment, or affect information. Architecture 10 may predict or determine mood, sentiment, or affect associated with the user based on, for example, input provided by the user and interactions with a particular object, such as the user viewing a page or content, the user uploading a post or other content, and interactions with other content of an online social network. In certain implementations, architecture 10 may use the user's previous activities and computed mood, sentiment, or affect to determine the current mood, sentiment, or affect. A user desiring to enable this functionality may indicate in their privacy settings that they select to permit architecture 10 to receive the input necessary to determine mood, sentiment, or affect. By way of example and not limitation, architecture 10 may determine that the default privacy setting will not receive any information necessary to determine mood, sentiment, or affect until the user explicitly indicates that architecture 10 may do so. Conversely, if the user does not select to permit architecture 10 to receive such input (or affirmatively selects not to permit architecture 10 to receive such input), architecture 10 may be blocked from receiving, collecting, recording, or storing such input or any information associated with such input. In certain embodiments, architecture 10 may use predicted mood, sentiment, or affect to provide recommendations or advertisements to the user. In certain embodiments, if the user desires to use this functionality for a particular purpose or application, the user may specify additional privacy settings to select to permit mood, sentiment, or affect information to be used for the particular purpose or application. By way of example and not limitation, architecture 10 may use the user's mood, sentiment, or affect to provide news feed entries, pages, friends, or advertisements. The user may specify in their privacy settings that architecture 10 may determine the user's mood, sentiment, or affect. The user may then be asked to provide additional privacy settings to indicate the purposes for which the user's mood, sentiment, or affect may be used. The user may indicate that architecture 10 may use his or her mood, sentiment, or affect to provide news feed content and recommended pages, but not for recommending friends or advertisements. Architecture 10 may then provide only news feed content or pages based on the user's mood, sentiment, or affect and may not use the information for any other purpose even if the privacy settings do not explicitly prohibit it.

[0043] Figure 2FIG. 250 shows a face interface 250 for a head-mounted device 200 according to some embodiments, the face interface including a plurality of sensors 225 applied to the face of a user 201. The sensors 225 may include electromagnetic sensors such as electrocardiogram (ECG) sensors, electroencephalogram (EEG) sensors, capacitance sensors (or contact sensors), or electric dipole sensors, etc. The sensors 225 may also include microphones (e.g., contact microphones), and IMU sensors (e.g., accelerometers and gyroscopes, etc.).

[0044] In some embodiments, the sensors 225 may include optical sensors (e.g., photoplethysmography (PPG) sensors) to evaluate the cardiovascular condition of the user 201 (e.g., heart rate and blood pressure from facial arteries). In some embodiments, the sensors 225 may include tactile sensors, humidity sensors, and / or temperature sensors.

[0045] Figure 3Chart 300 showing the following facial expressions according to some embodiments: Facial expression 301-1 (e.g., severe), Facial expression 301-2 (e.g., indignant), Facial expression 301-3 (e.g., angry), Facial expression 301-4 (e.g., enraged), Facial expression 301-5 (e.g., contemptuous), Facial expression 301-6 (e.g., averse), Facial expression 301-7 (e.g., disgusted), Facial expression 301-8 (e.g., extremely averse), Facial expression 301-9 (e.g., worried), Facial expression 301-10 (e.g., anxious), Facial expression 301-11 (e.g., fearful), Facial expression 301-12 (e.g., panicked), Facial expression 301-13 (e.g., satisfied), Facial expression 301-14 (e.g., happy), Facial expression 301-15 (e.g., joyful), Facial expression 301-16 (e.g., laughing), Facial expression 301-17 (e.g., dejected), Facial expression 301-18 (e.g., melancholy), Facial expression 301-19 (e.g., sad), Facial expression 301-20 (e.g., grief-stricken), Facial expression 301-21 (e.g., alert), Facial expression 301-22 (e.g., surprised), Facial expression 301-23 (e.g., astonished), and Facial expression 301-24 (e.g., shocked) (hereinafter collectively referred to as "Facial expression 301"), which are associated with the following facial muscles that activate them: Facial muscle 311-1 (occipitofrontalis, right), Facial muscle 311-2 (occipitofrontalis, left), Facial muscle 311-3 (temporalis, right), Facial muscle 311-4 (temporalis, left), Facial muscle 311-5 (orbital, right), Facial muscle 311-6 (orbital, left), Facial muscle 311-7 (maxillary, right), Facial muscle 311-8 (maxillary, left), Facial muscle 311-9 (mentalis, right), Facial muscle 311-10 (mentalis, left), Facial muscles 311-11 and 311-12 (orbicularis oris), and Facial muscle 311-13 (nasalis) (hereinafter collectively referred to as "Facial muscle 311").

[0046] For each of the facial expressions 301, the graph 300 may include the different facial muscles 311 that are activated and their degree of activation (e.g., "strong", "slight", "relaxed"). Muscle activation can be detected by the following sensors: an IMU sensor, a capacitance sensor, or an electronic sensor that detects the electric field in the neurons that activate the muscle or in the muscle movement itself. A humidity sensor can also detect the muscle movement or anxiety pattern of the user's face. Additionally, the graph 300 may be supplemented with a list of facial arteries 321-1, 321-2, 321-3, and 321-4 (collectively referred to hereinafter as arteries 321). Thus, the graph 300 may also be linked to each of the facial expressions 301, or to a particular artery 321 that increases its blood flow (taking into account the muscles that need to be activated). As disclosed herein, the blood flow through the arteries 321 can be detected by a PPG sensor, or even a temperature sensor.

[0047] Figure 4 is a flowchart showing steps in a method for real-time updating of an object avatar in an immersive reality application. Embodiments consistent with the present disclosure may include at least one or more of the multiple steps in method 400, which are performed by a processor circuit executing instructions stored in a memory circuit in a head-mounted device, a mobile device, or a remote server as disclosed herein (e.g., processor circuit 112 and memory circuit 120, head-mounted device 100, mobile device 110, and remote server 130). In some embodiments, a method consistent with the present disclosure includes receiving data from a plurality of sensors mounted on a facial interface of a head-mounted device as disclosed herein (e.g., sensors 125 and facial interface 250 in head-mounted device 100 or 200). In another embodiment, a method consistent with the present disclosure may include one or more of the multiple steps in method 400 performed in a different order, simultaneously, quasi-simultaneously, or temporally overlapping.

[0048] Step 402 includes receiving a signal from a sensor on the facial interface of the head-mounted device that indicates movement of the facial muscles of a user of the head-mounted device. In some embodiments, step 402 includes receiving a signal from one of the following: an inertial motion unit, an electronic sensor, a capacitance sensor, a contact microphone, an optical sensor, a tactile sensor, and a temperature sensor. In some embodiments, the signal is an electrical signal of neural activation from the facial muscles, and step 402 includes: identifying the facial muscles by the position of the sensors on the facial interface. In some embodiments, step 402 includes receiving a plurality of signals from a plurality of sensors disposed on the facial interface of the head-mounted device, and further includes: correlating the plurality of signals with each other to evaluate whether and to what extent one or more facial muscles of a user of the head-mounted device move.

[0049] Step 404 includes: determining a user's facial expression using signals from sensors in a facial interface of a head-mounted device, based on a chart that maps facial expressions to movements of facial muscles. In some embodiments, the facial muscles are the muscles that move the user's eye pupils of the head-mounted device, and step 404 includes determining the gaze direction of the user of the head-mounted device. In some embodiments, step 404 further includes determining the user's physical condition based on the user's facial expression. In some embodiments, step 404 further includes determining the user's mental condition based on the user's facial expression and the environment in the immersive reality application.

[0050] Step 406 includes adjusting an object avatar of the user of the head-mounted device based on the facial expression. In some embodiments, step 406 includes shaping a three-dimensional solid model of the user's face of the head-mounted device based on the facial expression.

[0051] Step 408 includes providing the object avatar to an immersive reality application hosted by a remote server. In some embodiments, step 408 further includes displaying a feedback message to the user in the head-mounted device based on the facial expression. In some embodiments, step 408 includes modifying the user's facial expression based on an audience in the immersive reality application.

[0052] Figure 5 is a flowchart showing steps in a method of updating an object avatar using sensor information in a VR / AR head-mounted device according to some embodiments. Embodiments consistent with the present disclosure may include at least one or more of the plurality of steps in method 500, which are performed by a processor circuit executing instructions stored in a memory circuit in a head-mounted device, a mobile device, or a remote server as disclosed herein (e.g., processor circuit 112 and memory circuit 120, head-mounted device 100, mobile device 110, and remote server 130). In some embodiments, a method consistent with the present disclosure includes receiving data from a plurality of sensors (e.g., sensors 125 and facial interface 250 in head-mounted device 100 or 200) mounted on a facial interface of a head-mounted device as disclosed herein. In yet another embodiment, a method consistent with the present disclosure may include one or more of the plurality of steps in method 500 performed in a different order, simultaneously, quasi-simultaneously, or temporally overlapping.

[0053] Step 502 includes providing an object avatar of a first participant from a remote server to one or more head-mounted devices used by a plurality of participants in an immersive reality application.

[0054] Step 504 includes receiving, from a first head-mounted device of a first participant, a signal indicating movement of facial muscles of the first participant. In some embodiments, step 504 includes receiving a signal from at least one of the following: an inertial motion sensor, a contact microphone, an electronic sensor, a capacitance sensor, an optical sensor, a tactile sensor, and a temperature sensor. In some embodiments, step 504 includes integrating multiple signals from one or more sensors in a facial interface of the first head-mounted device. In some embodiments, step 504 includes receiving, from a head-mounted device of a second participant, a signal indicating movement of facial muscles of the second participant; determining, based on the signal, a facial pose of the first participant; and updating, with the facial pose, an object avatar of the second participant.

[0055] Step 506 includes determining, based on the signal, a facial pose of the first participant. In some embodiments, step 506 includes consulting a chart that correlates movement of facial muscles with facial poses.

[0056] Step 508 includes updating, with the facial pose, an object avatar of the first participant.

[0057] Hardware Overview

[0058] Figure 6 is a block diagram showing a computer system for implementing a head-mounted device and a method of using the same according to some embodiments. In certain aspects, the computer system 600 can be implemented using hardware, or a combination of software and hardware, which is either in a dedicated server, integrated into another entity, or distributed across multiple entities. The computer system 600 can include a desktop computer, a laptop computer, a tablet computer, a phablet, a smartphone, a feature phone, a server computer, or others. The server computer can be located remotely in a data center or stored locally.

[0059] The computer system 600 includes a bus 608 or other communication mechanism for transferring information, and a processor 602 (e.g., processor 112) coupled to the bus 608 for processing information. As an example, one or more processors 602 can be used to implement the computer system 600. The processor 602 can be a general-purpose microprocessor, a microcontroller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a state machine, gated logic, discrete hardware components, or any other suitable entity that can perform calculations or other information operations.

[0060] In addition to the hardware, the computer system 600 can also include code that creates an execution environment for the computer programs being discussed, e.g., code that constitutes the following stored in the included memory 604 (e.g., memory 120): processor firmware, protocol stack, database management system, operating system, or a combination of one or more of them. The memory can be, for example, a random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable PROM (EPROM), registers, a hard disk, a removable disk, a compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), or any other suitable storage device. The memory is coupled to the bus 608 for storing information and instructions to be executed by the processor 602. The processor 602 and the memory 604 can be supplemented by, or incorporated into, special-purpose logic circuitry.

[0061] Instructions can be stored in the memory 604 and can be implemented in one or more computer program products, such as one or more modules of multiple computer program instructions encoded on a computer-readable medium for execution by or to control the operation of the computer system 600. And according to any method well known to those skilled in the art, these computer program instructions include, but are not limited to, computer languages such as data-oriented languages (e.g., SQL, dBase), system languages (e.g., C, object-oriented programming languages that extend C (Objective-C), C++, assembly), structural languages (e.g., Java,.NET), and application languages (e.g., PHP, Ruby, Perl, Python). The instructions can also be implemented in the following computer languages: for example, array languages, aspect-oriented languages, assembly languages, authoring languages, command-line interface languages, compiled languages, concurrent languages, curly-bracket languages, dataflow languages, data-structuring languages, declarative languages, esoteric languages, extension languages, fourth-generation languages, functional languages, interactive-mode languages, interpreted languages, iterative languages, list-based languages, little languages, logic-based languages, machine languages, macro languages, meta-programming languages, multi-paradigm languages, numerical analysis, non-English-based languages, class-based object-oriented languages, prototype-based object-oriented languages, off-side rule languages, procedural languages, reflective languages, rule-based languages, scripting languages, stack-based languages, synchronous languages, syntax-handling languages, visual languages, Wirth languages, and XML-based languages. The memory 604 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processor 602.

[0062] As discussed herein, a computer program need not correspond to a file in a file system. The program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple cooperating files (e.g., files that store one or more modules, subroutines, or portions of code). The computer program can be deployed to execute on one computer or on multiple computers, which may be located at one site or distributed across multiple sites and interconnected by a communication network. The processes and logical flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output.

[0063] Computer system 600 also includes a data storage device 606, such as a magnetic disk or optical disk, coupled to bus 608 for storing information and instructions. Computer system 600 can be coupled to various devices via input / output module 610. Input / output module 610 can be any input / output module. Exemplary input / output module 610 includes a data port (e.g., a USB port). Input / output module 610 is configured to connect to communication module 612. Exemplary communication module 612 includes a network interface card, such as an Ethernet card and a modem. In some aspects, input / output module 610 is configured to connect to multiple devices, e.g., input device 614 and / or output device 616. Exemplary input device 614 includes a keyboard and a pointing device (e.g., a mouse or trackball) through which a user can provide input to computer system 600. Other kinds of input devices 614 can also be used to provide interaction with the user, such input devices being, for example, haptic input devices, visual input devices, audio input devices, or brain-computer interface devices. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback; and input from the user can be received in any form including voice input, speech input, haptic input, or brainwave input. Exemplary output device 616 includes a display device for displaying information to the user, such as a liquid crystal display (LCD) monitor.

[0064] In accordance with one aspect of the present disclosure, the smart glasses 100 can be at least partially implemented using a computer system 600 in response to one or more sequences of one or more instructions contained in a memory 604 being executed by a processor 602. Such instructions can be read into the memory 604 from another machine-readable medium (such as a data storage device 606). Execution of the instruction sequences contained in the main memory 604 causes the processor 602 to perform the process steps described herein. One or more processors in a multiprocessing arrangement can also be employed to execute the instruction sequences contained in the memory 604. In an alternative aspect, hardwired circuitry can be used in place of software instructions, or hardwired circuitry can be combined with software instructions to implement various aspects of the present disclosure. Accordingly, aspects of the present disclosure are not limited to any particular combination of hardware circuitry and software.

[0065] Aspects of the subject matter described in this specification can be implemented in a computing system that includes a back-end component (e.g., a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification); or aspects of the subject matter described in this specification can be implemented in any combination of one or more such back-end components, one or more such middleware components, or one or more such front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). A communication network (e.g., network 150) can include, for example, any one or more of the following: a local area network (LAN), a wide area network (WAN), and the Internet, among others. Additionally, the communication network can include, but is not limited to, for example, any one or more of the following network topologies, including a bus network, a star network, a ring network, a mesh network, a star-bus network, or a tree or hierarchical network, among others. For example, the communication module can be a modem or an Ethernet card.

[0066] The computer system 600 can include a client and a server. The client and the server are typically located remotely from each other and typically interact through a communication network. The relationship of the client and the server is generated by computer programs running on respective computers and having a client-server relationship with each other. The computer system 600 can be, for example, but not limited to, a desktop computer, a laptop computer, or a tablet computer. The computer system 600 can also be embedded in another device, such as, for example, but not limited to: a mobile phone, a personal digital assistant (PDA), a mobile audio player, a global positioning system (GPS) receiver, a video game console, and / or a television set-top box.

[0067] As used herein, the term "machine-readable storage medium" or "computer-readable medium" refers to any one or more media that participate in providing instructions to a processor 602 for execution. Such media can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks, such as data storage device 606. Volatile media includes dynamic memory, such as memory 604. Transmission media includes coaxial cables, copper wire, and fiber optics, including the wires that form bus 608. Common forms of machine-readable media include, for example, floppy disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, DVD, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH EPROM, any other memory chip or cartridge, or any other medium readable by a computer. A machine-readable storage medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of substances that affect a machine-readable propagated signal, or a combination of one or more of them.

[0068] In one aspect, a method can be an operation, an instruction, or a function, and vice versa. In one aspect, a claim can be modified to include some or all of the words (e.g., instructions, operations, functions, or components) stated in one or more other claims, one or more words, one or more sentences, one or more phrases, one or more paragraphs, and / or one or more claims.

[0069] To illustrate the interchangeability of hardware and software, items such as various illustrative blocks, modules, components, methods, operations, instructions, and algorithms have generally been described in terms of their functionality. Whether such functionality is implemented as hardware, software, or a combination of hardware and software depends on the particular application and the design constraints imposed on the overall system. Skilled artisans may implement the described functionality in different ways for each particular application.

[0070] As used herein, the phrase "at least one of" after a series of items, together with the terms "and" or "or" used to separate any one of these items, modifies the list as a whole, rather than modifying each element (e.g., each item) of the list. The phrase "at least one of" does not require the selection of at least one item; rather, the phrase means to include at least one of any one of these items, and / or at least one of any combination of these items, and / or at least one of each of these items. As an example, the phrases "at least one of A, B, and C" or "at least one of A, B, or C" both refer to: only A, only B, or only C; any combination of A, B, and C; and / or, at least one of each of A, B, and C.

[0071] The term "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment described herein as "exemplary" is not necessarily to be construed as more preferred or advantageous than other embodiments. Phrases such as on the one hand, the aspect, on the other hand, some aspects, one or more aspects, one implementation, the implementation, another implementation, some implementations, one or more implementations, one embodiment, the embodiment, another embodiment, some embodiments, one or more embodiments, one configuration, the configuration, another configuration, some configurations, one or more configurations, the present subject matter technology, the disclosure, this disclosure, and other variations thereof are for convenience only and do not imply that the disclosure associated with one or more such phrases is essential to the present subject matter technology, nor do they imply that such disclosure applies to all configurations of the present subject matter technology. The disclosure associated with one or more such phrases may apply to all configurations, or one or more configurations. The disclosure associated with one or more such phrases may provide one or more examples. Phrases such as on the one hand or some aspects may refer to one or more aspects, and vice versa, and the same applies to the other foregoing phrases.

[0072] Unless specifically stated otherwise, a reference to an element in the singular is not intended to mean "one and only one" but "one or more". Masculine pronouns (e.g., his) include feminine and neuter pronouns (e.g., her and its), and vice versa. The term "some" refers to one or more. Headings and subheadings that are underlined and / or italicized are used for convenience only, do not limit the present subject matter technology, and do not refer to the interpretation of the description of the present subject matter technology. Related terms such as first and second may be used to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between these entities or actions.

[0073] Although this specification contains many details, these details should not be construed as limiting the scope of what might be described, but rather as describing particular embodiments of the subject matter. Certain features that are described in this specification in the context of different embodiments can also be implemented in combination in a single embodiment. Conversely, the various features that are described in the context of a single embodiment can also be implemented separately in multiple embodiments or in any suitable sub-combination. Additionally, although features might have been described above as acting in certain combinations and even initially claimed as such, in some cases, one or more features from a claimed combination can be removed from the combination, and the claimed combination can be directed to a sub-combination or variation of a sub-combination.

[0074] The subject matter of this specification has been described in particular aspects, but other aspects can be implemented and are within the scope of the appended claims. For example, although the operations are depicted in the drawings in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in a sequential order, or that all of the illustrated operations be performed to achieve the desired result. The acts recited in the claims can be performed in a different order and still achieve the desired result. As one example, the processes depicted in the drawings do not necessarily need the particular order or sequential order shown to achieve the desired result. In some cases, multitasking and parallel processing might be advantageous. Additionally, the separation of the various system components in the aspects described above should not be understood as requiring such separation in all aspects, but rather that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0075] The name, background art, brief description of the drawings, abstract, and drawings are hereby incorporated into this disclosure and are provided as illustrative examples of the disclosure rather than as limiting descriptions. It is understood at the time of filing that they will not be used to limit the scope or meaning of the claims. Additionally, as can be seen in the detailed description, the description provides illustrative examples, and different features are combined in different embodiments to simplify the disclosure. The methods of this disclosure should not be construed as reflecting an intention that the subject matter described requires more features than are expressly recited in each claim. Rather, as reflected by the claims, the inventive subject matter lies in less than all of the features of a single disclosed configuration or operation. The claims are hereby incorporated into the detailed description, with each claim standing on its own as a separately described subject matter.

[0076] The claims are not intended to be limited to the aspects described herein, but rather should be accorded the full scope consistent with the language of the claims and cover all legal equivalents. Nevertheless, none of the claims is intended to encompass subject matter that fails to meet the requirements of applicable patent law, nor should they be construed in such a way.

Claims

1. A computer-implemented method, comprising: receiving a signal from a sensor on a face interface of a head-mounted device, the signal indicating movement of facial muscles of a user of the head-mounted device; determining, based on a machine learning algorithm, a facial expression of the user using the signal from the sensor in the face interface of the head-mounted device, the machine learning algorithm being trained to associate the facial expression with the movement of the facial muscles; adjusting an object avatar of the user of the head-mounted device based on the facial expression; and providing the object avatar to an immersive reality application hosted by a remote server.

2. The computer-implemented method according to claim 1, wherein receiving a signal from a sensor includes receiving a signal from one of the following: an inertial motion unit, an electronic sensor, a capacitive sensor, a contact microphone, an optical sensor, a tactile sensor, a humidity sensor, and a temperature sensor.

3. The computer-implemented method according to claim 1 or claim 2, wherein the signal is an electrical signal of neural activation of the facial muscles, and the computer-implemented method further includes identifying the facial muscles by the positions of the sensors on the face interface.

4. The computer-implemented method according to any one of the preceding claims, wherein receiving a signal from a sensor includes receiving multiple signals from multiple sensors disposed on the face interface of the head-mounted device, and the computer-implemented method further includes: correlating the multiple signals to evaluate whether and to what extent one or more facial muscles of the user of the head-mounted device move.

5. The computer-implemented method according to any one of the preceding claims, wherein adjusting the object avatar of the user includes: shaping a three-dimensional solid model of the face of the user of the head-mounted device based on the facial expression.

6. The computer-implemented method according to any one of the preceding claims, wherein the facial muscles are the muscles that move the eye pupils of the user of the head-mounted device, and determining the facial expression of the user includes determining the gaze direction of the user of the head-mounted device.

7. The computer-implemented method according to any one of the preceding claims, further comprising determining a user's physical condition based on the facial expression of the user.

8. The computer-implemented method according to any one of claims 1 to 6, further comprising determining a user's mental condition based on the facial expression of the user and the environment in the immersive reality application.

9. The computer-implemented method according to any one of the preceding claims, further comprising displaying a feedback message to the user in the head-mounted device based on the facial expression; and / or preferably, the computer-implemented method further includes modifying the facial expression of the user based on the audience in the immersive reality application.

10. A head-mounted device, comprising: A facial interface, the facial interface including one or more sensors and configured to contact the skin around two eyes and the nose in the face of a user of a head-mounted device, wherein the one or more sensors are geometrically arranged on the facial interface to identify movements of facial muscles of the user of the head-mounted device; A memory storing instructions and a chart, the chart including a mapping from facial expressions of the user of the head-mounted device to movements of facial muscles; A processor configured to: at least partially receive a plurality of signals from the one or more sensors and identify a facial pose of the user of the head-mounted device based on the plurality of signals and the chart; A communication module configured to send the plurality of signals and the facial pose to a remote server, the remote server hosting an immersive reality application including an avatar of the user of the head-mounted device.

11. The head-mounted device according to claim 10, wherein, the one or more sensors include at least one of the following: an inertial motion sensor, an electronic sensor, a capacitance sensor, a contact microphone, an optical sensor, a tactile sensor, a humidity sensor, and a temperature sensor; and / or preferably, wherein the one or more sensors include a photoplethysmography sensor for determining cardiovascular activity across the nose of the user of the head-mounted device.

12. The head-mounted device according to claim 10 or claim 11, wherein, the one or more sensors include a plurality of electronic sensors symmetrically arranged around two eyes of the user of the head-mounted device and configured to evaluate the gaze direction of the user of the head-mounted device; and / or preferably, wherein a part of the facial interface including the one or more sensors is detachable from the head-mounted device.

13. A computer-implemented method, comprising: providing, from a remote server, an object avatar of a first participant to one or more head-mounted devices used by a plurality of participants in an immersive reality application; receiving a signal from a first head-mounted device of the first participant indicating a movement of facial muscles of the first participant; determining a facial pose of the first participant based on the signal; and updating the object avatar of the first participant with the facial pose.

14. The computer-implemented method according to claim 13, wherein, Receiving a signal indicative of movement of the facial muscles of the first participant includes receiving a signal from at least one of the following: an inertial motion sensor, a contact microphone, an electronic sensor, a capacitance sensor, an optical sensor, a tactile sensor, a humidity sensor, and a temperature sensor; and / or preferably, wherein receiving the signal indicative of movement of the facial muscles includes: integrating a plurality of signals from one or more sensors in the facial interface of the first head-mounted device; and / or preferably, the computer-implemented method further includes: receiving a signal from a head-mounted device of a second participant indicative of movement of the facial muscles of the second participant; determining a facial pose of the first participant based on the signal; and updating an object avatar of the second participant with the facial pose.

15. The computer-implemented method according to claim 13 or claim 14, wherein, determining the facial pose of the first participant includes consulting a chart that correlates movement of the facial muscles with the facial pose.

Images