Maintaining user representation appearance during device removal

Abstract

The present disclosure relates to maintaining a user representation appearance during device removal. Various implementations disclosed include devices, systems, and methods that use sensor data to identify scenarios where a user representation should be configured to account for removal or otherwise removal of a head-mounted device (HMD) from its normal use location, such as, for example, a head-mounted device (HMD). It is ensured that when the device is removed or otherwise removed from its normal use location, the user representation is not deformed or otherwise changed in an unnatural / undesired manner based on the image or other sensor data of the user's eyes and / or other facial portions being not captured from the intended capture location. As an example, this may involve using motion sensor data to detect changes in position and / or orientation of the device, and / or using image sensor data to determine when a user's eye (whether open or closed) is present within an eye-fit region of the device.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to electronic devices, and in particular to systems, methods, and devices for representing a user appearance based on image and other sensor data. BACKGROUND

[0002] Prior art can not adequately provide for avatars or other user representations that represent a user appearance of an electronic device in various situations. For example, a user representation can be presented in situations in which sensor data on which the user representation is based is interrupted by user activity, such as by the user taking off or otherwise removing a wearable device having sensors that provide such sensor data. SUMMARY

[0003] Various implementations disclosed herein include devices, systems, and methods that use sensor data to identify situations in which a user representation should be configured to account for a head-mounted device (HMD) being taken off from its normal use positioning or otherwise removed, e.g., to ensure that the user representation is not distorted or otherwise altered in a non-natural / non-desired manner based on images or other sensor data of the user’s eyes and / or other facial parts not being captured from an expected capture positioning when the device is taken off from its normal use positioning or otherwise removed. As an example, this can involve using motion sensor data to detect a device change in positioning and / or orientation, and / or using image sensor data to determine when the user’s eyes, whether open or closed, are present within the eyebox of the device and / or determine an abnormal positioning of one or both eyes relative to the device, e.g., determine that both eyes present in the sensor data have moved downward at the same time (which situation can occur, for example, when the device starts to move upward relative to the user’s face). For example, such a determination can be made promptly once such a lack of eye presence or abnormal positioning occurs, e.g., just as the user starts to move the device from its normal positioning (e.g., move the device upward), and appropriate mitigation is promptly performed. In such scenarios, the appearance of the user representation can be configured (e.g., altered), e.g., by freezing the current appearance of the user representation (e.g., the head of the user avatar) and / or displaying a fixed user representation appearance (e.g., displaying a neutral avatar appearance based on a pre-session / enrollment neutral avatar appearance).

[0004] In general, one innovative aspect of the subject matter described in this specification can be embodied in methods performed by a processor (e.g., on an HMD) executing instructions embodied in a non-transitory computer-readable medium. The method can involve obtaining, via one or more sensors, sensor data during a time period. The method can involve determining, based on the sensor data, a removal (e.g., a doffing or a taking off) of the HMD during the time period. The removal of the HMD corresponds to a change in the HMD from a first position in which one or more displays of the HMD are positioned in front of a user's eyes (e.g., a normal use position) to a second position in which the one or more displays are positioned in other locations relative to the user (e.g., a doffed or taken off position). The removal can include a change in the positioning of the device from its normal use position in front of the user's eyes to a taken off or doffed position. The method also involves generating, based on the sensor data, user representation data corresponding to a three-dimensional (3D) appearance of the user during the time period, where the user representation data is generated based on determining the removal of the HMD during the time period. A view of the user representation can be provided based on the user representation data, e.g., the view showing a current appearance of the user at a point in time during the time period prior to the removal, followed by showing an adjusted (e.g., neutral) appearance of the user at a point in time during the time period after the removal, based on current sensor data.

[0005] According to some implementations, a non-transitory computer-readable storage medium stores instructions that are computer-executable to perform or cause performance of any of the methods described herein. According to some implementations, a device includes one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing performance of any of the methods described herein. BRIEF DESCRIPTION OF DRAWINGS

[0006] For those skilled in the art, in order to make the present disclosure more understandable, a more detailed description will be made with reference to some aspects of the example implementations, some of which are shown in the accompanying drawings.

[0007] Figure 1 Devices that obtain sensor data from a user are illustrated in accordance with some implementations.

[0008] Figure 2 Example electronic devices that operate in different physical environments during a communication session are illustrated in accordance with some implementations.

[0009] Figures 3A-3B A portion of a user's face during an example time period in which the user removes a device is illustrated in accordance with some implementations.

[0010] Figures 4A-4B Examples are given for generating times within that time period based on some specific implementations. Figures 3A-3B The user's facial expression.

[0011] Figure 5 The graph depicts, based on some specific implementation examples, curves representing changes in orientation / positioning used to determine whether a user is performing a disengagement movement under different eye continuity conditions.

[0012] Figure 6 It is a flowchart representation of a method for maintaining the user's presentation appearance during device removal, based on some specific implementations.

[0013] Figure 7 This is a block diagram illustrating device components according to some specific implementations of exemplary devices.

[0014] Figure 8 It is a block diagram based on some specific implementation examples of head-mounted devices (HMDs).

[0015] As is customary practice, various features illustrated in the accompanying drawings may not be drawn to scale. Therefore, for clarity, the dimensions of various features may be arbitrarily expanded or reduced. Furthermore, some drawings may not depict all components of a given system, method, or apparatus. Finally, similar reference numerals may be used throughout the specification and drawings to denote similar features. Detailed Implementation

[0016] Numerous details have been described to provide a thorough understanding of the exemplary embodiments illustrated in the accompanying drawings. However, the drawings illustrate only some exemplary aspects of this disclosure and should not be considered limiting. Those skilled in the art will recognize that other effective aspects or variations do not include all the specific details described herein. Furthermore, well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure further relevant aspects of the exemplary embodiments described herein.

[0017] Figure 1 An example environment 100 is illustrated, in which an exemplary electronic device 105 operates within a physical environment 102. In some embodiments, the electronic device 105 may be able to share information with another device or with an intermediary device (such as an information system). Additionally, the physical environment 102 includes a user 110 wearing the device 105. In some embodiments, the device 105 is configured to present views of extended reality (XR) environments, which may be based on the physical environment 102 and / or include added content (such as virtual elements).

[0018] exist Figure 1In the example, physical environment 102 is a room that includes physical objects such as wall hangings 120, plants 125, and tables 130. Electronic device 105 may include one or more cameras, microphones, depth sensors, motion sensors, or other sensors that can be used to capture information about physical environment 102 and the objects within it and to evaluate the physical environment and these objects, as well as to capture information about user 110.

[0019] exist Figure 1 In this example, device 105 includes one or more sensors 116 (e.g., inward-facing sensors and / or outward-facing cameras) that capture light intensity images, depth sensor images, audio data, or other information about user 110. For example, one or more sensors 116 may capture images of the user's (e.g., user 110's) forehead, eyebrows, eyes, eyelids, cheeks, nose, lips, chin, face, head, hands, wrists, arms, shoulders, torso, legs, or other body parts. For example, an inward-facing sensor may see the interior of device 105 (e.g., the user's eyes and the area around them), and other external cameras may capture the user's face outside device 105 (e.g., an egocentric camera pointing outwards from device 105). As an example, sensor data about the user's eyes 111 may indicate various user characteristics, such as the user's gaze direction 119 over time, the user's saccade behavior over time, the user's eye expansion behavior over time, etc. One or more sensors 116 may capture audio information, including the user's voice and other sounds emitted by the user, as well as sounds within the physical environment 100.

[0020] In some embodiments, device 105 includes an eye-tracking system for detecting eye positioning and eye movement via eye gaze characteristic data. For example, the eye-tracking system may include one or more infrared (IR) light-emitting diodes (LEDs), an eye-tracking camera (e.g., a near-IR (NIR) camera), and an illumination source (e.g., an NIR light source) that emits light (e.g., NIR light) toward the user 110's eye. Furthermore, the illumination source of device 105 may emit NIR light to illuminate the user 110's eye, and the NIR camera may capture images of the user 110's eye. In some embodiments, the images captured by the eye-tracking system may be analyzed to detect the positioning and movement of the user 110's eye or to detect other information about the eye, such as color, shape, state (e.g., widening, strabismus, etc.), pupil dilation, or pupil diameter. Furthermore, the gaze point estimated from the eye-tracking images enables gaze-based interaction with content displayed on the near-eye display of device 105.

[0021] Additionally, one or more sensors 116 may capture images of the physical environment 100 (e.g., externally oriented sensors). For example, one or more sensors 116 may capture images of the physical environment 100 including physical objects such as wall hangings 120, plants 125, and tables 130. Furthermore, one or more sensors 116 may capture images (e.g., light intensity images and / or depth data).

[0022] One or more sensors (such as one or more sensors 115 on device 105) may identify user information based on proximity or contact with a portion of user 110. For example, one or more sensors 115 may capture sensor data that can provide biometric information related to the user’s cardiovascular status (e.g., pulse), body temperature, respiratory rate, etc.

[0023] One or more sensors 116 or one or more sensors 115 can capture data that can determine a user orientation 121 within the physical environment. In this example, user orientation 121 corresponds to the direction in which the user 110's torso is facing.

[0024] Some specific implementations disclosed herein determine user understanding or contextual understanding based on sensor data obtained from user wearable devices (such as the first device 105). This user understanding can indicate the user's state in relation to providing user assistance or facilitating communication sessions.

[0025] The content can be visible (e.g., displayed on a display of device 105) or audible (e.g., generated as audio 118 by a speaker of device 105). In the case of audio content, audio 118 can be generated in a manner that makes it likely only user 110 will hear it (e.g., via a speaker close to user's ear 112 or at a volume below a threshold), making it unlikely that nearby people will hear it. In some specific implementations, the audio pattern (e.g., volume) is determined based on whether other people are within a threshold distance or based on how close other people are relative to user 110.

[0026] In some implementations, the content provided by device 105 and the sensor features of device 105 may be provided using components, sensors, or software modules that are small enough and efficient in terms of power consumption and use to be adapted to and otherwise used in lightweight, battery-powered wearable products, such as wireless earbuds or other ear-hook devices or head-mounted devices (HMDs), such as smart / augmented reality (AR) glasses. A combination of multiple devices may be used to facilitate the features. For example, a smartphone (wirelessly connected and interacting with wearable devices) may provide computing resources, connectivity to cloud or internet services, location services, etc.

[0027] Device 105 can generate a user facial representation of user 110 based on image and / or other sensor data for various purposes. For example, user 110 can use device 105 with image sensors (e.g., a head-mounted device (HMD)) that capture images of portions of the user's face (e.g., images of the user's eyes captured via a camera inside the HMD and / or images of the user's cheeks, nose, and mouth captured via a downward-facing camera on the HMD). The stream of image and / or other sensor data can be acquired over time and used to animate the user's facial representation, for example, to provide a user avatar representing the user's face as the user forms facial expressions and otherwise moves their face over time.

[0028] The user can combine real-time data and previous data about the user. For example, real-time sensor data representing the current appearance of parts of the user's face (e.g., images of the user's eyes via a camera inside the HMD and / or images of the user's cheeks, nose, and mouth via a downward-facing camera on the HMD) can be combined with previous data representing the face at one or more previous times (e.g., registered data representing the face without the HMD in one or more expressions (e.g., neutral expression, smiling expression, etc.)).

[0029] User facial representation data can be 3D or otherwise uses information about the 3D appearance of a user's face. In some implementations, current sensor data corresponding to the user's current / real-time facial appearance (e.g., current images from inward-facing and downward-facing sensors) is combined with information about the 3D shape of the user's face to provide a user facial representation. User facial representations can be used for a variety of purposes, including, but not limited to, providing a representation of the user that is presented to one or more other users during a communication session.

[0030] Figure 2 Exemplary electronic devices operating in different physical environments during a communication session involving a first user at a first device and a second user at a second device, and a 3D representation of the second user relative to the first device, are illustrated. Specifically, Figure 2 Exemplary operating environments 200 are illustrated for electronic devices 210 and 265 operating in different physical environments 202 and 250 during a communication session (e.g., when electronic devices 210 and 265 share information with each other or with an intermediate device (such as a communication session system / server)). Figure 2 In this example, physical environment 202 is a room that includes wall mount 212, plant 214, and table 216 (e.g., Figure 1The physical environment 202. Electronic device 210 includes one or more cameras, microphones, depth sensors, or other sensors that can be used to capture information about the physical environment 202 and objects within it, as well as information about the user 225 of electronic device 210 (e.g., a handheld device), and to evaluate the physical environment and these objects. Information about the physical environment 202 and / or the user 225 can be used to provide visual content (e.g., for user representation) and audio content (e.g., for audible speech or text transcription) during a communication session. For example, a communication session may provide one or more participants in a 3D environment (e.g., user 225, user 260) with views generated from camera images and / or depth camera images of the physical environment 202, and a representation of user 225.

[0031] Additionally, in Figure 2 In this example, physical environment 250 is a room including wall mount 252, sofa 254, and coffee table 256. Electronic device 265 includes one or more cameras, microphones, depth sensors, or other sensors that can be used to capture information about physical environment 250 and objects within it, as well as information about user 260 (e.g., user wearable device or HMD device, such as device 105) and to evaluate the physical environment and these objects. Information about physical environment 250 and / or user 260 can be used to provide visual and audio content during a communication session. For example, the communication session can provide a view of a 3D environment generated based on camera images and / or depth camera images (from electronic device 265) of physical environment 250, and a representation of user 260 based on camera images and / or depth camera images (from electronic device 265). For example, in communication with device 265 via communication session command set 282 (e.g., through information system 290 via network connection 285), the 3D environment can be transmitted by device 210 via communication session command set 280.

[0032] Information system 290 can coordinate the sharing of assets (e.g., data associated with user representation 240, user representation 275) between two or more devices (e.g., electronic device 210 and electronic device 265).

[0033] Figure 2An example of a view 205 provided at device 210 is illustrated, wherein a representation 232 of wall mount 252 and a user representation 240 (e.g., a portrait of user 260) are provided, provided that the user representation of each user is viewed during a particular communication session. Specifically, the user representation 240 of user 260 is generated based on one or more user representation techniques. The generation of user representations is further discussed herein. Additionally, an electronic device 265 within physical environment 250 provides a view 266 that allows user 260 to view the representation 272 of wall mount 212 and a representation 275 (e.g., a portrait) of at least a portion of user 225 (e.g., from mid-torso upwards) within a 3D environment 270. The user representation 240 of user 260 can be generated at device 210 (e.g., a receiving / viewing device) by generating representations of user 260 for multiple moments within a time period based on data obtained from device 265 (e.g., a frame-specific 3D representation of user 260). Alternatively, in some embodiments, a user representation 240 of user 260 is generated at device 265 (e.g., a transmitting device) and the user representation is sent to device 210 (e.g., a receiving / viewing device to view a portrait of the sender). In some embodiments, each of the 3D representation 240 of user 260 and the 3D representation 275 of user 225 is generated by generating splats corresponding to the user representation data.

[0034] exist Figure 2 In the example, electronic devices 210 and 265 are exemplified as head-mounted devices (HMDs). However, either electronic device 210 or electronic device 265 can be a mobile phone, tablet, laptop, or any other form of wearable device (e.g., head-mounted devices (glasses), headphones, ear-worn devices, etc.). In some implementations, the functionality of each of devices 210 and 265 is implemented via two or more devices, such as a mobile device and a base station, or a head-mounted device and an ear-worn device. Various functions can be distributed across multiple devices, including but not limited to power functions, CPU functions, GPU functions, storage functions, memory functions, visual content display functions, audio content production functions, etc. The multiple devices used to implement the functions of electronic devices 210 and 265 can communicate with each other via wired or wireless communication. In some implementations, each device communicates with a separate controller or server to manage and coordinate the user experience (e.g., a communication session server). Such a controller or server may be located within physical environment 202 and / or physical environment 250 or may be remote relative to that physical environment.

[0035] Additionally, in Figure 2In the example, 3D environments 230 and 270 can be based on a common coordinate system that can be shared with other users (e.g., to provide a virtual room for a portrait used in a multi-user communication session). In other words, a common coordinate system is available for 3D environments 230 and 270. A common reference point can be used to align the coordinate system. In some implementations, the common reference point can be a virtual object within the 3D environment that each user can visualize within their respective view. For example, a common center piece table around which a user representation (e.g., a portrait of a user) is positioned within the 3D environment. Alternatively, the common reference point may not be visible within each view. For example, the common coordinate system of the 3D environment can use the common reference point to position each respective user representation (e.g., around a table / desk). Therefore, if the common reference point is visible, each view of the device will be able to visualize the “center” of the 3D environment for perspective when viewing other user representations. The visualization of the common reference point can become more relevant to the multi-user communication session, allowing each user’s view to add perspective to the position of each other user during the communication session.

[0036] In some implementations, each user's representation can be real or unreal and / or represent the user's current and / or previous appearance. For example, a photorealistic representation of user 225 or user 260 can be generated based on a combination of real-time images and previous images. Previous images can be used to generate the representation for portions of the real-time image data that are not available to them (e.g., portions of the user's face that are not in the field of view of the camera or sensors of electronic device 210 or electronic device 265 or may be occluded). In one example, electronic devices 210 and 265 are HMDs, and the real-time image data of the user's face includes images of the user's cheeks and mouth from a downward-facing camera and images of the user's eyes from an inward-facing camera. This real-time image data can be combined with previous image data of other parts of the user's face, head, and torso that are not currently observable from the device's sensors. Previous data about the user's appearance may be obtained earlier during a communication session, during previous use of the electronic device, during a registration process for obtaining sensor data about the user's appearance from multiple viewpoints and / or conditions, or otherwise.

[0037] In some specific implementations, such as Figure 2 The views illustrating the generation of one or more user representations for a communication session (e.g., generating user representation 240, user representation 275) can be based on one or more rendering techniques, such as using 3D meshes, 3D point clouds, or 3D Gaussian scatter rendering methods. Such 3D Gaussian scatter rendering methods can use UV mapping and generate proxy mesh representations.

[0038] Some specific implementations disclosed in this document involve maintaining the user presentation appearance during device removal (e.g., when the HMD is detached or removed). Figure 2 As illustrated, in a multi-user XR environment, device users can view each other's representations based on their respective actual appearances. For example, an avatar used by a user can be based on previous and / or current images and / or other sensor data captured about the user's appearance. In one example, a user avatar is based on a combination of images captured of the user before and during a shared XR session, such as images of the user's entire face (i.e., without an HMD) captured during the registration period or otherwise before the XR session (e.g., via an internal camera capturing the current / real-time appearance of the user's eye area, a downward-facing camera capturing the current / real-time appearance of the user's cheeks and mouth, an outward-facing camera capturing the current / real-time appearance of the user's hands, etc.) and images of portions of the user's face captured during the session.

[0039] Existing devices and methods may not adequately account for users removing (e.g., taking off or removing) wearable devices (such as HMDs) during user representation generation (e.g., when using an avatar during a multi-user XR session). Existing systems may not adequately identify situations where the appearance of the user avatar should be altered to account for removal (e.g., by lifting the HMD from the eye area to rest on the forehead or by completely removing the HMD from the head), and / or in such scenarios, the appearance of the user avatar may not be altered in the desired manner.

[0040] Some specific implementations disclosed herein utilize sensor data to identify situations where the appearance of a user representation (e.g., an avatar) should be changed to account for HMD removal (e.g., removal or detachment). For example, this could involve using motion sensor data to detect device changes in positioning and / or orientation, and / or using eye sensor data to determine when the user's eyes (whether open or closed) are within the device's eye-friendly area. Some implementations change the appearance of the user avatar in a desired manner in such scenarios, for example, by freezing the current appearance and / or positioning of the user avatar (e.g., the avatar's head) and / or displaying a fixed avatar appearance (e.g., displaying a neutral avatar appearance based on a pre-session / registration neutral avatar appearance).

[0041] Figures 3A-3B This example illustrates a portion of a user's face during a specific moment in a time period when the user removes the device (e.g., removing the HMD in this example). During this time period, a representation of the user's face will be generated based on sensor data. Figure 3AAs shown in the diagram, at the first moment, the user is wearing device 105 in its normal use positioning (relative to the user's face), and the user's face can be approached to be captured via one or more sensors (e.g., via one or more outward / downward facing sensors on device 105). After the first moment, in (as shown in the diagram) Figure 3B At the second moment (as shown), the user has moved device 105 from its initial normal use position to another different position (resting on the user's forehead, rather than in the normal use position in front of the user's eyes). In this second position, views from one or more sensors (e.g., one or more outward / downward facing sensors on device 105) will not capture the appropriate portion of the user's face for generating a user representation, or will capture images from unintended viewpoints on the user's face. If such sensor data is used to continue generating the current appearance of the user representation, this appearance may appear distorted or otherwise undesirable.

[0042] When the device is Figures 3A-3B When the device transitions between positions as depicted herein, sensor data may reflect unintended movements of the user's face (e.g., eyes moving downwards / sliding as the detachment begins). In this case, device 105 can interpret the detachment movement as actual head movement and generate a corresponding movement representation. Device 105 can be configured to rapidly detect and preemptively block such unintended consequences from such movements / changes. For example, the detection process used by device 105 can be configured to detect slight / minor movements and / or respond rapidly when such movements are initiated / first begin, enabling device 105 to quickly and accurately detect the occurrence of these movements and implement the mitigation techniques discussed herein. For example, any simultaneous abnormal movement of the user's eyes sensed by the sensors can be rapidly detected, and appropriate mitigation can be implemented immediately.

[0043] Figures 4A-4B Examples are given for generating times within that time period. Figures 3A-3B The representation of the user's face. Specifically, for ( Figure 3A (As illustrated in the example) In the first moment, data is generated based on real-time sensor data that captures the user's face (e.g., via one or more outward / downward facing sensors on device 105). Figure 4AUser representation 410 (exemplified in the example). User representation 410 depicts the real-time / current appearance of the user's face because sensor data from the normal expected positioning relative to the user's face can be obtained at this first moment. User representation 410 may provide this appearance using only real-time sensor data or a combination of real-time captured sensor data and previously captured sensor data (e.g., data from previously registered user accounts). This user representation shows the user's real-time / current appearance (e.g., if the user is currently smiling, user representation 410 would depict the user's mouth area as smiling, etc.). In this example user representation 410, the appearance of the face is depicted based on a combination of current sensor data (e.g., an inward-facing camera capturing the real-time / current state of the user's eye area and a downward-facing image camera capturing the real-time / current appearance of the user's lower face) and previously registered data (e.g., information about the appearance of the user's eyes captured in sensor data when the user is not wearing the device (such as color), information about parts of the user's face obscured by wearing an HMD, etc.).

[0044] for( Figure 3B (Example shown) the second moment, ( Figure 4B The user representation 420 (illustrated in the example) depicts at least a portion of the user's face in a non-real-time / non-current appearance because such a portion of the face is not adequately captured in the sensor data at this second moment (e.g., the camera's changing position relative to the user's face does not provide a proper view of the user's face for the purpose of generating an accurate user representation). Instead, information about the user's face in a previous appearance is used. Figure 4B The user described in the text stated that 420 maintained their source Figure 4A The user representation of 410 is depicted in the image. In this example, when the sensor is in (with) Figure 3A Correspondingly, at the first moment, when the user is in a position relative to normal user use and appropriate sensor data is available, a user representation 420 is generated based on the sensor data captured at this moment. For example, the user representation can maintain the appearance of the user representation from the previous first moment during the second moment. In this example user representation 420, based on sensor data from the first moment (e.g., captured...),... Figure 3A The user's appearance (from an inward-facing camera) and previously registered data (such as information about the appearance of the user's eyes captured in sensor data when the user is not wearing the device, such as color) can be combined to depict the eye region of the face. Therefore, user representation can combine information from user registration (such as information about the user's eye color) with other previously registered information about the user's face from a recent moment (such as information about the user's lower face and / or eyes, which is not currently available in the current / real-time sensor data but can be obtained from a previous recent moment (e.g., at a previous time). Figure 3AThe data obtained from the most recently acquired sensor data at the first moment is combined.

[0045] Some implementations utilize one or more algorithms or machine learning (ML) models to maintain the stability of the user avatar when the user removes the HMD (e.g., lifts the HMD to the user's forehead or removes the HMD completely). During such removal, for example, based on the fact that images of the user's eyes and / or other facial features were not captured from the intended capture location when the device was removed (e.g., detached or completely removed), the avatar's appearance may be distorted or otherwise altered in an unnatural / unintended manner.

[0046] Some implementations use sensor data corresponding to the user's eyes to determine that removal (e.g., detachment or removal) is occurring. The HMD device can be configured to track the user's eyes via one or more other systems for one or more other purposes (e.g., for gaze-based input, user identification, determining when the device is in use, locking the device in certain situations, etc.). Some implementations determine / track whether the user's eyes are visible. Some implementations can determine / track / distinguish between the time when the user's eyes are invisible due to eyelid closure and the time when the user's eyes are invisible because the head-mounted device has been removed and therefore the eyes are no longer in their intended position relative to the device. In some implementations, the HMD can detect unintended movement, such as any simultaneous downward movement of the eyes (e.g., that may occur when the device moves upward relative to the user's face). The system can track eye information in a way that distinguishes between closed eyes and the absence or unintended movement of the closed eyes, such that regardless of whether the eyes are open or closed, the system can determine whether the user's eyes are present or absent from the intended position (e.g., the normal use position relative to the HMD) / abnormal to that intended position. This information can be used to determine that removal (e.g., detachment or removal) is occurring. In some implementations, timely detection of device removal is facilitated by detecting abnormal positioning of one or both eyes relative to the device. The HMD can be activated precisely when the user begins to move the device from its normal position (e.g., ...). Figure 3A The device is detected when it is moved (e.g., the device is moved upwards) as depicted in the diagram, and mitigation is performed quickly accordingly.

[0047] In one example, a triggering event (such as detecting that the eyes are not visible to the eye image sensor) triggers the use of an eye-tracking continuity system. This triggering event can occur at a higher-than-normal frequency (e.g., 30 Hz to 45 Hz) to efficiently and rapidly detect when a user lifts the device relative to their face, such as lifting it to their forehead or away from their head. Triggering eye continuity only in certain situations where removal (e.g., taking it off or removing it) might begin can improve the system's efficiency, responsiveness, and / or accuracy.

[0048] In some implementations, the eye-tracking continuity system, intended for another purpose, is additionally used for removal (e.g., detachment or removal) detection. In other implementations, information (e.g., regarding eye continuity) is not available via another system, and the device specifically (e.g., only) uses the eye-tracking continuity system to detect removal (e.g., detachment or removal).

[0049] In some implementations, information from other sensors or systems is additionally or alternatively used to identify removal (e.g., detachment or removal). As an example, motion sensor data can be used to detect HMD orientation and / or positioning changes, comparing these changes to one or more thresholds to detect removal. In some implementations, three input signals are used (e.g., orientation change, positioning change, and loss of eye continuity).

[0050] In some implementations, relative curve assessments (e.g., regarding changes in orientation, positioning, and loss of eye continuity) are used to identify when removal (e.g., detachment or removal) is taking place. Figure 5 A graph depicts, based on some specific implementations, the use of orientation / positional change information to determine whether a user is performing a disengagement movement under different eye continuity conditions. One or more thresholds can be used to determine when the device has been removed. For example, a device removal event can be identified if / when the device's positional change exceeds threshold A. As another example, after loss of eye continuity, a device removal event can be identified if / when the device's orientation change exceeds threshold B.

[0051] How input signal information is used to identify device removal events (e.g., for associated thresholds to identify orientation and / or positioning changes used to identify removal) can depend on how long (e.g., how many frames) of eye continuity data has been lost. For example, one or more curves can be used to specify how orientation information (e.g., from a gyroscope) and / or positioning change information (e.g., from an accelerometer) will be used to determine whether a user is performing a disengagement motion under different eye continuity scenarios (e.g., shorter or longer periods). Figure 5 Curve 510 shown in the example is an example. Figure 5In this curve, the x-axis represents the average rotation angle, such as the average of three past rotation angles of the device, and the y-axis represents eye continuity data, such as the number of frames / time periods during which the eye (or both eyes) is not detected in its expected position. This curve 510 is based on the expectation that if there will be a large / rapid detachment movement (e.g., a large, rapid change in orientation or positioning), less eye continuity (e.g., fewer frames during which the eye is not detected in its expected position) is needed to determine that detachment is occurring. Using such a threshold adjustment / curve can prevent false negatives (e.g., detecting detachment when the user only adjusts the HMD on their face) while ensuring rapid detection of detachment in cases where the user rapidly detaches the HMD. It should be noted that eye movement of both eyes can be considered to determine whether detachment or other removal is occurring, for example, to distinguish between the device moving upwards and away from the user's eye area and the device tilting as a bend or temporary deflection, but generally still in front of the user's eyes.

[0052] Gyroscopes can be used to provide orientation (e.g., roll, pitch, yaw) information, for example, to detect orientation changes greater than a threshold. Accelerometers can be used to provide changes in position, velocity, and / or acceleration in 3D space, for example, relative to one or more specific axes (such as the y-axis), for example, to detect changes in position, velocity, and / or acceleration relative to one or more thresholds.

[0053] In some implementations, information about both eyes (e.g., stereoscopic) is used to detect removal (e.g., taking them off or removing them).

[0054] Once removal is detected, some implementations change the appearance of the user avatar in a desired manner. In some implementations, this involves using the avatar's location and / or appearance from a previous moment before the removal began (e.g., using the avatar's state from 5 frames ago, 150ms ago, etc.). In some implementations, this involves "freezing" the current location / skeleton of the user representation (e.g., the avatar's head) and / or displaying a fixed user representation appearance (e.g., displaying a neutral avatar appearance based on the pre-session / registration neutral avatar appearance). This can provide a natural-looking face while also indicating to other users who may be viewing the avatar that the user associated with the user representation has removed or is not wearing an HMD.

[0055] Once a user repositions the device in front of their open eyes (e.g., moving the device from a removal to a normal use location), the eye-tracking system on the device can detect that the open eyes are in their intended location, and this can trigger the end of changes to the user's appearance, such as the avatar appearance being based again on the user's current / real-time appearance.

[0056] Some implementations utilize information from previous user registrations when providing a current / real-time or non-current / non-real-time user representation appearance. During such registration, the system (e.g., HMD) may have captured images or other sensor data corresponding to the user's face in one or more specific configurations (e.g., smiling, frowning, neutral, mouth closed, expressionless, etc.). This information can be used for later periods when the user representation requires the current, last captured, or otherwise most recent user sensor data. Recently captured sensor information can be saved (e.g., stored in memory for a limited time period) for use in generating such a user representation, for example, by saving camera or other sensor data from one or more previous moments during the current time.

[0057] In situations where the user representation does not correspond to the user's current appearance (e.g., when the user representation is configured to take into account the removal of the device), one or more effects can be applied to provide a better viewing experience. For example, processing (e.g., blurring, feathering, etc.) can be applied to indicate to the viewer that the user representation does not necessarily correspond to the user's current appearance. In some implementations, the change to the user representation upon device removal is gradually implemented, for example, gradually changing the user representation to a neutral expression over a relatively short period of time. This can convey to the observer that the user's face is not frozen / stuck in a previous position and / or avoids continuing to display the representation in an unnatural or otherwise undesirable frozen posture (e.g., a frozen posture that appears to be accompanied by a wide-open mouth). In some examples, this involves (e.g., over a period of time) gradually altering or grading the appearance of the obscured portion of the user's face to a different expression, such as neutral / expressionless or other predetermined expression. For example, the user's face may initially be displayed in its previous posture, then gradually altered / graded back to a neutral mouth expression using registration data. This helps ensure that if a user is doing something unusual with their mouth, they won't simply be stuck on that unusual (e.g., funny / frozen-looking) mouth expression for an extended period of time after the device is removed.

[0058] Once the device returns to its normal operating position, the device (e.g., HMD) can, for example, use real-time sensor data of the user's face to blend back a pre-determined (e.g., neutral) animated view. In an alternative embodiment, once the device returns to its normal operating position, such as in the case of a brief removal of the device, the device blends back a real-time animated view from a representation based on the previous expression (or from a current blend of the previous expression and a pre-determined expression).

[0059] In some implementations, one or more visual processing techniques are applied to a portion of the user's face during a period not based on real-time current sensor data, such as during the removal (e.g., taking off or removing) of a device. This processing can blur, add (e.g., add a light blue glow), or otherwise modify the appearance. Additionally (or alternatively), such visual processing can convey to the observer that what the observer is seeing may not be the user's actual mouth; for example, the content may not depict the user's actual current facial expression. Visual effects can convey uncertainty or another measure of inaccuracy. The amount or other properties of the visual effect may depend on the amount of time that has elapsed since the device was removed, for example, by increasing the amount and / or magnitude of blurring and / or glow effects based on the amount of time elapsed.

[0060] In some implementations, the device (e.g., HMD) is configured to predict that the device will be removed, for example, based on: contextual understanding of what the user is doing, the user's hand positioning on the side of the device, and understanding of what the user is saying (e.g., the user says "I'm going to remove my device in a minute") and / or other contextual information about the user, other users, or the environment. Visual processing can be applied based on this prediction. In some implementations, during a period prior to the device's removal, the device acquires information that will be used once the device is removed (e.g., additional current / real-time sensor data, previously generated neutral facial expression data, etc.).

[0061] In some implementations, the representation of a user's face includes or is otherwise based on Gaussian scatter points, such as via a Gaussian scatter-based 3D representation. In such implementations, the blending of facial expressions over time can be considered based on scatter points. Scatter-based blending can appear very realistic and thus unintentionally convey to the observer that the user's face has an expression that is not the user's true current expression. Therefore, visual processing or other procedures can be performed to intentionally convey that the user's face may have different expressions. For example, instead of smoothly blending between the user's previous facial expressions and a predetermined / neutral expression, the transition can be intentionally speckled in a way that feels "smooth" but does not resemble natural human movement, using a classic film dissolving effect to modify the transition. For example, a film dissolving effect is smooth, but an external viewer can easily discern that the film dissolving effect is an artificial transition effect, rather than the user actually closing their mouth.

[0062] Figure 6 This is a flowchart illustrating exemplary method 600. In some specific implementations, the device (e.g., Figure 1The device 105) executes techniques for maintaining the user representation appearance during device removal. In some embodiments, the techniques for executing method 600 are performed on a mobile device, desktop computer, laptop computer, HMD, or server device. In some embodiments, method 600 is performed on processing logic components (including hardware, firmware, software, or combinations thereof). In some embodiments, method 600 is performed on a processor executing code stored in a non-transitory computer-readable medium (e.g., memory). In some embodiments, method 600 is implemented at the processor of a device (such as a viewing device) that renders the user representation (e.g., ...). Figure 2 The device 210 renders a 3D representation 240 of the user 260 based on data obtained from the device 265.

[0063] At box 610, method 600 relates to acquiring sensor data via one or more sensors during a time period. Such sensor data may include images, depth sensor data, or any other sensor data corresponding to the user’s appearance at a previous time point (e.g., at registration, during a previous session, or during the current session) and / or at the current time point.

[0064] At box 620, method 600 relates to determining, based on sensor data, the removal (e.g., detachment or removal) of the HMD during a given time period, wherein the removal of the HMD corresponds to a change in the HMD's position from a first position to a second position, where one or more displays of the HMD are positioned in front of the user's eyes, and where one or more displays are positioned in a different position relative to the user. Determining the removal of the HMD may include determining that the HMD is being detached. Determining the removal of the HMD may include determining a change in the user's eye position relative to the device's eye-fitting zone. Determining removal may involve determining that the device's position has changed from its normal use position in front of the user's eyes to a removal (detachment or removal) position. Determining removal may be based on (e.g., based on eye sensor data) detecting that the eye is no longer within the device's eye-fitting zone and that the device has changed its position or orientation by more than a threshold amount.

[0065] At box 630, method 600 relates to generating user representation data corresponding to the 3D appearance of a person during a given time period based on sensor data, wherein the user representation data is generated based on determining the removal of the HMD during that time period, and wherein the view of the user representation is provided based on the user representation data. The view of the user representation may, for example, be presented to another user during a real-time communication session with that other user.

[0066] Generating user representation data based on determining the removal of HMD during that time period can involve using current user data to generate user representation data before removal and not using current user data to generate representation data after removal.

[0067] After removal, user representation data can be configured to represent the user's neutral appearance. After removal, user representation data can be configured to represent the user's previous appearance. After removal, user representation data can be configured to represent the user's fixed appearance, such as a fixed appearance corresponding to the user's most recent appearance before removal. After removal, user representation data can be configured to represent the user's fixed appearance corresponding to the user's appearance before that time period (e.g., displaying a neutral avatar appearance based on the pre-session / registration neutral avatar appearance).

[0068] Generating user representation data based on determining the removal of HMD during that time period can involve providing user representation data that shows a gradual change from the current user appearance before the representation removal to a fixed user appearance after the representation removal.

[0069] Method 600 may also involve providing visual processing for the user representation, which indicates that the appearance of the user representation after removal may not depict the user's actual current appearance.

[0070] Figure 7 This is a block diagram of example device 700. Device 700 illustrates an exemplary device configuration for devices (e.g., devices 105, 210, 265, 410, etc.) described herein. Although certain specific features are illustrated, those skilled in the art will recognize from this disclosure that various other features have not been illustrated for the sake of brevity and so as not to obscure further relevant aspects of the specific implementations disclosed herein. Therefore, as a non-limiting example, in some specific implementations, device 700 includes one or more processing units 702 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 706, one or more communication interfaces 708 (e.g., USB, Firewire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZigBee, SPI, I2C, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 710, one or more displays 712, one or more internal and / or external image sensor systems 714, memory 720, and one or more communication buses 704 for interconnecting these components and various other components.

[0071] In some embodiments, one or more communication buses 704 include circuitry that interconnects system components and controls communication between system components. In some embodiments, the one or more I / O devices and sensors 706 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time-of-flight, etc.), etc.

[0072] In some embodiments, one or more displays 712 are configured to present a view of a physical or graphical environment to a user. In some embodiments, one or more displays 712 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conducting electron emitter display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical systems (MEMS), and / or similar display types. In some embodiments, one or more displays 712 correspond to waveguide displays such as diffraction, reflection, polarization, and holography. In one example, device 10 includes a single display. In another example, device 10 includes displays for each of the user's eyes.

[0073] In some embodiments, one or more image sensor systems 714 are configured to acquire image data corresponding to at least a portion of the physical environment 102. For example, one or more image sensor systems 714 include one or more RGB cameras, monochrome cameras, IR cameras, depth cameras, event-based cameras, etc. (e.g., having a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor). In various embodiments, one or more image sensor systems 714 also include an illumination source emitting light, such as a flash. In various embodiments, one or more image sensor systems 714 also include an on-camera image signal processor (ISP) configured to perform multiple processing operations on the image data.

[0074] Memory 720 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 720 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 720 may optionally include one or more storage devices remotely located to one or more processing units 702. Memory 720 includes a non-transitory computer-readable storage medium.

[0075] In some embodiments, memory 720 or a non-transitory computer-readable storage medium of memory 720 stores an optional operating system 730 and one or more instruction sets 740. Operating system 730 includes procedures for handling various basic system services and for performing hardware-related tasks. In some embodiments, instruction set 740 includes executable software defined by binary information stored in charge. In some embodiments, instruction set 740 is software executable by one or more processing units 702 to implement one or more of the techniques described herein.

[0076] Instruction set 740 includes registration instruction set 742, sensor data instruction set 744, user representation instruction set 746, and view generation instruction set 748. Instruction set 740 can be embodied in a single software executable file or multiple software executable files.

[0077] In some implementations, the registration instruction set 742 can be executed by the processing unit 702 to generate registration data from image or other sensor data. The registration instruction set 742 can be configured to provide instructions to a user to acquire image or other sensor information to generate registration data, and to determine whether additional information is required. For these purposes, in various implementations, the instructions include instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0078] In some embodiments, the sensor data instruction set 744 can be executed by the processing unit 702 to obtain sensor data, as described herein. For these purposes, in various embodiments, the instructions include instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0079] In some implementations, the user representation instruction set 746 can be executed by the processing unit 702 to generate and / or modify the user representation (e.g., user representation data) using one or more of the techniques discussed herein or as may otherwise be appropriate. For these purposes, in various implementations, the instructions include instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0080] In some implementations, the view generation instruction set 748 can be executed by the processing unit 702 to perform shading and / or rendering to provide a user-represented view using one or more of the techniques discussed herein or, as may otherwise be appropriate. For these purposes, in various implementations, the instructions include instructions and / or logic for the instructions, as well as heuristics and metadata for the heuristics.

[0081] Although instruction set 740 is shown as residing on a single device, it should be understood that in other specific implementations, any combination of elements may reside in separate computing devices. Furthermore, Figure 7 This is intended more as a functional description of various features present in a particular implementation than as a structural diagram of the specific implementation described herein. As those skilled in the art will recognize, items shown individually can be combined, and some items can be separate. The actual number of instruction sets and how features are allocated therein will vary depending on the specific implementation and may depend in part on the specific combination of hardware, software, and / or firmware chosen for that particular implementation.

[0082] Figure 8 A block diagram illustrating an exemplary head-mounted device 1000 according to some specific embodiments is shown. The head-mounted device 1000 includes a housing 1001 (or shell) housing various components of the head-mounted device 1000. The housing 1001 includes (or is coupled to) eye pads (not shown) disposed at a proximal end of the housing 1001 (relative to the user 105). In various specific embodiments, the eye pads are plastic or rubber components that comfortably and snugly hold the head-mounted device 1000 in a proper position on the face of the user 105 (e.g., around the eyes of the user 105).

[0083] The housing 1001 houses a display 1010 that displays images, thereby emitting light toward or onto the eyes of the user 105. In various embodiments, the display 1010 emits light through an eyepiece having one or more optical elements 1005 that refract the light emitted by the display 1010, making the display appear to the user 105 at a virtual distance greater than the actual distance from the eye to the display 1010. For example, the optical elements 1005 may include one or more lenses, waveguides, other diffractive optical elements (DOEs), etc. In order for the user 105 to focus on the display 1010, in various embodiments, the virtual distance is at least greater than the minimum focal length of the eye (e.g., 7 cm). Furthermore, to provide a better user experience, in various embodiments, the virtual distance is greater than 1 meter.

[0084] The housing 1001 also houses a tracking system including one or more light sources 1022, cameras 1024, 1032, 1034, and a controller 1080. One or more light sources 1022 emit light onto the eye of user 105, which is reflected as a light pattern (e.g., a flash) detectable by camera 1024. Based on this light pattern, controller 1080 can determine the eye-tracking characteristics of user 105. For example, controller 1080 can determine the gaze direction and / or blinking state (open or closed eyes) of user 105. As another example, controller 1080 can determine the pupil center, pupil size, or point of focus. Thus, in various embodiments, light is emitted by one or more light sources 1022, reflected from the eye of user 105, and detected by camera 1024. In various embodiments, light from the eye of user 105 is reflected from a hot mirror or transmitted through an eyepiece before reaching camera 1024.

[0085] Display 1010 emits light within a first wavelength range, and one or more light sources 1022 emit light within a second wavelength range. Similarly, camera 1024 detects light within the second wavelength range. In various specific embodiments, the first wavelength range is the visible wavelength range (e.g., a wavelength range of approximately 400 nm to 700 nm within the visible spectrum), and the second wavelength range is the near-infrared wavelength range (e.g., a wavelength range of approximately 700 nm to 1400 nm within the near-infrared spectrum).

[0086] In various implementations, eye tracking (or specifically, a defined gaze direction) is used to enable user interaction (e.g., user 105 selecting an option by looking at it on display 1010), to provide foveated rendering (e.g., rendering a higher resolution in the area of ​​display 1010 that user 105 is viewing and a lower resolution in other locations on display 1010), or (e.g., to correct distortion for an image to be displayed on display 1010). In various implementations, one or more light sources 1022 emit light toward the eye 35 of user 105, which is reflected in the form of multiple flashes.

[0087] In various implementations, camera 1024 is a frame / shutter-based camera that generates images of user 105's eye 35 at specific time points or multiple time points at a certain frame rate. Each image includes a matrix of pixel values ​​corresponding to the pixels of the image, with these pixels corresponding to the positions of the camera's light sensor matrix. In specific implementations, each image is used to measure or track pupil dilation by measuring changes in pixel intensity associated with one or both of the user's pupils.

[0088] In various specific implementations, camera 1024 is an event camera that includes multiple light sensors (e.g., a light sensor matrix) located at multiple corresponding positions, which generates an event message indicating a specific location of a particular light sensor in response to a particular light sensor detecting a change in light intensity.

[0089] In various specific implementations, cameras 1032 and 1034 are frame / shutter-based cameras capable of generating images of the user 105's face at specific or multiple time points at a certain frame rate. For example, camera 1032 captures an image of the user's face below the eyes, and camera 1034 captures an image of the user's face above the eyes. The images captured by cameras 1032 and 1034 may include light intensity images (e.g., RGB) and / or depth image data (e.g., time-of-flight, infrared, etc.).

[0090] It should be understood that the specific embodiments described above are cited by way of example, and this disclosure is not limited to what has been specifically shown and described above. Rather, the scope includes both combinations and sub-combinations of the various features described above, as well as variations and modifications of features that would occur to those skilled in the art upon reading the foregoing description and that are not disclosed in the prior art.

[0091] As described above, one aspect of the present invention is the collection and use of physiological data to improve the user experience with electronic devices when interacting with electronic content. This disclosure envisions that, in some cases, the collected data may include personal information data that uniquely identifies a particular person or can be used to identify the interests, characteristics, or preferences of a particular person. Such personal information data may include physiological data, demographic data, location-based data, telephone numbers, email addresses, home addresses, device characteristics of personal devices, or any other personal information.

[0092] This disclosure recognizes that the use of such personal information data in the present invention can benefit users. For example, personal information data can be used to improve the interactivity and controllability of electronic devices. Therefore, the use of such personal information data enables planned control of electronic devices. Furthermore, this disclosure envisions other uses for personal information data that benefit users.

[0093] This disclosure further envisions that entities responsible for the collection, analysis, disclosure, transmission, storage, or other use of such personal information and / or physiological data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and measures recognized as meeting or exceeding industry or governmental requirements for maintaining the privacy and security of personal information data. For example, personal information from users should be collected for legitimate and reasonable purposes of the entity and not shared or sold outside of these legitimate purposes. Furthermore, such collection should only be conducted after receiving informed consent from users. Additionally, such entities should take any necessary steps to safeguard and protect access to such personal information data and ensure that others with access to such personal information data comply with their privacy policies and procedures. Moreover, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices.

[0094] Regardless of the foregoing, this disclosure also envisions specific implementations that allow users to selectively block the use or access to personal information data. That is, this disclosure envisions providing hardware or software components to prevent or block access to such personal information data. For example, with regard to a content delivery service tailored to a user, the technology of this invention can be configured to allow the user to choose to "join" or "opt out" of the collection of personal information data during service registration. In another example, the user may choose not to provide personal information data for the target content delivery service. In yet another example, the user may choose not to provide personal information but allow the transmission of anonymous information for improving device functionality.

[0095] Therefore, while this disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, it also contemplates that various embodiments can be implemented without access to such personal information data. That is, various embodiments of the present invention will not become inoperable due to the absence of all or part of such personal information data. For example, preferences or settings can be inferred based on non-personal information data or an absolute minimum amount of personal information, such as content requested by a device associated with a user, other non-personal information available to the content delivery service, or publicly available information, thereby selecting content and delivering it to the user.

[0096] In some implementations, data is stored using a public / private key system that only allows the data owner to decrypt the stored data. In other implementations, data may be stored anonymously (e.g., without identification and / or without personal information about the user, such as legal name, username, time, and location data). This prevents other users, hackers, or third parties from identifying the user associated with the stored data. In some implementations, a user can access their stored data from a user device different from the device used to upload the stored data. In these cases, the user may need to provide login credentials to access their stored data.

[0097] This document sets forth numerous specific details to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will understand that the claimed subject matter can be practiced without these specific details. In other instances, methods, apparatus, or systems known to a person of ordinary skill have not been described in detail so as not to obscure the claimed subject matter.

[0098] Unless otherwise specifically stated, it should be understood that throughout this specification, discussions using terms such as “processing,” “calculating,” “calculating,” “determining,” and “identifying” refer to the actions or processes of computing devices, such as one or more computers or similar electronic computing devices, which manipulate or convert data representing physical electronic or magnetic quantities within the memory, registers, or other information storage, transmission, or display devices of a computing platform.

[0099] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include computer systems based on multi-purpose microprocessors that access stored software that programs or configures the computing system from a general-purpose computing device to a special-purpose computing device that implements one or more specific embodiments of the subject matter of this invention. The teachings contained herein may be implemented in the software used for programming or configuring the computing device using any suitable programming, scripting, or other type of language or combination of languages.

[0100] Specific implementations of the methods disclosed herein can be performed in the operation of such computing devices. The order of the boxes presented in the above examples can be changed; for example, the boxes can be reordered, combined, or divided into sub-boxes. Some boxes or processes can be executed in parallel.

[0101] The use of "applies to" or "configured to" in this document implies open and inclusive language, which does not exclude applicability to or configuration for performing additional tasks or steps. Additionally, the use of "based on" implies openness and inclusivity, as processes, steps, calculations, or other actions "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbering included herein are for illustrative purposes only and are not intended to be restrictive.

[0102] It will also be understood that while terms such as "first," "second," etc., may be used in this document to describe various objects, these objects should not be limited by these terms. These terms are merely used to distinguish one object from another. For example, a first node can be called a second node, and similarly, a second node can be called a first node, changing the meaning of the description, provided that all occurrences of "first node" are consistently renamed and all occurrences of "second node" are consistently renamed. First nodes and second nodes are both nodes, but they are not the same node.

[0103] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of these embodiments and the appended claims, the singular forms “a,” “an,” and “the” are intended to also cover the plural forms unless the context clearly indicates otherwise. It will also be understood that the term “or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will also be understood that the terms “comprising” or “including” as used in this specification specify the presence of the stated features, integers, steps, operations, objects, or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, objects, components, or groups thereof.

[0104] As used herein, the term "if" can be interpreted as meaning "when the prerequisite is true" or "when the prerequisite is true" or "in response to determination" or "according to determination" or "in response to detection" that the prerequisite is true, depending on the context. Similarly, the phrases "if it is determined [the prerequisite is true]" or "if [the prerequisite is true]" or "when [the prerequisite is true]" are interpreted as meaning "when it is determined that the prerequisite is true" or "in response to determination" or "according to determination" that the prerequisite is true or "when it is detected that the prerequisite is true" or "in response to detection" that the prerequisite is true, depending on the context.

[0105] The foregoing description and summary of the invention should be understood as exemplary and illustrative in every respect, and not restrictive, and the scope of the invention disclosed herein is determined not only by the detailed description of the exemplary specific implementations, but also by the full extent permitted by patent law.

[0106] It should be understood that the specific embodiments shown and described herein are merely illustrative of the principles of the invention, and those skilled in the art can make various modifications without departing from the scope and spirit of the invention.

Claims

1. A method, the method comprising: At the processor and one or more sensors in the head-mounted device (HMD): Sensor data is acquired via one or more sensors during the time period; The removal of the HMD during the time period is determined based on the sensor data, wherein the removal of the HMD corresponds to a change of the HMD from a first position to a second position, in which one or more displays of the HMD are positioned in front of the user's eyes, and in the second position, the one or more displays are positioned at other positions relative to the user. as well as User representation data is generated corresponding to the three-dimensional (3D) appearance of a person during the time period, wherein the user representation data is generated based on the determination of the removal of the HMD during the time period, and the view of the user representation is provided based on the user representation data.

2. The method of claim 1, wherein determining the removal of the HMD includes determining that the HMD is being removed.

3. The method of claim 1, wherein determining the removal of the HMD comprises detecting changes in the positioning or orientation of the HMD based on motion sensor data.

4. The method of claim 1, wherein determining the removal of the HMD includes determining a change in the user's eye positioning relative to the eye-friendly zone of the device.

5. The method of claim 1, wherein determining the removal of the HMD comprises detecting, based on eye sensor data, that the user's eyes are no longer within the eye-friendly zone of the HMD.

6. The method of claim 1, wherein generating the user representation data based on determining the removal of the HMD during the time period comprises: The user representation data is generated using the current user data prior to the removal. as well as The representation data is not generated using the current user data after the removal.

7. The method of claim 1, wherein after the removal, the user representation data represents a neutral appearance of the user.

8. The method of claim 1, wherein after the removal, the user representation data represents the user's previous appearance.

9. The method of claim 1, wherein after the removal, the user representation data represents a fixed appearance of the user corresponding to the user's most recent appearance before the removal.

10. The method of claim 1, wherein after the removal, the user representation data represents a fixed appearance of the user corresponding to the appearance of the user prior to the time period.

11. The method of claim 1, wherein generating the user representation data based on determining the removal of the HMD during the time period comprises: The user representation data is provided as a gradual change from representing the current user appearance before the removal to representing the fixed user appearance after the removal.

12. The method of claim 1, further comprising providing visual processing to the user representation, the visual processing indicating that the appearance of the user representation after the removal may not depict the user's actual current appearance.

13. The method of claim 1, wherein the view represented by the user is presented to another user during a real-time communication session.

14. A head-mounted device (HMD), the HMD comprising: Non-transitory computer-readable storage medium; One or more sensors; as well as One or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations, the operations including: Sensor data is acquired via one or more sensors during the time period; The removal of the HMD during the time period is determined based on the sensor data, wherein the removal of the HMD corresponds to a change in the HMD from a first position to a second position, where one or more displays of the HMD are positioned in front of the user's eyes, and where the one or more displays are positioned at a different location relative to the user at the second position; and User representation data is generated corresponding to the three-dimensional (3D) appearance of a person during the time period, wherein the user representation data is generated based on the determination of the removal of the HMD during the time period, and the view of the user representation is provided based on the user representation data.

15. The HMD of claim 14, wherein determining the removal of the HMD includes determining that the HMD is being removed.

16. The HMD of claim 14, wherein determining the removal of the HMD comprises: Changes in the positioning or orientation of the HMD are detected based on motion sensor data; Determine the change in the user's eye positioning relative to the eye-friendly zone of the device; or The system uses eye sensor data to detect that the user's eyes are no longer within the eye-friendly zone of the HMD.

17. The HMD of claim 14, wherein generating the user representation data based on determining the removal of the HMD during the time period comprises: The user representation data is generated using the current user data prior to the removal. as well as The representation data is not generated using the current user data after the removal.

18. The device of claim 14, wherein after the removal, the user representation data indicates: The user's neutral appearance; The user's previous appearance; The user's fixed appearance corresponding to the user's most recent appearance before removal; or The user's fixed appearance corresponding to the user's appearance before the stated time period.

19. The HMD of claim 14, wherein generating the user representation data based on determining the removal of the HMD during the time period comprises: The user representation data is provided as a gradual change from representing the current user appearance before the removal to representing the fixed user appearance after the removal.

20. A non-transitory computer-readable storage medium storing program instructions executable on a device to perform operations, the device comprising: One or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations, the operations including: Sensor data is acquired via one or more sensors during a given time period; The removal of the HMD during the time period is determined based on the sensor data, wherein the removal of the HMD corresponds to a change in the HMD from a first position to a second position, where one or more displays of the HMD are positioned in front of the user's eyes, and where the one or more displays are positioned in a different position relative to the user at the second position; and User representation data is generated corresponding to the three-dimensional (3D) appearance of a person during the time period, wherein the user representation data is generated based on the determination of the removal of the HMD during the time period, and the view of the user representation is provided based on the user representation data.

Images