Generating realistic avatar for augmented reality

Abstract

The present disclosure provides for generating a realistic avatar for augmented reality. A method for generating a realistic avatar of a user by an electronic device is presented. The method includes displaying, via at least one display, a sequence of frames to a user of the electronic device. The frame sequence includes a first frame including a first color component, a second frame including a second color component, and a third frame including a third color component. The method includes capturing, by one or more cameras, a plurality of images of a user while displaying a sequence of frames to the user. The method further includes, for each image of the plurality of images, determining a visible wavelength for each of the first color component, the second color component, and the third color component, and generating a realistic avatar of the user based on the visible wavelength for each of the first color component, the second color component, and the third color component.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. non-provisional patent application No. 18 / 458,708, filed on August 30, 2023, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates generally to extended reality environments, and more particularly, to generating realistic avatars for extended reality environments. Background Art

[0004] Extended reality is a form of reality that has been adjusted in some way before being presented to a user, and may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Extended reality content may include fully generated content, or generated content combined with captured content (e.g., a photo of the real world). Extended reality content may include video, audio, tactile feedback, or some combination thereof, and any of them may be presented in a single channel or multiple channels (e.g., stereoscopic video that produces a three-dimensional effect to the viewer). Extended reality may be associated with an application, product, accessory, service, or some combination thereof, for example, for creating content in extended reality and / or for use in extended reality (e.g., performing activities in extended reality). An extended reality system that provides extended reality content can be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a stand-alone HMD, a mobile device or computing system, or any other hardware platform capable of providing extended reality content to one or more viewers. In some extended reality applications, a user can manually create an avatar, which can include a virtual representation of the user as the user navigates and experiences the extended reality environment. However, manually creating an avatar can be time-consuming and complicated for the user, especially when the user wants to start a work-related extended reality application or experience, or other similar time-sensitive application or experience. Summary of the invention

[0005] The present embodiments relate to techniques for generating realistic avatars of users based on wavelengths of visible light obtained, detected, or calculated using monochrome cameras in extended reality (XR) display devices, which are suitable for eye tracking or facial tracking. In some embodiments, an electronic device may display a sequence of frames to a user of the electronic device through at least one display of the electronic device. In one embodiment, the sequence of frames may include a first frame, a second frame, and a third frame, the first frame including a first color component, the second frame including a second color component, and the third frame including a third color component. For example, in some embodiments, the first color component may include a red component, the second color component may include a green component, and the third color component may include a blue component. In some embodiments, the first frame may include only a red component, the second frame may include only a green component, and the third frame may include only a blue component.

[0006] In some embodiments, instead of displaying a sequence of frames to a user, an electronic device may project a sequence of light pulses onto the user through a plurality of light sources of the electronic device. For example, the sequence of light pulses may include a first light pulse, a second light pulse, and a third light pulse frame (e.g., a third light pulse), the first light pulse including only a red component, the second light pulse including only a green component, and the third light pulse frame including only a blue component. In one embodiment, the plurality of light sources may include a plurality of light-emitting diodes (LEDs). In some embodiments, while displaying the sequence of frames to the user, the electronic device may then capture a plurality of images of the user through one or more cameras of the electronic device. For example, in some embodiments, capturing a plurality of images of the user through one or more cameras may include: while displaying a first frame including a red component, capturing a first image of a plurality of images of the user; while displaying a second frame including a green component, capturing a second image of a plurality of images of the user; and while displaying a third frame including a blue component, capturing a third image of a plurality of images of the user.

[0007] In some embodiments, the electronic device may then determine the visible light wavelength of each of the first color component, the second color component, and the third color component for each of the plurality of images of the user. For example, the electronic device may determine the visible light wavelength for the red component of the first image, the visible light wavelength for the green component of the second image, and the visible light wavelength for the blue component of the third image. In some embodiments, determining the visible light wavelength for each of the plurality of images of the user may include determining a color for each of a plurality of features of the user. For example, in some embodiments, the plurality of features of the user may include one or more of the following: the user's eye color, the user's hair color, the user's skin complexion, the user's facial hair color, or the user's cheek complexion.

[0008] In some embodiments, the electronic device may then generate a realistic avatar of the user based on the visible light wavelength of each of the red component, the green component, and the blue component. For example, in some embodiments, generating a realistic avatar of the user may include combining multiple images of the user based on the visible light wavelength of each of the red component, the green component, and the blue component. In some embodiments, the electronic device may also determine the user's heart rate based on the visible light wavelength of each of the red component, the green component, and the blue component. In some embodiments, the electronic device may also determine the user's blood oxygen level based on the visible light wavelength of each of the red component, the green component, and the blue component.

[0009] The embodiments disclosed herein are merely examples, and the scope of the present disclosure is not limited to these embodiments. Certain embodiments may include all, some, or not all of the components, elements, features, functions, operations, or steps in the embodiments disclosed above, or may not include the components, elements, features, functions, operations, or steps of the embodiments disclosed herein. The attached claims relating to methods, storage media, systems, and computer program products disclose, in particular, embodiments according to the present invention, wherein any feature mentioned in one claim category (e.g., method) may also be claimed in another claim category (e.g., system). The dependencies or backreferences in the attached claims are selected for formal reasons only. However, any subject matter generated from an intentional reference to any previous claim (particularly a plurality of dependent claims) may also be claimed, so that any combination of a plurality of claims and their plurality of features is disclosed and may be claimed regardless of the dependencies selected in the attached claims. The subject matter that may be claimed includes not only the combination of features stated in the attached claims, but also any other combination of features in the claims, wherein each feature mentioned in the claims may be combined with any other feature or combination of other features in the claims. Furthermore, any of the embodiments and features described or depicted herein may be claimed in a separate claim and / or in any combination with any embodiment or feature described or depicted herein or in any combination with any features in the appended claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] This patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with one or more color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0011] Figure 1 A cross section of an example extended reality (XR) display device is illustrated.

[0012] Figure 2 Depicted are illustrative examples of generating a realistic avatar of a user based on wavelengths of visible light obtained, detected, or calculated using a monochrome camera suitable for eye tracking or facial tracking in an XR display device.

[0013] Figure 3 A timing diagram showing the timing of eye tracking and face tracking function operations, display operations, and camera operations is illustrated.

[0014] Figure 4A flow chart illustrating a method for generating a realistic avatar of a user based on wavelengths of visible light obtained, detected, or calculated using a monochrome camera suitable for eye tracking or facial tracking in an XR display device.

[0015] Figure 5 An example network environment associated with a social networking system is illustrated.

[0016] Figure 6 An example computer system is illustrated. DETAILED DESCRIPTION

[0017] The present embodiments relate to techniques for generating a realistic avatar of a user based on wavelengths of visible light obtained, detected, or calculated using a monochrome camera suitable for eye tracking or facial tracking in an extended reality (XR) display device. In some embodiments, an electronic device may display a sequence of frames to a user of the electronic device through at least one display of the electronic device. In one embodiment, the sequence of frames includes a first frame, a second frame, and a third frame, the first frame including a first color component, the second frame including a second color component, and the third frame including a third color component. For example, in some embodiments, the first color component may include a red component, the second color component may include a green component, and the third color component may include a blue component. In some embodiments, the first frame may include only a red component, the second frame may include only a green component, and the third frame may include only a blue component.

[0018] In some embodiments, instead of displaying a sequence of frames to a user, an electronic device may project a sequence of light pulses onto the user through a plurality of light sources of the electronic device. For example, the sequence of light pulses may include a first light pulse, a second light pulse, and a third light pulse frame, the first light pulse including only a red component, the second light pulse including only a green component, and the third light pulse frame including only a blue component. In one embodiment, the plurality of light sources may include a plurality of light emitting diodes (LEDs). In some embodiments, while displaying the sequence of frames to the user, the electronic device may capture a plurality of images of the user through one or more cameras of the electronic device. For example, in some embodiments, capturing a plurality of images of the user through one or more cameras may include: capturing a first image of a plurality of images of the user while displaying a first frame including a red component; capturing a second image of a plurality of images of the user while displaying a second frame including a green component; and capturing a third image of a plurality of images of the user while displaying a third frame including a blue component.

[0019] In some embodiments, the electronic device may then determine the visible light wavelength of each of the first color component, the second color component, and the third color component for each of the plurality of images of the user. For example, the electronic device may determine the visible light wavelength for the red component of the first image, the visible light wavelength for the green component of the second image, and the visible light wavelength for the blue component of the third image. In some embodiments, determining the visible light wavelength for each of the plurality of images of the user may include determining a color for each of a plurality of features of the user. For example, in some embodiments, the plurality of features of the user may include one or more of: the user's eye color, the user's hair color, the user's skin complexion, the user's facial hair color, or the user's cheek complexion.

[0020] In some embodiments, the electronic device may then generate a realistic avatar of the user based on the visible light wavelength of each of the red component, the green component, and the blue component. For example, in some embodiments, generating a realistic avatar of the user may include combining multiple images of the user based on the visible light wavelength of each of the red component, the green component, and the blue component. In some embodiments, the electronic device may also determine the user's heart rate based on the visible light wavelength of each of the red component, the green component, and the blue component. In some embodiments, the electronic device may also determine the user's blood oxygen level based on the visible light wavelength of each of the red component, the green component, and the blue component.

[0021] Figure 1 A cross section of an example extended reality (XR) display device 100 (e.g., a head mounted display (HMD) or XR glasses) according to presently disclosed embodiments is illustrated. The XR display device 100 may include one or more displays 110, which may include at least one waveguide 115. It should be understood that the XR display device 100 shown is an example of one embodiment of an HMD that may be useful in providing adaptive rendering according to presently disclosed embodiments. In another embodiment, for example, the XR display device 100 may include a see-through HMD that may not include a waveguide and may instead render an image directly on one or more transparent or semi-transparent mirrors that may be placed in front of the user's eyes, for example. Figure 1 Also shown is an eyebox 122, which is where the user's eyes 120 can be positioned when the user wears the XR display device 100. As long as the user's eyes 120 are aligned with the eyebox 122, the user can see a full-color image, or a pupil replication directed to the eyebox 122 by the waveguide 115.

[0022] In some embodiments, it can also be shown that when the user wears the XR display device 100, the user's face 124A, 124B can be positioned on both sides of the user's eyes 120. The waveguide 115 can generate a number of pupil copies and guide these pupil copies to the eye area 122. The waveguide 115 can be configured to guide the image light 160 to the eye area 122 located near the eye 120. For illustration purposes, Figure 1 A cross section associated with a single eye 120 and a single waveguide 115 is shown. In some embodiments, waveguide 115 or another waveguide can provide image light to an eye zone located at the user's other eye. In some embodiments, waveguide 115 can be composed of one or more materials (e.g., plastic, glass, etc.) having one or more refractive indices that effectively minimize the weight of display device 110 and expand the field of view (FOV) of display device 110.

[0023] In other embodiments, the display 110 may include one or more optical elements located between the waveguide 115 and the user's eye 120. The optical elements may be used, for example, to correct for deviations in the image light 160, to amplify the image light 160, to make some other optical adjustments to the image light 160, or to perform a combination thereof. Examples of optical elements may include an aperture, a Fresnel lens, a refractive (e.g., convex and / or concave) lens, a reflective surface, a filter, or any other suitable optical element that affects the image light. For example, the waveguide 115 may include a waveguide having one or more sets of Bragg gratings.

[0024] In one embodiment, a display 110 that may be used in an XR display device 100 may be referred to as a scanline or one-dimensional ("1D") waveguide display. In this display, a row of light sources may generate light for illuminating the entire vertical space (or horizontal space, as appropriate) of the display. Multiple smaller images may be combined to form a larger composite image perceived by a viewer. The scanning element may cause source light processed by the waveguide component to be output to the user's eye 120 in a specific pattern corresponding to the generation pattern used by the emitter to optimize the display drawing rate. For example, the light source may first be provided with color values ​​that correspond to a single row of pixels along the top of the displayed image. The light may be transmitted to the appropriate portion of the eye zone 122 using a waveguide-based process that utilizes an oscillating mirror driven by a microelectromechanical system (MEMS) to assist.

[0025] In some embodiments, after a short period of time, the light source can be provided with color values ​​corresponding to the next row of pixels (e.g., below the first row). The light for this portion of the image can then use the same process to position the color value in the appropriate location. Scanning displays can use less power to operate and may generate less heat than traditional displays composed of the same emitters. Scanning displays may also have a lighter weight, in part due to the quality of the materials used in the scanning elements and optical system. The frame rate of the display can be limited based on the oscillation speed of the mirror.

[0026] In another embodiment, a display 110 that can be used in the XR display device 100 can be a 2D or two-dimensional waveguide display. In such a display, an oscillating mirror may not be used because a light source including a vertical component and a horizontal component (e.g., in an array form) can be used. Where a 2D variant illuminates the display row by row, the 2D variant may be able to provide a significantly improved frame rate because the 2D variant may not rely on an oscillating mirror to provide the vertical component of the image. To further increase the frame rate, the light source of the 2D waveguide display may be coupled to a controller and / or memory that provides drive instructions for the display system. For example, the light source may be coupled to a memory that stores color instructions for the display and / or drive transistors. As a result of this configuration, the light source for such a display can operate at a fairly fast frame rate.

[0027] In some embodiments, the XR display device 100 may include a light source such as a projector 112 that emits projection light 155 depicting one or more images. Many suitable display light source technologies are contemplated, including but not limited to liquid crystal display (LCD), liquid crystal on silicon (LCOS), light emitting diode (LED), organic LED (OLED), micro-LED (μLED), digital micromirror device (DMD), or any combination thereof. The projection light 155 may be received by a first coupler 150 of a waveguide 115. The waveguide 115 may combine the projection light 155 with real-world scene light 116 received by a second coupler 152. For example, the scene light 116 may be light from a real-world environment and may pass through a transparent (or translucent) surface 154 to reach the second coupler 152. In some embodiments, the XR display device 100 may also include multiple light sources (e.g., a set of light emitting diodes (LEDs)) that may be used to project emitted projection light 155 onto the user's eyes 120 and / or the user's face 124A, 124B.

[0028] In some embodiments, the transparent surface 154 can be, for example, protective curved glass or a lens formed of glass, plastic, or other transparent material. The coupling components of the waveguide 115 can guide the projection light 155 along the total internal reflection path of the waveguide 115. The scene light 116 can be seen by the user's eye 120. In addition, the projection light 155 can first pass through a small air gap between the projector 112 and the waveguide 115 before interacting with a coupling element (e.g., the first coupler 150) contained in the waveguide. In some examples, the optical path can include grating structures or other types of optical decoupling structures that decouple portions of the light from the total internal reflection path to guide multiple instances of the image ("pupil replicas") out of the waveguide 115 at different locations and toward the eye zone 122 of the XR display device 100.

[0029] In certain embodiments, such as Figure 1 As further depicted, the XR display device 100 may include a plurality of cameras 126A, 126B, and 126C. In certain embodiments, the plurality of cameras 126A, 126B, and 126C may include one or more infrared cameras 126A, one or more monochrome cameras 126B, and one or more depth cameras 126C, which may be adapted to perform eye tracking of the user's eyes 120, facial tracking of the user's face 124A, 124B, or a combination of eye tracking of the user's eyes 120 and facial tracking of the user's face 124A, 124B. For example, in one embodiment, one or more monochrome cameras 126B may operate in a narrow infrared band (e.g., 850 nanometers (nm)) and exclude visible spectrum data capture. In certain embodiments, as will be described below with reference to Figures 2 to 4 As further understood, one or more monochrome cameras 126B (or one or more other cameras in the plurality of cameras 126A, 126B, and 126C) may be utilized to generate a realistic avatar of a user (e.g., including the user's eye color, the user's hair color, the user's skin complexion, the user's facial hair color, the user's cheek complexion, etc.) by temporarily removing the infrared bandpass filter in the one or more monochrome cameras 126B and not receiving any input from the user.

[0030] In some embodiments, one or more controllers 130 can control the operation of projector 112 and multiple cameras 126A, 126B, and 126C. Controller 130 can generate display instructions for the display system of projector 112 or image capture instructions for multiple cameras 126A, 126B, and 126C. For example, the display instructions may include instructions for projecting or transmitting one or more images, and the image capture instructions may include instructions for capturing one or more images in a continuous sequence. In some embodiments, the display instructions and image capture instructions may include frame image color or monochrome data. The display instructions and image capture instructions may be received from, for example, a display system including Figure 1 The display instructions may also include instructions for moving the projector 112, for moving the waveguide 115 by activating an actuation system, or for moving or adjusting the lens of one or more cameras 126A, 126B, and 126C. The controller 130 may include a combination of hardware, software, and / or firmware not explicitly shown herein to avoid obscuring other aspects of the present disclosure.

[0031] Figure 2 An illustrative example 200 of the current technology for generating a realistic avatar of a user using a monochrome camera suitable for eye tracking or facial tracking in an XR display device according to the currently disclosed embodiments is depicted. It should be understood that although the illustrative example 200 may be discussed theoretically herein with respect to the red apple in examples 202, 204, and 206, in this embodiment, red light, green light, and blue light may be shone on the user's eyes 120 and the user's face 124A, 124B. In some embodiments, when the user first places the XR display device 100 on her head, the display 110 of the XR display device 100 may display a sequence of frames to the user (e.g., shown by red light 208A, green light 208B, and blue light 208C). In one embodiment, the sequence of frames may include a first frame, a second frame, and a third frame, the first frame including only a red component, the second frame including only a green component, and the third frame including only a blue component.

[0032] For example, as depicted in illustrative example 200, display 110 may display (e.g., for a few milliseconds (ms) flash) a frame including only a red component as shown in example 202, a frame including only a green component as shown in example 204, and a frame including only a blue component as shown in example 206. In one embodiment, instead of display 110 displaying a sequence of frames to a user, one or more light sources (e.g., a plurality of red (R) LEDs, green (G) LEDs, blue (B) LEDs) of XR display device 100 may project a sequence of light pulses onto the user. For example, the sequence of light pulses (e.g., shown by red light 208A, green light 208B, and blue light 208C) may include a first light pulse including only a red component, a second light pulse including only a green component, and a third light pulse frame including only a blue component.

[0033] In some embodiments, while displaying a sequence of red, green, and blue frames to the user, one or more monochrome cameras 126B of the XR display device 100 can capture multiple images of the user. Specifically, one or more monochrome cameras 126B can capture reflected light 210A of red light 208A, reflected light 210B of green light 208B, and reflected light 210C of blue light 208C. For example, in some embodiments, as will be described with reference to FIG. Figure 3 As further understood, one or more monochrome cameras 126B can capture a first image of the user while the display 110 displays a first frame including a red component, a second image of the user while the display 110 displays a second frame including a green component, and a third image of the user while the display 110 displays a third frame including a blue component, in a time-synchronized sequence.

[0034] In some embodiments, the XR display device 100 may then determine the visible light wavelength of each of the red component, the green component, and the blue component for each of the images of the user captured. For example, the XR display device 100 may detect or calculate the visible light wavelength of each of the red component, the green component, and the blue component based on the reflected light 210A of the red light 208A, the reflected light 210B of the green light 208B, and the reflected light 210C of the blue light 208C captured by one or more monochrome cameras 126B. In some embodiments, the XR display device 100 may use the detected or calculated visible light wavelengths to determine the color of each of the user's multiple features. For example, in some embodiments, according to the currently disclosed technology, the user's multiple features may include one or more of the following: the user's eye color, the user's hair color, the user's skin color, the user's facial hair color, or the user's cheek skin color.

[0035] In some embodiments, the XR display device 100 may then generate a realistic avatar of the user based on the visible light wavelength of each of the red component, the green component, and the blue component. For example, in some embodiments, the XR display device 100 may generate a realistic avatar of the user (e.g., as depicted by the RGB image 212) by combining the captured image of the reflected light 210A of the red light 208A, the image of the reflected light 210B of the green light 208B, and the image of the reflected light 210C of the blue light 208C. In another embodiment, the XR display device 100 may also determine the heart rate of the user based on the visible light wavelength of each of the red component, the green component, and the blue component. In another embodiment, the XR display device 100 may also determine the blood oxygen level of the user based on the visible light wavelength of each of the red component, the green component, and the blue component.

[0036] Figure 3 A timing diagram 300 is illustrated according to the present embodiment, which shows a timing signal 302 for eye tracking and face tracking (ET / FT) functional operation, a timing signal 304 for display operation, and a timing signal 306 for camera (e.g., one or more monochrome cameras) operation. In some embodiments, the timing signal 302 (e.g., eye tracking and face tracking functional operation), the timing signal 304 (e.g., display operation), and the timing signal 306 (e.g., camera operation) may each include a rising edge, for example, the rising edge of the timing signal 302 may indicate when the eye tracking and face tracking functional operation is operating to track the user's eyes 120 or the user's face 124A, 124B, the rising edge of the timing signal 304 may indicate when the display 110 is operating to display (e.g., flashing for a few ms) a sequence of red frames, green frames, and blue frames, and the rising edge of the timing signal 306 may indicate when the one or more monochrome cameras 126B are to operate to capture images of the user's eyes 120 and the user's face 124A, 124B. For example, as depicted, in some embodiments, timing signal 304 (eg, display operation) and timing signal 306 (eg, camera operation) may be synchronized.

[0037] For example, according to the present embodiment, the timing signal 302 (e.g., eye tracking and facial tracking functionality operating) may begin at a logic low at the signal portion 308 such that the eye tracking and facial tracking functionality of the XR display device 100 may not operate while the functionality of generating a realistic avatar of a user using detected or calculated visible light wavelengths is operating. Specifically, as further depicted, the timing signal 304 (e.g., display operation) and the timing signal 306 (e.g., camera operation) can be logic high at signal portions 310 and 312, respectively, and logic low at substantially the same time, so that the display 110 and the one or more monochrome cameras 126B operate, for example, according to the following sequence: the display 110 displays a red frame to the user (e.g., a full red screen or a bright red light flashes), and while the display 110 displays the red frame to the user, the one or more monochrome cameras 126B capture a first image of the user; then, the display 110 displays a green frame to the user (e.g., a full green screen or a bright green light flashes), and while the display 110 displays the green frame to the user, the one or more monochrome cameras 126B capture a second image of the user; and finally, the display 110 displays a blue frame to the user (e.g., a full blue screen or a bright blue light flashes), and while the display 110 displays the blue frame to the user, the one or more monochrome cameras 126B capture a third image of the user.

[0038] In some embodiments, when the timing signal 304 (e.g., display operation) and the timing signal 306 (e.g., camera operation) become logic low at the signal portion 314 (e.g., after the display 110 displays a blue frame to the user and the one or more monochrome cameras 126B capture a third image of the user), the timing signal 302 (e.g., eye tracking and face tracking functionality operation) may become logic high at the signal portion 316 and then operate synchronously with the timing signal 306 (e.g., camera operation) to perform, for example, eye tracking and face tracking of the user. In this way, by offsetting the operation of the eye tracking and face tracking functionality and the operation of the functionality of generating a realistic avatar of the user using detected or calculated visible light wavelengths as described herein, the display 110 may not adversely affect the eye tracking and face tracking functionality of the XR display device 100.

[0039] Figure 4A flow chart of a method 400 for generating a realistic avatar of a user according to the presently disclosed embodiments is illustrated. The method 400 may be performed using one or more processors, which may include hardware (e.g., a general purpose processor, a graphic processing unit (GPU), an application-specific integrated circuit (ASIC), a system-on-chip (SoC), a microcontroller, a field-programmable gate array (FPGA), or one or more other processing devices suitable for processing image data), software (e.g., instructions running / executing on one or more processors), firmware (e.g., microcode), or any combination thereof.

[0040] Method 400 may begin at block 402, where one or more processors display a sequence of frames to a user of the electronic device through at least one display of the electronic device. For example, the sequence of frames may include a first frame, a second frame, and a third frame, the first frame including a first color component, the second frame including a second color component, and the third frame including a third color component. Method 400 may continue to block 404, where the one or more processors capture multiple images of the user through one or more cameras of the electronic device while displaying the sequence of frames to the user. Method 400 may continue to block 406, where the one or more processors determine, for each of the multiple images of the user, a wavelength of visible light for each of the first color component, the second color component, and the third color component. Method 400 may end at block 408, where the one or more processors generate a realistic avatar of the user based on the wavelength of visible light for each of the first color component, the second color component, and the third color component.

[0041] Figure 5 An example network environment 500 associated with a virtual reality system is illustrated. The network environment 500 includes a user 501 interacting with a client system 530, a social networking system 560, and a third-party system 570 connected to each other via a network 510. Figure 5A particular arrangement of user 501, client system 530, social networking system 560, third-party system 570, and network 510 is illustrated, but the present disclosure contemplates any suitable arrangement of user 501, client system 530, social networking system 560, third-party system 570, and network 510. By way of example and not limitation, two or more of user 501, client system 530, social networking system 560, and third-party system 570 may be directly connected to each other, bypassing network 510. As another example, two or more of client system 530, social networking system 560, and third-party system 570 may be physically or logically co-located in whole or in part with each other. Furthermore, although Figure 5 A particular number of users 501, client systems 530, social networking systems 560, third-party systems 570, and networks 510 are illustrated, but the present disclosure contemplates any suitable number of client systems 530, social networking systems 560, third-party systems 570, and networks 510. By way of example and not limitation, network environment 500 may include a plurality of users 501, a plurality of client systems 530, a plurality of social networking systems 560, a plurality of third-party systems 570, and a plurality of networks 510.

[0042] The present disclosure contemplates any suitable network 510. By way of example and not limitation, one or more portions of network 510 may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), a portion of the Internet, a portion of a Public Switched Telephone Network (PSTN), a cellular telephone network, or a combination of two or more of these networks. Network 510 may include one or more networks 510. Link 550 may connect client system 530, social networking system 560, and third-party system 570 to communication network 510, or may connect the client system, social networking system, and third-party system to each other. The present disclosure contemplates any suitable link 550. In some embodiments, one or more links 550 include one or more wired (e.g., Digital Subscriber Line (DSL) or Data Over Cable Service Interface Specification (DOCSIS)) links, one or more wireless (e.g., Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX)) links, or one or more optical (e.g., Synchronous Optical Network (SONET) or Synchronous Digital Hierarchy (SDH)) links. In some embodiments, one or more links 550 each include an ad hoc network, an intranet, an extranet, a VPN, a LAN, a WLAN, a WAN, a WWAN, a MAN, a portion of the Internet, a portion of the PSTN, a network based on cellular technology, a network based on satellite communication technology, another link 550, or a combination of two or more such links 550. Multiple links 550 do not have to be identical throughout the network environment 500. In one or more aspects, one or more first links 550 may be different from one or more second links 550.

[0043] In some embodiments, the client system 530 may be an electronic device that includes hardware, software, or embedded logic components or a combination of two or more such components and is capable of performing appropriate functions implemented or supported by the client system 530. By way of example and not limitation, the client system 530 may include a computer system such as a desktop computer, a notebook or laptop computer, a netbook, a tablet computer, an e-book reader, a global positioning system (GPS) device, a camera, a personal digital assistant (PDA), a handheld electronic device, a cellular phone, a smart phone, a virtual reality head-mounted device (headset) and controller, other suitable electronic devices, or any suitable combination thereof. The present disclosure contemplates any suitable client system 530. The client system 530 may enable network users at the client system 530 to access the network 510. The client system 530 may enable its users to communicate with other users at other client systems 530. The client system 530 may generate a virtual reality environment for users to interact with content.

[0044] In some embodiments, the client system 530 may include a virtual reality (or augmented reality) head mounted device 532, and one or more virtual reality input devices 534 (e.g., virtual reality controllers). A user at the client system 530 may wear the virtual reality head mounted device 532 and use the one or more virtual reality input devices to interact with a virtual reality environment 536 generated by the virtual reality head mounted device 532. Although not shown, the client system 530 may also include a separate processing computer, and / or any other components of the virtual reality system. The virtual reality head mounted device 532 may generate a virtual reality environment 536, which may include system content 538 (including but not limited to an operating system), such as software or firmware updates, and the virtual reality environment may also include third-party content 540, such as content from an application or content dynamically downloaded from the Internet (e.g., web content). The virtual reality head mounted device 532 may include one or more sensors 542 (e.g., accelerometers, gyroscopes, magnetometers), which are used to generate sensor data that tracks the position of the virtual reality head mounted device 532. The virtual reality head mounted device 532 may also include an eye tracker for tracking the position of the user's eyes or the viewing direction of the user's eyes. The client system may use data from the one or more sensors 542 to determine the speed, orientation, and gravity of the head mounted device.

[0045] The one or more virtual reality input devices 534 may include one or more sensors 544 (e.g., accelerometers, gyroscopes, magnetometers, and touch sensors) for generating sensor data that tracks the position of the input device 534 and the position of the user's fingers. The client system 530 may utilize outside-in tracking, in which a tracking camera (not shown) is placed outside of the virtual reality headset 532 and within the line of sight of the virtual reality headset 532. In outside-in tracking, the tracking camera may track the position of the virtual reality headset 532 (e.g., by tracking one or more infrared light emitting diode (LED) markers on the virtual reality headset 532). Alternatively or additionally, the client system 530 may utilize inside-out tracking, in which a tracking camera (not shown) may be placed on or within the virtual reality headset 532 itself. In inside-out tracking, the tracking camera may capture images of the real world surrounding the tracking camera, and may use the changing perspective of the real world to determine the position of the tracking camera itself in space.

[0046] The third-party content 540 may include a web browser (e.g., MICROSOFT INTERNET EXPLORER, GOOGLE CHROME, or MOZILLA FIREFOX), and may have one or more attachments, plug-ins, or other extensions (e.g., a tool bar or a YAHOO TOOLBAR). A user at the client system 530 may enter a URL or other address that directs the web browser to a specific server (e.g., server 562 or a server associated with the third-party system 570), and the web browser may generate a Hyper Text Transfer Protocol (HTTP) request and transmit the HTTP request to the server. The server may receive the HTTP request and transmit one or more Hyper Text Markup Language (HTML) files to the client system 530 in response to the HTTP request. The client system 530 may render a web interface (e.g., a web page) based on the HTML file from the server for presentation to the user. The present disclosure contemplates any suitable source file. As an example and not limitation, a network interface may be rendered according to an HTML file, an Extensible Hyper Text Markup Language (XHTML) file, or an Extensible Markup Language (XML) file, depending on specific needs. These interfaces may also execute scripts, such as but not limited to those written in JAVASCRIPT, JAVA, MICROSOFT SILVERLIGHT, and a combination of markup languages ​​and scripts (e.g., asynchronous JAVASCRIPT and XML (AJAX)), etc. In this document, where appropriate, a reference to a network interface includes one or more corresponding source files (which a browser may use to render the network interface), and vice versa.

[0047] In some embodiments, social networking system 560 may be a network addressable computing system that can host an online social network. Social networking system 560 may generate, store, receive, and transmit social networking data, such as user profile data, concept profile data, social graph information, or other suitable data related to an online social network. Social networking system 560 may be accessed directly by other components in network environment 500 or via network 510. By way of example and not limitation, client system 530 may access social networking system 560 directly or via network 510 using a web browser of third-party content 540, or a local application associated with social networking system 560 (e.g., a mobile social networking application, a messaging application, another suitable application, or any combination thereof). In some embodiments, social networking system 560 may include one or more servers 562. Each server 562 may be a single server, or a distributed server across multiple computers or multiple data centers. Server 562 can be of various types, such as, but not limited to, a web server, a news server, a mail server, a message server, an advertising server, a file server, an application server, an exchange server, a database server, a proxy server, another server suitable for performing the functions or processes described herein, or any combination thereof.

[0048] In some embodiments, each server 562 may include hardware, software, or embedded logic components or a combination of two or more such components for performing appropriate functions implemented or supported by the server 562. In some embodiments, the social networking system 560 may include one or more data repositories 564. The data repository 564 can be used to store various types of information. In some embodiments, the information stored in the data repository 564 can be organized according to a specific data structure. In some embodiments, each data repository 564 can be a relational database, a columnar database, an associated database, or other suitable database. Although the present disclosure describes or illustrates a specific type of database, the present disclosure contemplates any suitable type of database. Some embodiments may provide such interfaces: these interfaces enable the client system 530, the social networking system 560, or the third-party system 570 to manage, retrieve, modify, add, or delete the information stored in the data repository 564.

[0049] In some embodiments, the social networking system 560 may store one or more social graphs in one or more data stores 564. In some embodiments, the social graph may include multiple nodes—these nodes may include multiple user nodes (each user node corresponds to a specific user) or multiple concept nodes (each concept node corresponds to a specific concept)—and multiple edges connecting these nodes. The social networking system 560 may provide users of the online social network with the ability to communicate and interact with other users. In some embodiments, a user may join an online social network via the social networking system 560, and may subsequently add connections (e.g., relationships) with multiple other users in the social networking system 560 with whom he or she wants to connect. In this document, the term "friend" may refer to any other user in the social networking system 560 with whom a user has formed a connection, association, or relationship via the social networking system 560.

[0050] In some embodiments, social networking system 560 may provide users with the ability to take actions on various types of items or objects supported by social networking system 560. By way of example and not limitation, these items and objects may include: groups or social networks to which users of social networking system 560 may belong, events or calendar entries that users may be interested in, computer-based applications that users may use, transactions that allow users to buy or sell items through the service, interactions with advertisements that users may perform, or other suitable items or objects. Users may interact with anything that can be represented in social networking system 560, or with anything that can be represented by an external system of third-party system 570 that is separate from social networking system 560 and coupled to social networking system 560 via network 510.

[0051] In some embodiments, the social networking system 560 may be able to link various entities. As an example and not limitation, the social networking system 560 may enable multiple users to interact with each other and receive content from the third-party system 570 or other entities, or may allow users to interact with these entities through an application programming interface (API) or other communication channel. In some embodiments, the third-party system 570 may include one or more types of servers, one or more data repositories, one or more interfaces (including but not limited to APIs), one or more network services, one or more content sources, one or more networks, or any other suitable components (e.g., components with which a server can communicate). The third-party system 570 may be operated by an entity different from the entity operating the social networking system 560. However, in some embodiments, the social networking system 560 and the third-party system 570 may operate in conjunction with each other to provide social networking services to users of the social networking system 560 or users of the third-party system 570. In this sense, the social networking system 560 may provide a platform or backbone network that other systems (e.g., the third-party system 570) can use to provide social networking services and functions to users on the Internet.

[0052] In some embodiments, third-party system 570 may include a third-party content object provider. A third-party content object provider may include one or more sources of content objects that may be transmitted to client system 530. By way of example and not limitation, content objects may include information about things or activities of interest to a user, such as movie showtimes, movie reviews, restaurant reviews, restaurant menus, product information and reviews, or other suitable information. By way of another example and not limitation, content objects may include incentive content objects, such as coupons, discount tickets, gift certificates, or other suitable incentive objects.

[0053] In some embodiments, the social networking system 560 also includes user-generated content objects that can enhance the user's interaction with the social networking system 560. User-generated content can include anything that a user can add, upload, send, or "post" to the social networking system 560. By way of example and not limitation, a user transmits a post from a client system 530 to the social networking system 560. A post can include data such as a status update or other textual data, location information, photos, videos, links, music, or other similar data or media. Content can also be added to the social networking system 560 by a third party through a "communication channel" (e.g., a news feed or stream). In some embodiments, the social networking system 560 can include various servers, subsystems, programs, modules, logs, and data repositories. In some embodiments, the social networking system 560 can include one or more of the following: a network server, an action logger, an API request server, a relevance and ranking engine, a content object classifier, a notification controller, an action log, a third-party content object disclosure log, an inference module, an authorization / privacy server, a search module, an ad targeting module, a user interface module, a user profile repository, a contact repository, a third-party content repository, or a location repository. Social-networking system 560 may also include suitable components, such as web interfaces, security mechanisms, load balancers, failover servers, management and network operations consoles, other suitable components, or any suitable combination thereof.

[0054] In some embodiments, the social networking system 560 may include one or more user profile repositories for storing user profiles. User profiles may include, for example, biographical information, personal background information, behavioral information, social information, or other types of descriptive information, such as work experience, education, hobbies or preferences, interests, in-laws, or addresses. Interest information may include interests associated with one or more categories. Categories may be general or specific. As an example and not limitation, if a user "likes" an item about a shoe brand, the category may be the brand, or may be a general category of "shoes" or "clothing." A contact repository may be used to store contact information about a user. Contact information may indicate users who have similar or common work experience, group membership, hobbies, education, or are related or share common attributes in any way. Contact information may also include user-defined connections between different users and content (both internal and external). A network server may be used to link the social networking system 560 to one or more client systems 530 or one or more third-party systems 570 via a network 510. The network server may include a mail server or other messaging functionality for receiving and routing messages between the social networking system 560 and one or more client systems 530. The API request server may allow third-party systems 570 to access information from the social networking system 560 by calling one or more APIs. The action logger may be used to receive information from the network server regarding actions of users logging into or out of the social networking system 560.

[0055] In conjunction with the action log, a third-party content object log of user contact with third-party content objects can be maintained. The notification controller can provide information about content objects to the client system 530. The information can be pushed to the client system 530 as a notification, or the information can be extracted from the client system 530 in response to a request received from the client system 530. The authorization server can be used to implement one or more privacy settings of users of the social networking system 560. The privacy settings of the user can determine how specific information associated with the user can be shared. The authorization server can allow users to choose to have their actions recorded or not recorded by the social networking system 560 or shared with other systems (e.g., third-party systems 570), for example, by setting appropriate privacy settings. The third-party content object repository can be used to store content objects received from a third party (e.g., third-party system 570). The location repository can be used to store location information received from the client system 530 associated with the user. The advertising pricing module can combine social information, current time, location information, or other suitable information to provide relevant advertisements to users in the form of notifications.

[0056] Figure 6An example computer system 600 is illustrated, which can be used to perform one or more of the aforementioned techniques as currently disclosed herein. In some embodiments, one or more computer systems 600 perform one or more steps of one or more methods described or shown herein. In some embodiments, one or more computer systems 600 provide the functions described or shown herein. In some embodiments, software running on one or more computer systems 600 performs one or more steps of one or more methods described or shown herein, or provides the functions described or shown herein. Certain embodiments include one or more parts of one or more computer systems 600. In this article, where appropriate, references to computer systems may include computing devices, and vice versa. In addition, where appropriate, references to computer systems may include one or more computer systems.

[0057] The present disclosure contemplates any suitable number of computer systems 600. The present disclosure contemplates computer systems 600 in any suitable physical form. By way of example and not limitation, computer system 600 may be: an embedded computer system, a system on a chip (SoC), a single-board computer system (SBC) (e.g., a computer-on-module (COM) or a system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a computer system network, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented reality / virtual reality device, or a combination of two or more of these systems. Where appropriate, computer system 600 may include one or more computer systems 600; may be single or distributed; may span multiple locations; may span multiple machines; may span multiple data centers; or may be located in a cloud, which includes one or more cloud components in one or more networks. Where appropriate, one or more computer systems 600 may perform one or more steps in one or more methods described or shown herein without substantial spatial or temporal limitations.

[0058] As an example and not limitation, one or more computer systems 600 may perform one or more steps of one or more methods described or shown herein in real time or in batch processing mode. Where appropriate, one or more computer systems 600 may perform one or more steps of one or more methods described or shown herein at different times or in different locations. In some embodiments, computer system 600 includes processor 602, memory 604, storage 606, input / output (I / O) interface 608, communication interface 610, and bus 612. Although the present disclosure describes and illustrates a particular computer system with a particular number of particular components in a particular arrangement, the present disclosure contemplates any suitable computer system with any suitable number of any suitable components in any suitable arrangement.

[0059] In some embodiments, the processor 602 includes hardware for executing multiple instructions, such as those that constitute a computer program. By way of example and not limitation, to execute instructions, the processor 602 may retrieve (or read) these instructions from an internal register, an internal cache, a memory 604, or a storage device 606; decode and execute these instructions; and then write one or more results to an internal register, an internal cache, a memory 604, or a storage device 606. In some embodiments, the processor 602 may include one or more internal caches for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 602 that includes any suitable number of any suitable internal caches. By way of example and not limitation, the processor 602 may include one or more instruction caches, one or more data caches, and one or more page table caches (translation lookaside buffers, TLBs). The instructions in the instruction cache may be copies of the instructions in the memory 604 or the storage device 606, and the instruction cache may speed up the retrieval of these instructions by the processor 602.

[0060] The data in the data cache may be a copy of the data in the memory 604 or the storage device 606 for operation by the instructions executed at the processor 602; may be the result of the previous instruction executed at the processor 602 for access by the subsequent instructions executed at the processor 602 or for writing to the memory 604 or the storage device 606; or may be other suitable data. The data cache may speed up the read operation or write operation of the processor 602. The TLB may speed up the virtual address translation of the processor 602. In some embodiments, the processor 602 may include one or more internal registers for data, instructions, or addresses. Where appropriate, the present disclosure contemplates a processor 602 including any suitable number of any suitable internal registers. Where appropriate, the processor 602 may include one or more arithmetic logic units (ALUs); may be a multi-core processor; or may include one or more processors 602. Although the present disclosure describes and illustrates a specific processor, the present disclosure contemplates any suitable processor.

[0061] In some embodiments, the memory 604 includes a main memory for storing instructions for execution by the processor 602 or data for operation by the processor 602. As an example and not limitation, the computer system 600 can load instructions from the storage device 606 or another source (e.g., another computer system 600) to the memory 604. Then, the processor 602 can load the instructions from the memory 604 to an internal register or an internal cache memory. In order to execute the instructions, the processor 602 can retrieve the instructions from the internal register or the internal cache memory and decode these instructions. During or after executing the instructions, the processor 602 can write one or more results (the one or more results can be intermediate results or final results) to the internal register or the internal cache memory. Then, the processor 602 can write one or more of these results to the memory 604. In some embodiments, the processor 602 only executes instructions in one or more internal registers or internal cache memories or in the memory 604 (rather than the storage device 606 or other location), and only operates on data in one or more internal registers or internal cache memories or in the memory 604 (rather than the storage device 606 or other location).

[0062] One or more memory buses (which may each include an address bus and a data bus) may couple the processor 602 to the memory 604. As described below, the bus 612 may include one or more memory buses. In some embodiments, one or more memory management units (MMUs) reside between the processor 602 and the memory 604 and facilitate access to the memory 604 requested by the processor 602. In some embodiments, the memory 604 includes a random access memory (RAM). Where appropriate, the RAM is a volatile memory. Where appropriate, the RAM may be a dynamic RAM (DRAM) or a static RAM (SRAM). In addition, where appropriate, the RAM may be a single-port RAM or a multi-port RAM. The present disclosure contemplates any suitable RAM. Where appropriate, the memory 604 includes one or more memories 604. Although the present disclosure describes and illustrates specific memories, the present disclosure contemplates any suitable memory.

[0063] In some embodiments, storage device 606 includes a large capacity storage device for data or instructions. As an example and not limitation, storage device 606 may include a hard disk drive (HDD), a floppy disk drive (FDD), a flash memory, an optical disk, a magneto-optical disk, a tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, storage device 606 may include removable media or non-removable (or fixed) media. Where appropriate, storage device 606 may be inside or outside computer system 600. In some embodiments, storage device 606 is a non-volatile solid-state memory. In some embodiments, storage device 606 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these ROMs. The present disclosure contemplates mass storage 606 in any suitable physical form. Where appropriate, storage 606 may include one or more storage control units that facilitate communication between processor 602 and storage 606. Where appropriate, storage 606 may include one or more storage devices 606. Although the present disclosure describes and illustrates specific storage devices, the present disclosure contemplates any suitable storage devices.

[0064] In some embodiments, the I / O interface 608 includes the following hardware, software, or both hardware and software: the hardware, software, or both hardware and software provide one or more interfaces for communication between the computer system 600 and one or more I / O devices. Where appropriate, the computer system 600 may include one or more of these I / O devices. One or more of these I / O devices can enable communication between a person and the computer system 600. As an example and not limitation, the I / O device may include a keyboard, a keypad, a microphone, a monitor, a mouse, a printer, a scanner, a speaker, a still camera, a stylus, a tablet computer, a touch screen, a trackball, a camera, another suitable I / O device, or a combination of two or more of these. The I / O device may include one or more sensors. The present disclosure contemplates any suitable I / O device and any suitable I / O interface 608 for the I / O device. Where appropriate, the I / O interface 608 may include one or more device drivers or software drivers that enable the processor 602 to drive one or more of these I / O devices. Where appropriate, I / O interface 608 may include one or more I / O interfaces 608. Although this disclosure describes and illustrates a particular I / O interface, this disclosure contemplates any suitable I / O interface.

[0065] In some embodiments, the communication interface 610 includes hardware, software, or both that provide one or more interfaces for communication (e.g., packet-based communication) between the computer system 600 and one or more other computer systems 600 or with one or more networks. By way of example and not limitation, the communication interface 610 may include a network interface controller (NIC) or a network adapter for communicating with an Ethernet or other wired-based network, or may include a wireless NIC (WNIC) or a wireless adapter for communicating with a wireless network (e.g., a Wi-Fi network). The present disclosure contemplates any suitable network and any suitable communication interface 610 for any suitable network.

[0066] As an example and not limitation, the computer system 600 can communicate with the following network: an ad hoc network, a personal area network (PAN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), or one or more parts of the Internet, or a combination of two or more of these networks. One or more parts of one or more of these networks can be wired or wireless. As an example, the computer system 600 can communicate with the following network: a wireless PAN (wireless PAN, WPAN) (e.g., Bluetooth WPAN), a WI-FI network, a WI-MAX network, a cellular phone network (e.g., a global system for mobile communications (GlobalSystem for Mobile Communications, GSM) network), or other suitable wireless networks, or a combination of two or more of these networks. Where appropriate, the computer system 600 may include any suitable communication interface 610 for any of these networks. Where appropriate, the communication interface 610 may include one or more communication interfaces 610. Although the present disclosure describes and illustrates a specific communication interface, the present disclosure contemplates any suitable communication interface.

[0067] In some embodiments, bus 612 includes hardware, software, or both that couples multiple components of computer system 600 to each other. By way of example and not limitation, bus 612 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a front-side bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an INFINIBAND interconnect, a low-pin-count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a serial advanced technology attachment (SATA) bus, a Video Electronics Standards Association local (VLB) bus, or another suitable bus, or a combination of two or more of these busses. Where appropriate, bus 612 may include one or more buses 612. Although this disclosure describes and illustrates a particular bus, this disclosure contemplates any suitable bus or interconnect.

[0068] In this document, where appropriate, one or more computer-readable non-transitory storage media may include: one or more semiconductor-based integrated circuits (integrated circuits, ICs) or other ICs (e.g., field programmable gate arrays (FPGAs) or application-specific ICs (application-specific ICs, ASICs)), hard disk drives (hard disk drives, HDDs), hybrid hard disk drives (hybrid hard drives, HHDs), optical disks, optical disc drives (optical disc drives, ODDs), magneto-optical disks, magneto-optical drives, floppy disks, floppy disk drives (floppy disk drives, FDDs), magnetic tapes, solid-state drives (solid-state drives, SSDs), RAM drives, secure digital (SECURE DIGITAL) cards or secure digital drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these. Where appropriate, the computer-readable non-transitory storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0069] In some embodiments, a non-transitory computer-readable medium may include instructions that, when executed by one or more processors of an electronic device, cause the electronic device to perform one or more steps of one or more methods described or illustrated herein.

[0070] As used herein, "or" is inclusive and not exclusive, unless expressly indicated otherwise or indicated by context. Thus, as used herein, "A or B" means "A, B, or both," unless expressly indicated otherwise or indicated by context. Furthermore, "and" is both joint and several, unless expressly indicated otherwise or indicated by context. Thus, as used herein, "A and B" means "A and B, jointly or severally," unless expressly indicated otherwise or indicated by context.

[0071] The scope of the present disclosure covers all changes, substitutions, variations, transformations and modifications to the example embodiments described or shown herein that will be understood by those of ordinary skill in the art. The scope of the present disclosure is not limited to the example embodiments described or shown herein. In addition, although the present disclosure describes and shows the various embodiments herein as including specific components, elements, features, functions, operations, or steps, any embodiment in these embodiments may include any combination or arrangement of any of the components, elements, features, functions, operations, or steps described or shown anywhere herein that will be understood by those of ordinary skill in the art. In addition, references in the appended claims to a device or system that is suitable for, arranged to, capable of, configured to, enabled to, capable of operating, or operable to perform a specific function, or a component in a device or system, include the device, system, component, regardless of whether the device, system, component, or the specific function is activated, turned on, or unlocked, as long as the device, system, or component is so adapted to, arranged to, capable of, configured to, enabled to, capable of operating, or operable. Additionally, while this disclosure may describe or illustrate certain embodiments as providing particular advantages, certain embodiments may not provide these advantages, or may provide some or all of these advantages.

Claims

1. A method for generating a realistic avatar of a user, the method comprising, by an electronic device: The frame sequence is displayed to a user of the electronic device via at least one display of the electronic device, wherein The frame sequence includes a first frame, a second frame and a third frame, the first frame includes a first color component, the second frame includes a second color component, and the third frame includes a third color component; While displaying the sequence of frames to the user, capturing multiple images of the user via one or more cameras of the electronic device; determining, for each of the plurality of images of the user, a visible light wavelength of each of the first color component, the second color component, and the third color component; as well as A realistic avatar of the user is generated based on the visible light wavelength of each of the first color component, the second color component, and the third color component.

2. The method according to claim 1, wherein: The first color component includes a red component; The second color component comprises a green component; and The third color component includes a blue component.

3. The method according to claim 2, wherein: The first frame includes only the red component; The second frame includes only the green component; and The third frame includes only the blue component.

4. The method according to claim 1, wherein: Determining the wavelength of visible light for each of the plurality of images of the user includes determining a color of each of a plurality of features of the user.

5. The method according to claim 4, wherein: The plurality of characteristics of the user include one or more of: the user's eye color, the user's hair color, the user's skin complexion, the user's facial hair color, or the user's cheek complexion.

6. The method according to claim 1, wherein: Generating the realistic avatar of the user includes combining the multiple images of the user based on the visible light wavelength of each of the first color component, the second color component, and the third color component.

7. The method according to claim 1, further comprising: Instead of displaying the frame sequence to the user, a light pulse sequence is projected onto the user through multiple light sources of the electronic device, wherein the light pulse sequence includes a first light pulse, a second light pulse, and a third light pulse frame, the first light pulse only includes the first color component, the second light pulse only includes the second color component, and the third light pulse frame only includes the third color component.

8. The method according to claim 7, wherein: The plurality of light sources include a plurality of light emitting diodes (LEDs).

9. The method according to claim 1, wherein: Capturing the plurality of images of the user through the one or more cameras further comprises: capturing a first image of the plurality of images of the user while displaying the first frame including the first color component; while displaying the second frame including the second color component, capturing a second image of the plurality of images of the user; and While displaying the third frame including the third color component, a third image of the plurality of images of the user is captured.

10. The method according to claim 1, further comprising: A heart rate of the user is determined based on the visible light wavelength of each of the first color component, the second color component, and the third color component.

11. The method according to claim 1, further comprising: The blood oxygen level of the user is determined based on the visible light wavelength of each of the first color component, the second color component, and the third color component.

12. An electronic device comprising: at least one display; One or more cameras; one or more non-transitory computer-readable storage media comprising instructions; and one or more processors coupled to the one or more non-transitory computer-readable storage media and the one or more cameras, the one or more processors configured to execute the instructions to: Displaying a frame sequence to a user of the electronic device via the at least one display, wherein the frame sequence includes a first frame, a second frame, and a third frame, the first frame including a first color component, the second frame including a second color component, and the third frame including a third color component; capturing a plurality of images of the user via the one or more cameras while displaying the sequence of frames to the user; determining, for each of the plurality of images of the user, a visible light wavelength of each of the first color component, the second color component, and the third color component; and A realistic avatar of the user is generated based on the visible light wavelength of each of the first color component, the second color component, and the third color component.

13. The electronic device according to claim 12, wherein: The first color component includes a red component; The second color component comprises a green component; and The third color component includes a blue component.

14. The electronic device according to claim 13, wherein: The first frame includes only a red component; The second frame includes only a green component; and The third frame includes only the blue component.

15. The electronic device according to claim 12, wherein: The instructions for determining the wavelength of visible light for each of the plurality of images of the user also include instructions for determining a color for each of a plurality of features of the user.

16. The electronic device according to claim 15, wherein: The plurality of characteristics of the user include one or more of: the user's eye color, the user's hair color, the user's skin complexion, the user's facial hair color, or the user's cheek complexion.

17. The electronic device according to claim 12, wherein: The instructions for generating a realistic avatar of the user also include instructions for combining the multiple images of the user based on the visible light wavelength of each of the first color component, the second color component, and the third color component.

18. The electronic device according to claim 12, wherein: The instructions also include instructions for: Instead of displaying the frame sequence to the user, a light pulse sequence is projected onto the user through multiple light sources of the electronic device, wherein the light pulse sequence includes a first light pulse, a second light pulse, and a third light pulse frame, the first light pulse only includes the first color component, the second light pulse only includes the second color component, and the third light pulse frame only includes the third color component.

19. The electronic device according to claim 12, wherein: The instructions for capturing the plurality of images of the user via the one or more cameras also include instructions for: capturing a first image of the plurality of images of the user while displaying the first frame including the first color component; capturing a second image of the plurality of images of the user while displaying the second frame including the second color component; as well as While displaying the third frame including the third color component, a third image of the plurality of images of the user is captured.

20. A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of an electronic device, cause the electronic device to: The frame sequence is displayed to a user of the electronic device via at least one display of the electronic device, wherein The frame sequence includes a first frame, a second frame and a third frame, the first frame includes a first color component, the second frame includes a second color component, and the third frame includes a third color component; While displaying the sequence of frames to the user, capturing multiple images of the user via one or more cameras of the electronic device; determining, for each of the plurality of images of the user, a visible light wavelength of each of the first color component, the second color component, and the third color component; as well as A realistic avatar of the user is generated based on the visible light wavelength of each of the first color component, the second color component, and the third color component.

Images