Aligning user representations

Abstract

The invention relates to aligning user representations. Various implementations disclosed herein include devices, systems, and methods of generating a combined 3D representation of a user from alignment based on 3D reference points. For example, a process may include obtaining a predetermined three-dimensional (3D) representation associated with a 3D reference point defined relative to a skeletal representation of the user. The process may also include obtaining a sequence of frame-specific 3D representations corresponding to a plurality of times within a time period, each of the frame-specific 3D representations representing a second portion of the user at a respective one of the plurality of times within the time period. The process may also include generating a combined 3D representation of the user, the combined 3D representation generated by combining the predetermined 3D representation with a corresponding frame-specific 3D representation based on an alignment, the alignment based on the 3D reference point.

Description

[0001] This application is a divisional application of the Chinese patent application with the application date of September 22, 2023, the national application number 202311230102.8, and the invention name “Aligned User Representation”. Technical Field

[0002] The present disclosure relates generally to electronic devices, and particularly to systems, methods, and devices for representing users in computer-generated content. Background Art

[0003] Existing techniques may not accurately or truthfully present a current (e.g., real-time) representation of the appearance of a user of an electronic device. For example, a device may provide an avatar representation of a user based on images of the user's face obtained minutes, hours, days, or even years ago. Such a representation may not accurately represent the user's current (e.g., real-time) appearance, such as not showing the user's avatar smiling when the user is smiling or not showing the user's current beard. Therefore, it may be desirable to provide an apparatus that can effectively provide a more accurate, truthful, and / or current representation of a user. Summary of the invention

[0004] Various implementations disclosed herein include devices, systems, and methods for generating a set of values ​​that represent the three-dimensional (3D) shape and appearance of a user's face at a point in time for use in generating a user representation (e.g., an avatar). In some implementations, the set of values ​​may be defined relative to a surface having a non-planar shape (e.g., a curved planar shape). The set of values ​​may include depth values ​​that define the depth of a portion of a face relative to multiple points on this surface (e.g., points in a grid on a partially cylindrical surface). For example, a depth value for a point may define that a portion of the face is at a depth D1 behind the position of the point on the surface, for example, at a depth D1 along an orthogonal ray starting at the point. The techniques described herein use depth values ​​that are different from depth values ​​in existing RGBDA images (e.g., red-green-blue-depth alpha images) because existing RGBDA images define content depth relative to a single camera position, while the techniques described herein define depth relative to multiple points on a surface having a planar shape (e.g., a curved planar shape).

[0005] Several advantages may be achieved using this relatively simple set of values ​​having depth values ​​defined relative to multiple points on a surface. This set of values ​​may require less computation and bandwidth than using a 3D mesh or 3D point cloud, while achieving a more accurate user representation than an RGBDA image. Furthermore, this set of values ​​may be formatted / packaged in a manner similar to existing formats (e.g., RGBDA images), which may enable more efficient integration with systems based on such formats.

[0006] Various specific implementations disclosed herein include devices, systems, and methods for generating a 3D representation of a user for each of a plurality of moments by combining the same predetermined 3D data of a first portion of the user with frame-specific 3D data of a second portion of the user captured at a plurality of moments. The predetermined 3D data may be a mesh of the user's upper body and head generated from registration data (e.g., one-time pixel alignment implicit function (PIFu) data). The predetermined 3D data (such as PIFu data) may include a highly effective implicit representation that locally aligns the pixels of a 2D image with the global background of its corresponding 3D object. The frame-specific data may represent the user's face at each of a plurality of time points, for example, a live sequence of frame-specific 3D representation data, such as a set of values ​​representing the 3D shape and appearance of the user's face at a certain time point as described herein. The 3D data from these two different sources (e.g., PIFu data and frame-specific 3D data) may be combined for each moment by spatially aligning the data using 3D reference points associated with the two data sets (e.g., points defined relative to a skeleton representation). A 3D representation of the user at multiple times may be generated on a viewing device that combines the data and uses the combined data to present the view, such as during a live communication (eg, copresence) session.

[0007] Various specific implementations disclosed herein include devices, systems, and methods for adjusting the color of a first 3D representation of a first part of a user (e.g., head, hair, upper body, clothing, etc.) determined via a first technique (e.g., PIFu). The color is adjusted using a sample 3D representation of a second part of the user (e.g., face / hair in a sample previously captured during a registration process) determined via a second different technique. This adjustment provides better color matching in a combined 3D representation generated by combining the first 3D representation with one or more second 3D representations (e.g., frame-specific 3D representations) generated via a second technique (e.g., a live sequence of frame-specific 3D representation data, such as the set of values ​​representing the 3D shape and appearance of the user's face at a certain point in time as described herein). For example, the first representation can be based on PIFu and can be adjusted so that the neck color better matches the facial color of the frame-specific 3D face. The color can be adjusted using a transformation (e.g., determined using a Monge-Kanorovich color transfer technique, etc.). Determining such a transformation can be facilitated by reconstructing the first representation into the same space as the sample 3D representation (e.g., reconstructing the PIFu representation in UV space). This enables a more accurate transformation based on a comparison of the colors of corresponding user parts.

[0008] Various implementations disclosed herein include devices, systems, and methods for adjusting the color of a first 3D representation of a first part of a user (e.g., a hand). The color is adjusted using a sample 3D representation of a second part of the user (e.g., face / hair). In some implementations, there is no overlap between the first 3D representation and the sample 3D representation of the second part of the user (e.g., one representation shows a face and the other shows a hand). Color adjustment accuracy can be improved by filtering out non-skin data (e.g., hair, clothing, etc.). A transform can be used to adjust the color (e.g., determined using a Monge-Kanorovich color transfer technique). In some implementations, shadows in the sample representation can produce a color adjustment that is too dark, which can be addressed by taking only a subset of the data (e.g., the lightest 25% of the data).

[0009] Generally speaking, one innovative aspect of the subject matter described in this specification can be embodied in a method comprising the following actions: at a processor of a device, obtaining a predetermined three-dimensional (3D) representation of at least a first portion of a user, wherein the predetermined 3D representation is associated with a 3D reference point defined relative to a skeletal representation of the user; obtaining a sequence of frame-specific 3D representations corresponding to multiple moments in time within a time period, each of the frame-specific 3D representations representing a second portion of the user at a corresponding moment in the multiple moments in time within the time period, and each of the frame-specific 3D representations is associated with the 3D reference point; and generating a combined 3D representation of the user for the multiple moments in time within the time period, wherein each of the combined 3D representations is generated by combining the predetermined 3D representation with the corresponding frame-specific 3D representation based on an alignment, wherein the alignment is based on the 3D reference point.

[0010] These and other embodiments can optionally include one or more of the following features.

[0011] In some aspects, the second portion of the user is a sub-portion of the first portion. In some aspects, the first portion of the user includes a facial portion of the user and an additional portion, and the second portion of the user includes the facial portion without the additional portion of the user. In some aspects, the 3D reference point is associated with a 3D position of an atlas joint of a skeletal representation of the user.

[0012] In some aspects, the 3D reference point is associated with a center of the user's eye, with the center defined at a location offset from the atlas joint.

[0013] In some aspects, combining the predetermined 3D representation with the corresponding frame-specific 3D representation includes adjusting a sub-portion of the predetermined 3D representation.

[0014] In some aspects, adjusting the sub-portion of the predetermined 3D representation includes adjusting positions of vertices of the predetermined 3D representation and applying a texture based on each of the frame-specific 3D representations.

[0015] In some aspects, each of the frame-specific 3D representations is defined using a height value specified within a height field positionally associated with the 3D reference point.

[0016] In some aspects, vertices of the predetermined 3D representation are mapped to locations on the height field and associated with height values, wherein the vertices are mapped based on the 3D reference point.

[0017] In some aspects, combining the predetermined 3D representation with each of the frame-specific 3D representations includes adjusting at least some of the height values ​​associated with the vertex based on the height value specified for the respective frame-specific 3D representation.

[0018] In some aspects, adjusting the vertex includes blending at least some of the height values ​​associated with the vertex with the height value specified for the corresponding frame-specific 3D representation using a specified alpha value. In some aspects, adjusting the vertex includes blending at least some of the texture values ​​associated with the vertex with texture values ​​specified for the corresponding frame-specific representation using the specified alpha value. In some aspects, the first portion represented by the predetermined 3D representation represents an upper torso and a head of the user.

[0019] In some aspects, the predetermined 3D representation is generated using a pixel aligned implicit function (PIFu) technique that locally aligns pixels of a 2D registration image with a global background to form the predetermined 3D representation.

[0020] In some aspects, each of the frame-specific 3D representations is generated based on sensor data from an inward-facing or downward-facing camera and based on enrollment data, wherein the enrollment data includes an image of the second portion of the user exhibiting a plurality of different facial expressions.

[0021] In some aspects, each of the frame-specific 3D representations includes a configuration of a skeletal representation of the user at a corresponding moment in the plurality of moments in the time period, wherein the 3D reference point is defined based on a node of the skeletal representation.

[0022] In some aspects, the actions also include presenting a view of the combined 3D representation. In some aspects, the presenting occurs during a communication session in which the second device captures sensor data and provides a sequence of frame-specific 3D representations corresponding to the plurality of moments in the time period based on the sensor data.

[0023] In some aspects, the second device provides the predetermined 3D representation of the user during the communication session. In some aspects, the second device generates the combined 3D representation of the user and provides the combined 3D representation to the device during the communication session.

[0024] In some aspects, the views of the combined 3D representation are displayed on the device in real time relative to the plurality of moments in the time period. In some aspects, the device is a head mounted device (HMD).

[0025] These and other embodiments can optionally include one or more of the following features.

[0026] According to some specific implementations, a non-transitory computer-readable storage medium stores instructions, which are computer-executable to perform or cause the performance of any method described herein. According to some specific implementations, a device includes one or more processors, a non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the performance of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0028] Figure 1 Devices for obtaining sensor data from a user according to some implementations are shown.

[0029] Figure 2 An example of a three-dimensional (3D) representation of at least a portion of a user is shown, according to some implementations.

[0030] Figure 3A and Figure 3B An example of a surface of a two-dimensional (2D) manifold provided as a visualization of a height field representation of a face is shown according to some implementations.

[0031] Figure 4 An example of updating a portion of a user's facial representation is shown according to some implementations.

[0032] Figure 5A and Figure 5B Examples of 3D reference points defined relative to a skeletal representation of a user are shown according to some implementations.

[0033] Figure 6An example is shown in which a predetermined 3D representation and a parameterized mesh are combined to generate a representation of a portion of a user based on 3D reference points, according to some implementations.

[0034] Figure 7 Examples of generating and displaying portions of a facial representation of a user according to some implementations are shown.

[0035] Figure 8 A flow diagram of a system that can generate a combined representation of a user based on predetermined representation data and frame-specific representation data according to some implementations is shown.

[0036] Fig. 9 Depicted are exemplary electronic devices operating in different physical environments during a communication session between a first user at a first device and a second user at a second device, and a view of a 3D representation of the second user of the first device, according to some implementations.

[0037] Fig.10 is a flowchart representation of a method for generating a combined 3D representation of a user for multiple moments in time within a time period based on alignment based on 3D reference points, according to some implementations.

[0038] Fig.11 An example of adjusting one or more colors of a first 3D representation based on colors in the first 3D representation and colors in a sample 3D representation is shown.

[0039] Fig.12 According to some specific implementations, Fig.10 An example of adjusting one or more colors of a 3D representation to generate a 3D representation of a user.

[0040] Fig.13 is a flowchart representation of a method for adjusting one or more colors in a 3D representation to generate a combined 3D representation, according to some implementations.

[0041] Fig.14 is a flowchart representation of a method for adjusting one or more colors in a 3D representation based on filtering an identified portion to generate a combined 3D representation, according to some implementations.

[0042] Fig.15 is a block diagram illustrating device components of an exemplary device according to some implementations.

[0043] Fig.16 is a block diagram of an exemplary head mounted device (HMD) according to some implementations.

[0044] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. DETAILED DESCRIPTION

[0045] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the exemplary implementations described herein.

[0046] Figure 1 An exemplary environment 100 of a real-world environment 105 (e.g., a room) is shown that includes a device 10 having a display 15. In some implementations, the device 10 displays content 20 to a user 25. For example, the content 20 can be a button, a user interface icon, a text box, a graphic, an avatar of the user or another user, etc. In some implementations, the content 20 can occupy the entire display area of ​​the display 15.

[0047] The device 10 obtains image data, motion data, and / or physiological data (e.g., pupil data, facial feature data, etc.) from the user 25 via multiple sensors (e.g., sensors 35a, 35b, and 35c). For example, the device 10 obtains eye gaze characteristic data 40b via sensor 35b, upper facial feature characteristic data 40a via sensor 35a, and lower facial feature characteristic data 40c via sensor 35c.

[0048] Although this example and other examples discussed herein show a single device 10 in a real-world environment 105, the techniques disclosed herein are applicable to multiple devices and other real-world environments. For example, the functionality of device 10 may be performed by multiple devices, with sensors 35a, 35b, and 35c located on each respective device, or dispersed among them in any combination.

[0049] In some implementations, the plurality of sensors (e.g., sensors 35a, 35b, and 35c) may include any number of sensors that collect data related to the appearance of user 25. For example, when a head mounted device (HMD) is worn, one sensor (e.g., a camera within the HMD) may collect pupil data for eye tracking, and one sensor on a separate device (e.g., one camera, such as a wide angle view) may be able to capture all of the user's facial feature data. Alternatively, if device 10 is an HMD, a separate device may not be necessary. For example, if device 10 is an HMD, in one implementation, sensor 35b may be located within the HMD to capture pupil data (e.g., eye gaze characteristic data 40b), and additional sensors (e.g., sensors 35a and 35c) may be located on the HMD but on an outer surface of the HMD facing the user's head / face to capture facial feature data (e.g., capturing upper facial feature characteristic data 40a via sensor 35a, and capturing lower facial feature characteristic data 40c via sensor 35c). For example, user 25 may hold the HMD in front of his or her face, with the HMD facing the user, and an external camera on the HMD may capture an image of the user's face for enrollment.

[0050] In some specific implementations, such as Figure 1 As shown, device 10 is a handheld electronic device (e.g., a smartphone or tablet). In some implementations, device 10 is a laptop or desktop computer. In some implementations, device 10 has a trackpad, and in some implementations, device 10 has a touch-sensitive display (also referred to as a "touch screen" or "touch screen display"). In some implementations, device 10 is a wearable device, such as an HMD.

[0051] In some implementations, device 10 includes an eye tracking system for detecting eye position and eye movement via eye gaze characteristic data 40b. For example, the eye tracking system may include one or more infrared (IR) light emitting diodes (LEDs), an eye tracking camera (e.g., a near infrared (NIR) camera), and an illumination source (e.g., a NIR light source) that emits light (e.g., NIR light) toward the eyes of user 25. In addition, the illumination source of device 10 may emit NIR light to illuminate the eyes of user 25, and the NIR camera may capture images of the eyes of user 25. In some implementations, the images captured by the eye tracking system may be analyzed to detect the position and movement of the eyes of user 25, or to detect other information about the eyes such as color, shape, state (e.g., wide open, squinting, etc.), pupil dilation, or pupil diameter. In addition, the gaze point estimated from the eye tracking image may enable gaze-based interaction with content shown on a near-eye display of device 10.

[0052] In some implementations, the device 10 has a graphical user interface (GUI), one or more processors, a memory, and one or more modules, programs, or instruction sets for performing multiple functions stored in the memory. In some implementations, the user 25 interacts with the GUI through finger contacts and gestures on the touch-sensitive surface. In some implementations, these functions include image editing, drawing, presentation, word processing, web page creation, disk editing, spreadsheet production, playing games, making and receiving calls, video conferencing, sending and receiving emails, instant messaging, fitness support, digital photography, digital video recording, web browsing, digital music playback, and / or digital video playback. Executable instructions for performing these functions may be included in a computer-readable storage medium or other computer program product configured to be executed by one or more processors.

[0053] In some implementations, the device 10 employs various physiological sensors, detection or measurement systems. The detected physiological data may include, but is not limited to: electroencephalogram (EEG), electrocardiogram (ECG), electromyogram (EMG), functional near infrared spectroscopy signal (fNIRS), blood pressure, skin conductance, or pupil response. In addition, the device 10 may detect multiple forms of physiological data simultaneously in order to benefit from the synchronous collection of physiological data. In addition, in some implementations, the physiological data represents involuntary data, that is, responses that are not under conscious control. For example, a pupil response may represent an involuntary movement.

[0054] In some implementations, one or both eyes 45 of user 25 (including one or both pupils 50 of user 25) present physiological data (e.g., eye gaze characteristic data 40b) in the form of pupil responses. The pupil responses of user 25 result in changes in the size or diameter of pupil 50 via the optic nerve and the oculomotor cranial nerve. For example, the pupil responses may include a constriction response (miosis), i.e., a narrowing of the pupil, or a dilation response (mydriasis), i.e., a widening of the pupil. In some implementations, device 10 may detect a pattern of physiological data representing a time-varying pupil diameter.

[0055] The user data (eg, upper facial feature characteristic data 40a, lower facial feature characteristic data 40c, and eye gaze characteristic data 40b) may change over time and device 10 may use the user data to generate and / or provide a representation of the user.

[0056] In some implementations, the user data (e.g., upper facial feature characteristic data 40a and lower facial feature characteristic data 40c) includes texture data of facial features, such as eyebrow movements, jaw movements, nose movements, cheek movements, etc. For example, when a person (e.g., user 25) smiles, the upper facial features and lower facial features (e.g., upper facial feature characteristic data 40a and lower facial feature characteristic data 40c) may include a number of muscle movements that can be repeated by a representation of the user (e.g., an avatar) based on captured data from sensor 35.

[0057] According to some specific implementations, an electronic device (e.g., device 10) may generate an extended reality (XR) environment during a communication session and present the XR environment to one or more users. Compared to a physical environment that people can sense and / or interact with without the help of an electronic device, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via an electronic device. For example, an XR environment may include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, and the like. In the case of an XR system, a subset of a person's physical movements or a representation thereof is tracked, and in response, one or more features of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one physical law. As one example, the XR system may detect head movement, and in response, adjust the graphics content and sound field presented to the person in a manner similar to how such views and sounds change in a physical environment. As another example, the XR system may detect movement of an electronic device (e.g., a mobile phone, tablet, laptop, etc.) that presents the XR environment, and in response, adjust the graphics content and sound field presented to the person in a manner similar to how such views and sounds would change in a physical environment. In some cases (eg, for accessibility reasons), the XR system may adjust features of graphical content in the XR environment in response to representations of physical motion (eg, voice commands).

[0058] There are many different types of electronic systems that enable people to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, heads-up displays (HUDs), vehicle windshields with integrated display capabilities, windows with integrated display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headphones / earpieces, speaker arrays, input systems (e.g., wearable or handheld controllers with or without tactile feedback), smartphones, tablets, and desktop / laptop computers. A head-mounted system may have an integrated opaque display and one or more speakers. Alternatively, a head-mounted system may be configured to accept an external opaque display (e.g., a smartphone). A head-mounted system may incorporate one or more imaging sensors for capturing images or video of a physical environment, and / or one or more microphones for capturing audio of a physical environment. Instead of an opaque display, a head-mounted system may have a transparent or translucent display. A transparent or translucent display may have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light source, or any combination of these technologies. The medium can be an optical waveguide, a hologram medium, an optical combiner, an optical reflector, or any combination thereof. In some implementations, a transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection technology that projects a graphic image onto a person's retina. The projection system can also be configured to project a virtual object into a physical environment, such as a hologram or onto a physical surface.

[0059] Figure 2 An example of a 3D representation 200 of at least a portion of a user according to some implementations is shown. For example, 3D representation 200 may represent a portion of user 25 after being scanned by one or more sensors of device 10 (e.g., during a registration process). In an exemplary implementation, 3D representation 200 may be generated using a pixel aligned implicit function (PIFu) technique that locally aligns pixels of a 2D registration image with a global background to form a 3D representation 200 (also referred to as a PIFu mesh). 3D representation 200 includes a plurality of vertices and polygons that may be determined in a registration process based on image data (such as RGB data and depth data). For example, as shown in extended area 202, vertex 204 is circled as a point between two or more polygons that are part of a 3D PIFu mesh.

[0060] Figure 3A and Figure 3BAn example of a surface of a two-dimensional (2D) manifold provided as a visualization of a height field representation of a face according to some implementations is shown. A "height field representation" may also be referred to herein as a parameterized mesh. Specifically, Figure 3A An exemplary environment 300A of a height field representation of a face is shown, which combines three different types of data to provide a height field representation of a face as shown by a face representation grid 308. These different types of data include RGB data 302, alpha data 304, and depth data 306. For each frame of image data obtained, the techniques described herein determine the RGB data 302, alpha data 304, and depth data 306, and provide such unconventional "RGBDA" data as shown by the face representation grid 308. For example, the face representation grid 308 provides a mapping to locations on a 2D manifold based on ray origins and ray directions. The face representation grid 308, or ray grid, provides depth data to generate and / or update a 3D reconstruction of the face (e.g., as a user moves his or her face, such as when speaking in a communication session). Figure 3B and Figure 4 The application of the application face representation grid 308 is further described.

[0061] Figure 3B An exemplary environment 300B of a surface of a two-dimensional manifold provided as a visualization of a user's facial representation according to some implementations is shown. Specifically, environment 300B shows a user (e.g., Figure 1 The parametric image 320 shows a facial representation of the user 25). The parametric image 320 shows Figure 3A308. For example, a frame-specific representation instruction set may obtain live image data of a user's face (e.g., image 310) and parameterize different points on the face based on a surface of a shape (e.g., curved planar shape 315). In other words, the frame-specific representation instruction set may generate a set of values ​​that represent the 3D shape and appearance of the user's face at a point in time for generating a user representation (e.g., an avatar). In some implementations, using a surface with a non-planar shape (e.g., curved planar shape 315) provides less distortion than using a flat / planar surface or using a single point. The set of values ​​includes depth values ​​(e.g., vector arrows pointing to the user's representation of the face to represent depth values, similar to a height field or height map or parameterized grid) that define the depth of a portion of the face relative to a plurality of points on the surface (e.g., points in a grid on a partially cylindrical surface, such as point array 325). The parameterized values ​​may include fixed parameters such as ray position, endpoints, direction, etc., and the parameterized values ​​may include varying parameters such as depth, color, texture, opacity, etc. that are updated using live image data. For example, as shown in extended area 330 of the user's nose, a depth value for a point (e.g., point 332 at the tip of the user's nose) may define a portion of the face at a depth D1 behind the position of the point on the surface, e.g., at depth D1 along an orthogonal ray starting at the point and orthogonal to the point.

[0062] The techniques described herein use depth values ​​that are different from those in existing RGBDA images (e.g., red-green-blue-depth alpha images) because existing RGBDA images define depth of content relative to a single camera position / point, whereas the techniques described herein define depth as portions of a face relative to multiple points on a surface having a planar shape (e.g., a curved planar shape). The curved surface (such as the curved planar shape 315) implemented for the parameterized image 320 is used to reduce distortion of the user representation (e.g., an avatar) at areas of the user representation that are not visible from the flat projection surface. In some implementations, the projection surface having a planar shape can be bent and shaped in any manner to reduce distortion in desired areas based on the application of the parameterization. Using different bent / curved shapes allows the user representation to be clearly presented from more viewpoints.

[0063] Figure 3BPoints of a surface (e.g., a 2D manifold surface) are shown as being spaced at regular intervals along vertical and horizontal lines on the surface (e.g., uniformly spaced vector arrows pointing to the face of a representation of a user). In some implementations, the points may be unevenly distributed across the 2D manifold surface, such as being irregularly spaced along vertical and horizontal grid lines around the surface, but may be focused on specific areas of the user's face. For example, some areas where there may be more detail / motion in the facial structure may have more points, and some points may have fewer points in areas where there may be less detail / motion, such as the forehead (less detail) and the nose (not much motion). In some implementations, when generating a representation of a user during a communication session (e.g., generating an avatar), the techniques described herein may selectively focus more on areas of the eyes and mouth that will likely move more during a conversation, thereby producing a more accurate representation of the person during the communication session. For example, the techniques described herein may present updates to the user's representation around the mouth and eyes at a faster frame rate than other parts of the face that do not move as much during a conversation (e.g., forehead, ears, etc.).

[0064] Figure 4 An exemplary environment 400 for updating a portion of a user's facial representation according to some implementations is shown. Specifically, Figure 4 An application is shown that utilizes a facial representation mesh 410 (e.g., facial representation mesh 308) and updated depth data 420 and maps the updated facial representation mesh 410 to the user's face as shown in a mapped image 430. The updated mapped image 430 can then be utilized to update the user's representation 440 in real time (e.g., as additional frames of RGBDA data are obtained). In an exemplary implementation, the mapped data is based on 3D reference points defined relative to a skeletal representation, such as based on a defined atlas joint of the user, as described herein with reference to FIG. Figure 5A , Figure 5B and Figure 6 Further described.

[0065] Figure 5A and Figure 5B Examples of 3D reference points defined relative to a skeletal representation of a user are shown according to some implementations. Figure 5A and Figure 5B The user is shown in different head positions and orientations (e.g. Figure 1 25) to show different bone positions. Specifically, Figure 5A and Figure 5BEach shows a 3D reference point 510 determined based on an offset 515 from a determined atlas joint 520. The 3D reference point 510 can be used to track the user's kinematic movement by tracking skeletal movement relative to the atlas joint (e.g., providing tracking on an x-axis aligned with the ear canal and on a z-axis relative to the Frankfurt plane). In some embodiments, the 3D reference point 510 is associated with the center of the user's eyes, which is defined at a location offset from the atlas joint. For example, during the registration process, an offset can be determined to provide a mid-pupil origin for a parameterized mesh (e.g., a height field representation). In some embodiments, the 3D reference point can be a point centered between the user's eyes based on the skeletal atlas joint and user-specific head shape characteristics (e.g., an offset position of the 3D reference point 510 associated with the location of the atlas joint 520 determined based on the offset 515 in the figure). Reference herein Figure 6 An example of combining a predetermined 3D representation and a parameterized mesh using 3D reference points 510 to generate a representation of a portion of a user is further described.

[0066] Figure 6An exemplary environment 600 is shown in which a predetermined 3D representation and a parameterized mesh are combined to generate a representation of a portion of a user based on 3D reference points according to some implementations. In an exemplary implementation, at step 610, a predetermined 3D representation 612 (e.g., 3D representation 200) is obtained (e.g., from a registration process), the predetermined 3D representation including the location of a 3D reference point 602 (e.g., 3D reference point 510 associated with the center of the user's eye, the center being defined at a location offset from the atlas joint to track skeletal motion). Then at step 620, a frame of a parameterized mesh 622 is obtained, and a depth matching process associated with the predetermined 3D representation 612 has been initiated. For example, facial points of the parameterized mesh 622 (e.g., a PIFu mesh) are projected outward to find corresponding points on the predetermined 3D representation 612 (e.g., a curved projection plane). The parameterized mesh 622 also includes the location of a 3D reference point 624 (e.g., a 3D reference point 510 associated with the center of the user's eye, the center being defined at a location offset from the atlas joint to track skeletal motion), which is used to initialize a mapping between the predetermined 3D representation 612 and the parameterized mesh 622. Then, at step 630, the frame of the parameterized mesh 622 is combined on the predetermined 3D representation 612 based on the 3D reference points 602, 624. At step 640, an updated representation 642 of the user is determined based on the mapped combination of the frames of the predetermined 3D representation 612 and the parameterized mesh 622. In some specific implementations where a frame-specific 3D representation is defined using the parameterized mesh 622 (e.g., a height field), the combination of data can be facilitated by mapping vertices of the predetermined 3D representation to locations on the parameterized mesh 622 based on the 3D reference points (e.g., 3D reference points 602, 624). The mapping using 3D reference points enables the frame-specific facial data specified on the parameterized mesh 622 to be directly used to adjust the positions of the vertices of the predetermined 3D representation 612. In some implementations, the positions of the vertices can be adjusted by blending the predetermined vertex positions of the vertices with their frame-specific data vertex positions (e.g., using a specified alpha value). In other words, the predetermined 3D representation vertices can be mapped onto the parameterized mesh 622, the parameterized mesh 622 can be adjusted using real-time data corresponding to the user's head / face, and the adjusted parameterized mesh 622 represents a combined 3D representation of the user that combines the predetermined 3D representation with one of the frame-specific 3D representations.

[0067] In some implementations, combining the predetermined 3D representation 612 with the corresponding frame-specific 3D representation of the parameterized mesh 622 includes adjusting a sub-portion (e.g., a facial portion) of the predetermined 3D representation 612. In some implementations, adjusting the sub-portion of the predetermined 3D representation 612 includes adjusting the predetermined 3D representation 612 (e.g., a PIFu mesh, such as Figure 2The adjustment may be performed based on the positions of vertices of the 3D representation 200 of the user and applying a texture (e.g., parameterized mesh 622) based on each of the frame-specific 3D representations. For example, the adjustment may deform and color a predetermined sub-portion (e.g., face) to correspond to the real-time shape and color of the portion (e.g., face) of the user at each moment in time.

[0068] Figure 7 An example of generating and displaying a portion of a user's facial representation according to some specific implementations is shown. Specifically, Figure 7 An exemplary environment 700 is shown for a process of combining enrollment data 710 (e.g., enrollment image data 712 and generated predetermined 3D representation 714) and live data 720 (e.g., live image data 722 and generated frame-specific 3D representation 724) to generate user representation data 730 (e.g., avatar 735). The enrollment image data 712 shows a user (e.g., Figure 1 25) during the enrollment process. For example, an enrollment avatar may be generated when the system obtains image data (e.g., RGB images) of the user's face as the user provides different facial expressions. For example, the user may be told to "raise your eyebrows," "smile," "frown," etc. in order to provide the system with a series of facial features for the enrollment process. An enrollment avatar preview may be shown to the user as the user provides an enrollment image to obtain a visualization of the status of the enrollment process. In this example, enrollment image data 710 shows an enrollment avatar with four different user expressions, however, more or fewer different expressions may be used to collect sufficient data for the enrollment process. The predetermined 3D representation 714 (e.g., 3D representation 200) includes a plurality of vertices and polygons that may be determined during the enrollment process based on image data (such as RGB data and depth data).

[0069] Live image data 722 represents examples of images of the user captured while using the device (such as during an XR experience) (e.g., when using Figure 1 For example, the live image data 722 represents the live image data when the user moves the Figure 1 The image captured when the device 10 is worn as an HMD. For example, if the device 10 is an HMD, in one implementation, the sensor 35b may be located inside the HMD to capture pupil data (e.g., eye gaze characteristic data 40b), and additional sensors (e.g., sensors 35a and 35c) may be located on the HMD but on the outer surface of the HMD facing the user's head / face to capture facial feature data (e.g., capturing upper facial feature characteristic data 40a via sensor 35a, and capturing lower facial feature characteristic data 40c via sensor 35c). The generated frame-specific 3D representation 724 may be generated based on the obtained live image data 722.

[0070] The user representation data 730 is an exemplary illustration of a user during the avatar display process. For example, avatar 735A (facing sideways) and avatar 735B (facing frontward) are generated based on the acquired registration data 710 and are updated as the system obtains and analyzes real-time image data of live data 720 and updates different values ​​for planar surfaces (e.g., updating the values ​​of vector points of the array for frame-specific 3D representation 724 for each acquired live image data).

[0071] Figure 8 800 is a system flow diagram of an exemplary environment 800 in which a system can generate a combined representation of a user based on predetermined representation data and frame-specific representation data according to some implementations. In some implementations, the system flow of the exemplary environment 800 can be performed on two or more devices (e.g., Figure 1 The image of the exemplary environment 800 may be displayed on a device having a screen for displaying images and / or a screen for viewing stereoscopic images (e.g., Figure 1 The system flow of the exemplary environment 800 is executed on a device 10 such as a head mounted device (HMD). In some implementations, the system flow of the exemplary environment 800 is executed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the system flow of the exemplary environment 800 is executed on a processor executing code stored in a non-transitory computer readable medium (e.g., a memory).

[0072] In some implementations, the system flow of the exemplary environment 800 includes a registration process, a predetermined representation process, a frame-specific representation process, and an avatar display process based on a combined representation. Alternatively, the exemplary environment 800 may include only a predetermined representation process, a frame-specific representation process, and an avatar display process, and obtain registration data from another source (e.g., previously stored registration data). In other words, the registration process may have occurred so that the user's registration data has been provided because the registration process has been completed. In an exemplary implementation, the system flow of the exemplary environment 800 for the avatar display process is executed at a receiving device for displaying an avatar, and obtains data from a sending device, wherein the sending device includes a registration process, a predetermined representation process, and a frame-specific representation process.

[0073] The system flow of the registration process of the exemplary environment 800 is from the physical environment (e.g., Figure 1The user's face is captured by a sensor (e.g., RGB data) of the user's physical environment 105) and generates enrollment data. The enrollment data may include texture, muscle activation, etc. of most, if not all, of the user's face. In some implementations, the enrollment data may be captured when different instructions are provided to the user for capturing different poses of the user's face. For example, the user may be told to "raise your eyebrows," "smile," "frown," etc. in order to provide the system with a series of facial features for the enrollment process.

[0074] The system flow of the avatar display process of the exemplary environment 800 is derived from the physical environment (e.g., Figure 1 Sensors of the physical environment 105 (e.g., RGB, depth, IR, etc.) of the user's face are collected, parameterized data of facial features is determined, enrollment data is obtained and evaluated, and a portion of a facial representation of the user (e.g., a 3D avatar) is generated and displayed based on the parameterized values. For example, the techniques described herein for generating and displaying a portion of a user's facial representation may be implemented on real-time sensor data streamed to an end user (e.g., a 3D avatar overlaid on an image of the physical environment within a CGR environment). In one exemplary implementation, the avatar display process occurs during real-time display (e.g., the avatar is updated in real time as the user makes facial gestures and changes are made to his or her facial features). Alternatively, the avatar display process may occur while analyzing streaming image data (e.g., generating a 3D avatar of a person from a video).

[0075] In one exemplary implementation, environment 800 includes an image synthesis pipeline that collects or obtains data of a physical environment (e.g., image data from image sources such as sensors 812A-812N). Exemplary environment 800 is an example of collecting image sensor data (e.g., light intensity data - RGB) for a registration process to generate registration data 824 (e.g., image data of different head poses and / or different facial expressions), performing a predetermined representation process using registration data 824, and collecting image sensor data 815 (e.g., light intensity data, depth data, and position information) for a frame-specific representation process of multiple image frames. For example, diagram 806 (e.g., Figure 1 100) represents when a user is in a physical environment (e.g., Figure 1812A, 812B through 812N (hereinafter referred to as sensors 812) may include one or more depth cameras that capture depth data, one or more light intensity cameras (e.g., RGB cameras) that capture light intensity image data (e.g., a sequence of RGB image frames), and one or more position sensors for capturing positioning information.

[0076] For positioning information, some specific implementations include a visual inertial odometry (VIO) system to estimate the distance traveled by determining equivalent odometry information using a camera sequence image (e.g., light intensity data). Alternatively, some specific implementations of the present disclosure may include a SLAM system (e.g., a position sensor). The SLAM system may include a multidimensional (e.g., 3D) laser scanning and range measurement system that is independent of GPS and provides real-time simultaneous positioning and mapping. The SLAM system can generate and manage very accurate point cloud data generated by reflections from laser scanning of objects in the environment. Over time, the movement of any point in the point cloud is accurately tracked so that the SLAM system can use the point in the point cloud as a reference point for the position, maintaining an accurate understanding of its position and orientation as it travels through the environment. The SLAM system can also be a visual SLAM system that relies on light intensity image data to estimate the position and orientation of a camera and / or device.

[0077] In an exemplary implementation, environment 800 includes an enrollment instruction set 820 configured with instructions that can be executed by a processor to generate enrollment data from sensor data. For example, enrollment instruction set 820 acquires image data of illustration 806 from a sensor, such as light intensity image data (e.g., an RGB image from a light intensity camera), and generates enrollment data 822 of a user (e.g., facial feature data such as texture, muscle activation, etc.). For example, enrollment instruction set generates enrollment image data 824 (e.g., Figure 7 Registration image data 710).

[0078] In an exemplary implementation, the environment 800 also includes a predetermined representation instruction set 830 configured with instructions executable by a processor to generate a 3D representation 836 (e.g., a PIFu mesh) representing the 3D shape and appearance of the user's upper torso, head, and face at multiple points in time during the enrollment process from the enrollment information 824 (e.g., enrollment images 826). For example, the predetermined representation instruction set 830 collects enrollment data 824 such as light intensity image data (e.g., RGB data from a light intensity camera), depth image data (e.g., depth image data from a depth camera such as an infrared or time-of-flight sensor), and the user in the physical environment (e.g., Figure 1 5 ), and generates representation data 832 (e.g., muscle activations, geometric shapes, a latent space of facial expressions, etc.). Additionally, the predetermined representation instruction set 830 determines reference data 834 that associates the representation data 832 with 3D reference points defined relative to a skeletal representation of the user (e.g., 3D reference points 510 of FIG. 5 ). For example, a 3D representation 836 may be generated using a PIFu technique that locally aligns pixels of a 2D registration image 826 with a global background to form a 3D representation 836 (also referred to as a PIFu mesh). The 3D representation 836 (e.g., Figure 2 The representation 200 includes a plurality of vertices and polygons that may be determined during a registration process based on image data, such as RGB data and depth data.

[0079] In an exemplary implementation, the environment 800 includes a frame-specific representation instruction set 840 configured with instructions executable by a processor to generate representation data 842 from live image data (e.g., sensor data 815), which may include a set of values ​​(e.g., appearance values, depth values, etc.) representing the 3D shape and appearance of a user's face at a certain point in time. For example, the frame-specific representation instruction set 840 collects sensor data 815 from the sensor 812, such as light intensity image data (e.g., RGB data from a light intensity camera), depth image data (e.g., depth image data from a depth camera such as an infrared or time-of-flight sensor), and a user in a physical environment (e.g., Figure 1The user 25 in the physical environment 105 of the user 25 (e.g., camera positioning information, such as position and orientation data from a position sensor, such as posture data), and generate parameterized data for facial parameterization of the representation data 842 (e.g., muscle activation, geometry, a latent space of facial expressions, etc.). For example, the parameterized data may be generated by parameterizing the image 846 (e.g., with respect to the present invention) by changing parameters such as appearance values ​​(such as texture data, color data, opacity, etc.) and depth values ​​of different points of the face based on the sensor data 815. Figure 3B The facial parameterization technique for the frame-specific representation instruction set 840 may include acquiring a local view acquired from the sensor data 815 and determining a small set of parameters (e.g., muscles of the face) from a geometric model to update the user representation. For example, the geometric model may include data sets for eyebrows, eyes, cheeks under the eyes, mouth areas, jaw areas, etc. The parametric tracking of the frame-specific representation instruction set 840 may provide the geometry of the user's facial features. Additionally, the frame-specific representation instruction set 840 determines reference data 844 that associates the representation data 842 with 3D reference points (e.g., 3D reference points 510 of FIG. 5 ) defined relative to a skeletal representation of the user.

[0080] In an exemplary implementation, environment 800 also includes a combined representation instruction set 850. In an exemplary implementation, combined representation instruction set 850 is located at a receiving device for displaying a combined representation, and sensor 812 and other instruction sets (e.g., registration instruction set 820, predetermined representation instruction set 830, and frame-specific representation instruction set 840) are located at another device (e.g., a device of a user whose avatar will be generated at the receiving device of the combined representation data). Alternatively, in some implementations, combined representation instruction set 850, sensor 812 and other instruction sets (e.g., registration instruction set 820, predetermined representation instruction set 830, and frame-specific representation instruction set 840) are located at another device (e.g., a sending device) so that the receiving device will receive data for the combined representation from the other device for display.

[0081] Combined representation instruction set 850 is configured with instructions that can be executed by the processor to generate a representation of the user (e.g., a 3D avatar) based on combining representation data 832 and reference data 834 based on aligning (e.g., via alignment module 852) 3D reference points provided by reference data 834, 844. Alignment module 852 provides instructions to identify 3D reference points in both reference data 834 and reference data 844 in order to align the data sets. Additionally, combined representation instruction set 850 is configured with instructions that can be executed by the processor to display portions of the representation based on corresponding alignment data when these instructions are updated with representation data 842 (e.g., when live image data is acquired and processed from another device by frame-specific representation instruction set 840). For example, the combined representation instruction set 850 collects representation information 832 (e.g., PIFu mesh) from the predetermined representation instruction set 830, collects representation information 842 (e.g., updated appearance values ​​and depth values ​​from live image information) from the frame-specific representation instruction set 840, and generates combined representation information 854 (e.g., a real-time representation of the user, such as a 3D avatar). For example, the frame-specific representation instruction set 840 may generate representation data 842 and generate a parameterized image 846 (e.g., Figure 7 's avatar 735).

[0082] In some implementations, the combined representation instruction set 850 can be repeated for each frame captured during each moment / frame of a live communication session or other experience. For example, for each iteration, while the user is using the device (e.g., wearing an HMD), the exemplary environment 800 can involve continuously obtaining representation data 842 (e.g., appearance values ​​and depth values), and for each frame, updating a displayed portion of the 3D representation 856 based on the updated values. For example, for each new frame of parameterized data, the system can update the display of the 3D avatar based on the new data.

[0083] Fig. 9 An exemplary electronic device operating in different physical environments during a communication session between a first user at a first device and a second user at a second device, and a view of a 3D representation of the second user of the first device are shown according to some specific implementations. Specifically, Fig. 9 An exemplary operating environment 900 is shown of electronic devices 910, 965 operating in different physical environments 902, 950, respectively, during a communication session (e.g., when the electronic devices 910, 965 share information with each other or with an intermediary device such as a communication session server). Fig. 9In this example, the physical environment 902 is a room that includes wall hangings 912, plants 914, and tables 916. The electronic device 910 includes one or more cameras, microphones, depth sensors, or other sensors that can be used to capture information about the physical environment 902 and objects therein and information about a user 925 of the electronic device 910, as well as to evaluate the physical environment and objects therein. Information about the physical environment 902 and / or the user 925 can be used to provide visual content (e.g., for user representation) and audio content (e.g., for text transcription) during the communication session. For example, the communication session may provide one or more participants (e.g., users 925, 960) with the following views: a 3D environment generated based on a camera image and / or a depth camera image of the physical environment 902, a representation of the user 925 based on a camera image and / or a depth camera image of the user 925, and / or a text transcription (e.g., a transcription bubble) of audio spoken by the user. As shown in FIG. Fig. 9 As shown, user 925 is speaking to user 960 , as shown by speaking word 915 .

[0084] In this example, the physical environment 950 is a room that includes a wall hanging 952, a sofa 954, and a coffee table 956. The electronic device 965 includes one or more cameras, microphones, depth sensors, or other sensors that can be used to capture information about the physical environment 950 and objects therein and information about a user 960 of the electronic device 965, as well as to evaluate the physical environment and objects therein. Information about the physical environment 950 and / or the user 960 can be used to provide visual and audio content during the communication session. For example, the communication session can provide a view of a 3D environment generated based on a camera image and / or a depth camera image (from the electronic device 965) of the physical environment 950 and a representation of the user 960 based on a camera image and / or a depth camera image (from the electronic device 965) of the user 960. For example, in a communication with the device 965 via the communication session instruction set 990, the 3D environment can be sent by the device 910 via the communication session instruction set 980 (e.g., via the network connection 985). As Fig. 9 As shown, audio spoken by user 925 (e.g., spoken word 915) is transcribed (e.g., via communication instruction set 990) at device 965 (or via a remote server), and view 966 provides user 960 with a text transcription of the audio spoken by the speaker (user 925) via transcription bubble 976 (e.g., "Nice avatar!").

[0085] Fig. 9An example of a view 905 of a virtual environment (e.g., a 3D environment 930) at a device 910 is shown, wherein a representation 932 of a wall hanging 952 and a user representation 940 (e.g., an avatar of a user 960) are provided, provided that the user representation of each user is viewed during a particular communication session. Specifically, the user representation 940 of the user 960 is generated based on the combined user representation techniques described herein (e.g., for more realistic avatars generated in real time). Additionally, an electronic device 965 within the physical environment 950 provides a view 966 that enables the user 960 to view a representation 975 (e.g., an avatar) of at least a portion of the user 925 within the 3D environment 970 (e.g., from the mid-torso upwards) and a transcription of a word spoken by the user 925 via a transcription bubble 976 (e.g., "Nice avatar!"). In other words, at device 910, a more realistic-looking avatar (e.g., user representation 940 of user 960) is generated by generating a combined 3D representation of user 960 for multiple moments in a time period based on data obtained from device 965 (e.g., a predetermined 3D representation of user 960 and a corresponding frame-specific 3D representation of user 960). Alternatively, in some embodiments, user representation 940 of user 960 is generated at device 965 (e.g., a transmitting device of the speaker) and sent to device 910 (e.g., a viewing device viewing the avatar of the speaker). Specifically, each 3D representation of the combined 3D representation 940 of user 960 is generated by combining the predetermined 3D representation of user 960 with the corresponding frame-specific 3D representation of user 960 based on alignment (e.g., alignment of 3D reference points) according to the techniques described herein.

[0086] exist Fig. 9In the example of , electronic devices 910 and 965 are shown as handheld devices. Electronic devices 910 and 965 can be mobile phones, tablet computers, laptops, etc. In some specific implementations, electronic devices 910 and 965 can be worn by users. For example, electronic devices 910 and 965 can be watches, head-mounted devices (HMD), head-mounted devices (glasses), headphones, ear-hook devices, etc. In some specific implementations, the functions of devices 910 and 965 are implemented via two or more devices, such as mobile devices and base stations or head-mounted devices and ear-hook devices. Various functions can be distributed in multiple devices, including but not limited to power functions, CPU functions, GPU functions, storage functions, memory functions, visual content display functions, audio content production functions, etc. Multiple devices that can be used to implement the functions of electronic devices 910 and 965 can communicate with each other via wired or wireless communications. In some specific implementations, each device communicates with a separate controller or server to manage and coordinate the user's experience (e.g., a communication session server). Such a controller or server may be located in physical environment 902 and / or physical environment 950 or may be remotely located relative to the physical environment.

[0087] Additionally, in Fig. 9 In the example of , 3D environments 930 and 970 are XR environments based on a common coordinate system that can be shared with other users (e.g., a virtual room for avatars of a multi-person communication session). In other words, the common coordinate system of 3D environments 930 and 970 is different from the coordinate system of physical environments 902 and 950, respectively. For example, a common reference point can be used to align the coordinate system. In some specific implementations, the common reference point can be a virtual object within the 3D environment that each user can visualize within its respective view. For example, a common centerpiece table around which a user representation (e.g., a user's avatar) is positioned within the 3D environment. Alternatively, the common reference point is not visible within each view. For example, the common coordinate system of the 3D environment can use the common reference point to position each respective user representation (e.g., around a table / desk). Therefore, if the common reference point is visible, each view of the device will be able to visualize the "center" of the 3D environment for perspective when viewing other user representations. The visualization of the common reference point can become more relevant to a multi-user communication session, so that each user's view can add perspective to the position of each other user during the communication session.

[0088] In some implementations, the representation of each user may be real or unreal and / or may represent the user's current and / or previous appearance. For example, a photo-realistic representation of user 925 or 960 may be generated based on a combination of the user's live image and previous images. Previous images may be used to generate representations of portions of the representation for which live image data is not available (e.g., portions of the user's face that are not in the field of view of a camera or sensor of the electronic device 910 or 965 or that may be obscured, for example, by a head-mounted device or otherwise). In one example, electronic devices 910 and 965 are head-mounted devices (HMDs), and live image data of the user's face includes downward-facing camera images of the user's cheeks and mouth and inward-facing camera images of the user's eyes, which may be combined with previous image data of other portions of the user's face, head, and torso that are currently not observable from the device's sensors. Previous data on the user's appearance may be obtained at an earlier time during a communication session, during previous use of the electronic device, during a registration process for obtaining sensor data of the user's appearance from multiple viewpoints and / or conditions, or otherwise.

[0089] Some embodiments provide for a representation of at least a portion of a user within a 3D environment other than the user's physical environment during a communication session, and based on a detected condition, a representation of another object of the user's physical environment is provided to provide context. For example, during a communication session, representations of one or more other objects of the physical environment may be displayed in a view. For example, based on a determination that the user 925 is interacting with a physical object in the physical environment 902, a representation (e.g., real or proxy) may be displayed in a view to provide context for the user 925's interaction. For example, if the first user 925 picks up an object such as a family photo frame to show another user, the view may include a real view of the photo frame (e.g., a live video). Thus, when displaying the XR environment, the view may present a virtual object representing the user picking up a generic object, display a virtual object similar to a photo frame, display a previously acquired image of an actual photo frame from an obtained 3D scan, and the like.

[0090] Fig.10 1 is a flow chart illustrating an exemplary method 1000. In some implementations, a device (e.g., Figure 1 Device 10 or Fig. 9The method 1000 is performed on a device 965 of the embodiment of the present invention to generate a combined 3D representation of the user for multiple moments in a time period based on the alignment based on the 3D reference points. In some specific implementations, the method 1000 is performed on a mobile device, a desktop computer, a laptop computer, an HMD, or a server device. In some specific implementations, the method 1000 is performed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, the method 1000 is performed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).

[0091] In some implementations, method 1000 is implemented at a processor of a device (such as a viewing device) that presents a combined 3D representation (e.g., Fig. 9 Device 910 presents a 3D representation 940 (avatar) of user 960 based on the time obtained from device 965).

[0092] At block 1002, method 1000 obtains a predetermined three-dimensional (3D) representation of at least a first portion of a user, wherein the predetermined 3D representation is associated with 3D reference points defined relative to a skeletal representation of the user. For example, the predetermined 3D representation may represent an upper torso and a head of the user. The predetermined 3D representation may be generated using a pixel aligned implicit function (PIFu) technique that locally aligns pixels of a 2D registration image with a global background to form the predetermined 3D representation (e.g., Figure 2 200).

[0093] In some implementations, the 3D reference point is associated with the 3D location of the atlas joint of the user's skeletal representation. For example, the 3D reference point can be the head / atlas joint, which can be determined by tracking the x-axis aligned with the ear canal and / or the z-axis on the Frankfurt plane. In some implementations, the 3D reference point is associated with the center of the user's eye, which is defined at a location offset from the atlas joint. For example, during the registration process, an offset can be determined to provide a mid-pupil origin for the parameterized mesh. In some implementations, the 3D reference point can be based on the skeletal atlas joint and user-specific head shape characteristics (e.g., based on the atlas joint of the user). Figure 5A and Figure 5B The offset 515 in the figure determines the offset position of the 3D reference point 510 associated with the position of the atlas joint 520) at a point centered between the user's eyes.

[0094] At block 1004, method 1000 obtains a sequence of frame-specific 3D representations corresponding to a plurality of moments in time within a time period, each of the frame-specific 3D representations representing a second portion of the user at a corresponding moment in time in the plurality of moments in time within the time period, and each of the frame-specific 3D representations being associated with a 3D reference point. For example, each frame-specific 3D representation may represent a face of the user, and thus the first portion and the second portion may overlap and represent some common area of ​​the user.

[0095] In some implementations, the second portion of the user is a sub-portion (e.g., only the face) of the first portion (e.g., the face is represented). In some implementations, the first portion of the user includes the facial portion and the additional portion of the user, and the second portion of the user includes the facial portion without the additional portion of the user.

[0096] In some implementations, each frame-specific 3D representation can be generated based on sensor data captured by the sending device during a communication (e.g., coexistence) session. In some implementations, each frame-specific 3D representation can be generated using sensor data from an inward / downward facing camera and using enrollment data (e.g., images of faces in different expressions, images of parts of the face that cannot be captured when the user is wearing the HMD, or images of parts that cannot be otherwise captured during actual use). In some implementations, each frame-specific 3D representation can represent the user's face using a curved parameterized mesh positioned relative to a 3D reference point.

[0097] In some implementations, a sequence of frame-specific 3D representations corresponding to multiple moments in a time period is generated based on sensor data to represent a set of values, wherein the set of values ​​(e.g., parameterized values) may include: i) depth values ​​defining 3D positions of parts of the user relative to multiple 3D positions of points of a projection surface, and ii) appearance values ​​defining the appearance of the parts of the user. For example, generating a set of values ​​(e.g., RGB values, alpha values, and depth values ​​- RGBDA) representing the user based on sensor data may involve using live sensor data from an inward / downward facing camera and using enrollment data, such as images of a face in different expressions without wearing an HMD. In some implementations, generating the set of values ​​may involve using a machine learning model trained to produce the set of values.

[0098] The set of values ​​may include depth values ​​defining the 3D position of a portion of the user relative to a plurality of 3D positions of points of the projection surface. For example, the depth value of a point may define that a portion of the face is at a depth D1 behind the position of the point on the surface, e.g., along an orthogonal ray starting at the point (e.g., Figure 3Bray 332) is at depth D1. In some implementations, the depth value defines the distance between a portion of the user and a corresponding point on the projection surface, which is located at the location of the corresponding point along an orthogonal ray perpendicular to the projection surface. The techniques described herein use depth values ​​that are different from depth values ​​in existing RGBDA images, which define the depth of content relative to a single camera position. The appearance value may include values ​​that define the appearance of the portion of the user, such as RGB data and alpha data. For example, the appearance value may include color, texture, opacity, etc.

[0099] In some implementations, the term "surface" refers to a 2D manifold that can be planar or non-planar. In some implementations, the points of a surface (e.g., a 2D manifold surface) are spaced at regular intervals along vertical and horizontal lines on the surface. In some implementations, the points are spaced regularly along vertical and horizontal grid lines on a partially cylindrical surface, such as Figure 3A and Figure 3B As shown. Alternatively, other planar surfaces and non-planar surfaces may be utilized. For example, a flat surface of a cylindrical surface may be oriented / curved around different axes. Additionally or prior, the flat surface may be a hemispherical manifold. In some implementations, the points may be unevenly distributed across the 2D manifold surface, such as irregularly spaced along vertical and horizontal grid lines around the surface, but may be focused on specific areas of the user's face. For example, some areas where there may be more detail / motion in the facial structure may have more points, and some points may have fewer points in areas where there may be less detail / motion, such as the forehead (less detail) and nose (not much motion). For example, a higher density of points may be around the eyes and mouth.

[0100] In some implementations, the set of values ​​is generated based on an alignment such that a subset of the points on the central region of the surface corresponds to a central portion of the user's face. Figure 3B As shown, the focal area of ​​the user's nose is at area 330, and the feature point of ray 332 is the tip of the human nose.

[0101] In some implementations, generating the set of values ​​is further based on images of the user's face captured in different poses, and / or when the user is expressing multiple different facial expressions. For example, the set of values ​​is determined based on enrollment images of the face when the user is facing the camera, the left side of the camera, and the right side of the camera, and / or when the user smiles, raises eyebrows, puffs cheeks, etc. In some implementations, the sensor data corresponds only to a first area of ​​the user (e.g., a portion not obscured by a device (such as an HMD)), and the set of image data (e.g., enrollment data) corresponds to a second area that includes a third area different from the first area. For example, the second area may include some of the portions that are obscured by the HMD when the HMD is worn by the user.

[0102] In some implementations, determining the user-specific parameterization (e.g., generating the set of values) may be adapted for each specific user. For example, the parameterization may be fixed based on the registered identity (e.g., to better cover the head size or nose shape of a person), or the parameterization may be based on the current expression (e.g., when the mouth is open, the parameterization may become longer). In an exemplary implementation, method 1000 may also include: obtaining additional sensor data of the user associated with a second time period; updating the set of values ​​representing the user based on the additional sensor data for the second time period; and providing the updated set of values, wherein the depiction of the user is updated in the second time period based on the updated set of values ​​(e.g., updating the set of values ​​based on the current expression so that when the mouth is open, the parameterization also becomes longer).

[0103] In some implementations, generating the set of values ​​representing the user is based on a machine learning model that is trained to produce the set of values. For example, a process for generating representation data 832 of a predetermined representation instruction set 830 and / or parameterized data of a frame-specific representation instruction set 840 is provided by a machine learning model (e.g., a trained neural network) to identify patterns in textures (or other features) in the enrollment data 822 and sensor data 815 (live image data, such as image 816). In addition, the machine learning model can be used to match these patterns with learned patterns corresponding to the user 25 (such as smiling, frowning, talking, etc.). For example, when a smile pattern is determined by teeth showing, there can also be a determination of other parts of the face that also change when the user smiles (e.g., cheek movement, eyebrows, etc.). In some implementations, the techniques described herein can learn patterns specific to the user 25. Figure 1 pattern of a specific user 25.

[0104] At block 1006, method 1000 generates a combined 3D representation of the user for a plurality of moments in the time period, wherein each of the combined 3D representations is generated by combining the predetermined 3D representation with the corresponding frame-specific 3D representation based on an alignment, wherein the alignment is based on the 3D reference point. In some implementations, the frame-specific 3D representations may each represent a second portion of the user, the second portion being a sub-portion (e.g., only the face) of the first portion (e.g., the represented face), such that the frame-specific 3D representation may be combined with the predetermined 3D representation by simply adjusting the sub-portion (e.g., the face portion) of the predetermined 3D representation.

[0105] In some implementations, obtaining a frame-specific 3D representation may involve adjusting the positions of some of the vertices of the predetermined 3D representation, and then applying texture / color based on each of the frame-specific 3D representations. This may deform and color a predetermined sub-portion (e.g., face) to correspond to the real-time shape and color of that portion (e.g., face) of the user at each moment in time.

[0106] In some implementations where a parameterized mesh (e.g., a height field map) is used to define a frame-specific 3D representation, the 3D representation may be constructed by averaging the height of the frame based on 3D reference points (e.g., Figure 6 The 3D reference points 602 and 624 of the predetermined 3D representation are used to map vertices of the predetermined 3D representation to positions on the parameterized mesh to facilitate the combination of data. The mapping using the 3D reference points enables frame-specific facial data specified on the parameterized mesh to be directly used to adjust the positions of the vertices of the predetermined 3D representation. In some implementations, the positions of the vertices can be adjusted by blending the predetermined vertex positions of the vertices with their frame-specific data vertex positions (e.g., using a specified alpha value). In other words, the predetermined 3D representation vertices can be mapped to the parameterized mesh, the parameterized mesh can be adjusted using live facial data, and the adjusted parameterized mesh represents a combined 3D representation of the user combining the predetermined 3D representation with one of the frame-specific 3D representations. In some implementations, the mapping can be determined during the enrollment process.

[0107] In some implementations, combining the predetermined 3D representation with the corresponding frame-specific 3D representation includes adjusting a sub-portion (e.g., a facial portion) of the predetermined 3D representation. In some implementations, adjusting the sub-portion of the predetermined 3D representation includes adjusting positions of vertices of the predetermined 3D representation and applying a texture based on each frame-specific 3D representation in the frame-specific 3D representation. For example, the adjustment may deform and color the predetermined sub-portion (e.g., the face) to correspond to the real-time shape and color of the portion (e.g., the face) of the user at each moment in time.

[0108] In some implementations, method 1000 may further include presenting a view of the combined 3D representation. Fig. 9As shown, the presentation occurs during a communication session in which a second device (e.g., device 965) captures sensor data (e.g., image data of a user 960 and a portion of an environment 950) and provides a sequence of frame-specific 3D representations corresponding to multiple moments in a time period based on the sensor data. For example, the second device 965 provides / transmits the sequence of frame-specific 3D representations to the device 910, and the device 910 generates a combined 3D representation to display a live 3D video-like facial depiction (e.g., a realistic moving avatar) of the user 960 (e.g., a representation 940 of the user 960). Alternatively, in some specific implementations, the second device provides a predetermined 3D representation of the user (e.g., a representation 940 of the user 960) (e.g., a realistic moving avatar) during the communication session. For example, the combined representation is determined at the device 965 and sent to the device 910. In some specific implementations, the view of the combined 3D representation is displayed on the device (e.g., device 910) in real time relative to multiple moments in the time period. For example, a depiction of the user is displayed in real-time (eg, an avatar shown to the second user on a display of the second device of the second user).

[0109] In some implementations, providing the set of values ​​includes sending a sequence of frames of 3D video data during a communication session with a second device, the sequence of frames including the frame containing the set of values, wherein the second device presents an animated depiction of the user based on the sequence of frames of 3D video data. For example, the set of points can be a frame of 3D video data sent during a communication session with another device, and the other device uses the set of values ​​(along with information about how to interpret the depth values) to present a view of the user's face. Additionally or alternatively, consecutive frames of facial data (a set of values ​​representing the 3D shape and appearance of the user's face at different points in time) can be transmitted and used to display a live 3D video-like facial depiction.

[0110] In some implementations, the depiction of the user may include sufficient data to enable a stereoscopic view of the user (e.g., left / right eye view) so that the face can be perceived with some depth. In one implementation, the depiction of the face includes a 3D model of the face, and views of the representation seen from left and right eye positions are generated to provide a stereoscopic view of the face.

[0111] In some implementations, certain parts of a face that may be important for conveying a realistic or faithful appearance (such as the eyes and mouth) may be generated differently than other parts of the face. For example, parts of a face that may be important for conveying a realistic or faithful appearance may be based on current camera data, while other parts of the face may be based on previously obtained (e.g., enrolled) facial data.

[0112] In some implementations, a representation of a face is generated with textures, colors, and / or geometries of various facial parts that identifies estimates of how confident the generation technique is, per frame of data, that such textures, colors, and / or geometries accurately correspond to the true textures, colors, and / or geometries of those facial parts based on depth values ​​and appearance values. In some implementations, the depiction is a 3D avatar. For example, the representation is a representation of a user (e.g., Figure 1 3D model of user 25).

[0113] In some implementations, the predetermined 3D representation and / or frame-specific 3D representation sequence is based on obtaining sensor data of the user. For example, sensor data (e.g., live data, such as video content including light intensity data (RGB) and depth data) is associated with a point in time, such as from an inward / downward facing sensor (e.g., when the user is wearing the HMD) Figure 1 An image of the sensor 35a, 35b, 35c) shown in FIG. 3 is associated with a frame. In some implementations, the sensor data includes depth data (e.g., infrared, time of flight, etc.) and light intensity image data obtained during the scanning process.

[0114] In some implementations, obtaining sensor data may include obtaining a first set of data (e.g., enrollment data) corresponding to features (e.g., texture, muscle activation, shape, depth, etc.) of the user's face from the device in a plurality of configurations (e.g., Figure 7 710 of enrollment image data). In some implementations, the first set of data includes unobstructed image data of the user's face. For example, an image of the face may be captured when the user smiles, raises eyebrows, puffs cheeks, etc. In some implementations, the enrollment data may be obtained by the user taking off the device (e.g., HMD) and capturing an image when the device does not cover the face, or using another device (e.g., mobile device) when the device (e.g., HMD) does not cover the face. In some implementations, the enrollment data (e.g., the first set of data) is collected from a light intensity image (e.g., an RGB image). The enrollment data may include texture, muscle activation, etc. of most, if not all, of the user's face. In some implementations, the enrollment data may be captured when different instructions are provided to the user for capturing different poses of the user's face. For example, a user interface guide may instruct the user to "raise your eyebrows," "smile," "frown," etc. in order to provide the system with a series of facial features for the enrollment process.

[0115] In some implementations, obtaining sensor data may include obtaining a second set of data corresponding to one or more partial views of a face from one or more image sensors while the user is using (e.g., wearing) an electronic device (e.g., an HMD). For example, obtaining sensor data may include Figure 7 In some implementations, the second set of data includes a partial image of the user's face and therefore may not represent all features of the face represented in the enrollment data. For example, the second set of images may include images from an upward-facing sensor (e.g., Figure 1 Additionally or alternatively, the second set of images may include images of some of the front face / eyes of the sensor 35a (e.g., facial feature characteristic data 40a). Additionally or alternatively, the second set of images may include images from the inward-facing sensor (e.g., Figure 1 Additionally or alternatively, the second set of images may include images from downward-facing sensors (e.g., Figure 1 In some implementations, the electronic device includes a first sensor (e.g., Figure 1 sensor 35a) and a second sensor (e.g., Figure 1 The method further comprises the step of: obtaining a second set of data from at least one partial image of the user's face seen from a first viewpoint from the first sensor (e.g., upper facial feature data 40a) and at least one partial image of the user's face seen from a second viewpoint different from the first viewpoint from the second sensor (e.g., lower facial feature data 40c) (e.g., multiple IFC cameras are used to capture different viewpoints of the user's face and body movements).

[0116] In some implementations, method 1000 may be repeated for each frame captured during each moment / frame of a live communication session or other experience. For example, for each iteration, while the user is using the device (e.g., wearing an HMD), method 1000 may involve continuously obtaining live sensor data (e.g., eye gaze characteristic data and facial feature data), and for each frame, updating a portion of the display of the representation based on updated parameterized values ​​(e.g., RGBDA values) of a frame-specific 3D representation sequence. For example, for each new frame, the system may update the parameterized values ​​to update the display of a 3D avatar based on the new data.

[0117] Fig.11An exemplary environment 1100 is shown for implementing a process for adjusting one or more colors of a predetermined 3D representation based on colors in the predetermined 3D representation and colors in a sample 3D representation. The environment 1100 shows the adjustment of colors based on a combination of predetermined 3D representation data 1110 (e.g., a PIFU-based 3D representation of a user's upper body and head) and frame-specific representation data 1120 (e.g., 3D representation data of a user's head / face based on a parametric mesh that can be updated at each frame) to generate an adjusted combined representation 1130 according to the techniques described herein. The predetermined 3D representation data 1110 includes inconsistent representation data 1112 that can be transformed into reconstructed representation data 1114. For example, the inconsistent representation data 1112 shows actual input texture data from the PIFu algorithm that is different and may be difficult to correlate with other representation data. Therefore, the inconsistent representation data 1112 can be reconstructed into reconstructed representation data 1114, which can then be mapped / combined with other representation data (such as the frame-specific representation data described herein). For example, the process may include adjusting the predetermined 3D representation (reconstructed representation data 1114) by adjusting one or more colors of the predetermined 3D representation based on the colors in the predetermined 3D representation and the colors in the frame-specific 3D representation (frame-specific representation data 1120). For example, the adjustment may be based on a transformation (e.g., determined using a Monge-Kanorovich color transfer technique). In some implementations, determining such a transformation may be facilitated by reconstructing the predetermined representation into the same space as the sample 3D reconstruction. In some implementations, adjusting the predetermined 3D representation includes generating a transformation based on the predetermined 3D representation and the sample 3D representation. In some implementations, generating the transformation includes calculating an average color of a first 3D representation corresponding to a portion of the user, calculating a second average color of a sample 3D representation corresponding to the portion of the user (e.g., the same portion), and deriving a transformation based on the first average color and the second average color, wherein the transformation is a linear transformation matrix. For example, generating the transformation may include applying a Monge-Kantorovich color transfer technique to derive a linear matrix, wherein the matrix maps colors from the PIFu image to a parameterized based image while retaining details of the PIFu image. Use of the transformation as described herein provides results in 3D representations that include more consistent hair and skin color (eg, more realistic avatars).

[0118] Fig.12 According to some specific implementations, a method for implementing Fig.101200. The environment 1000 illustrates a process for generating a 3D representation of a user by adjusting one or more colors of a 3D representation of the user. The environment 1000 illustrates a process for color adjustment based on utilizing predetermined 3D representation data 1210, frame-specific representation data 1220 (e.g., based on a parameterized image (parameterized mesh 622) or a height field map) and a color transform 1230 to generate a combined representation 1240 according to the techniques described herein. For example, the process as shown in the environment 1200 may include adjusting the color of a first 3D representation of a first portion of a user (e.g., head, hair, upper body, clothing, etc.) determined via a first technique (e.g., PIFu-predetermined 3D representation 1210). The color may be adjusted using a sample 3D representation of a second portion of the user determined via a second different technique (e.g., frame-specific representation 1220 (e.g., height field representation of a face)). For example, the face / hair in a sample captured at enrollment may be utilized. Color adjustments made according to the techniques described herein provide better color matching in a combined 3D representation generated by combining a first 3D representation with one or more second 3D representations (e.g., frame-specific 3D representations) generated via a second technique (e.g., a live sequence of frame-specific 3D representation data, such as the set of values ​​representing the 3D shape and appearance of a user's face at a point in time as described herein). For example, the predetermined 3D representation data 1210 may be PIFu-based and may be adjusted so that the neck color better matches the facial color of the frame-specific 3D facial representation 1220. The color may be adjusted using a transformation 1230 (e.g., determined using a Monge-Kanorovich color transfer technique, etc.). Determining the transformation 1230 may be facilitated by reconstructing the predetermined 3D representation data 1210 into the same space as the frame-specific 3D facial representation 1220. For example, the color transformation 1230 may include reconstructing the PIFu representation in UV space. This enables a more accurate transformation based on a comparison of the colors of the corresponding user parts.

[0119] In some implementations, color adjustments are applied to other portions of the predetermined 3D representation data 1210, such as the user's hands, where the data does not overlap with the frame-specific representation data 1220, which may correspond only to the user's face and / or head (e.g., a cylindrical 2D shape designed to update facial features during a communication session). In some implementations, filtering techniques may be utilized to identify different portions of the frame-specific representation data 1220 that correspond to non-skin features (e.g., hair, clothing, etc.). In some implementations, filtering the frame-specific representation data 1220 includes identifying portions of the sample 3D representation that correspond to the user's hair or clothing. In some implementations, filtering the sample 3D representation includes excluding non-skin features from the filtered sample 3D representation. Such non-skin features may be identified via an algorithm or machine learning model (e.g., using a semantic segmentation algorithm or model). In some implementations, filtering the sample 3D representation includes excluding portions of the sample 3D representation based on brightness (e.g., taking only the top 25% to address shadow issues). In some implementations, adjusting the first 3D representation includes generating a transformation based on the first 3D representation and the filtered sample 3D representation. In some implementations, adjusting the first 3D representation includes applying the transformation to change each sub-portion (e.g., texel) of the first 3D representation (e.g., correcting the color of all parts of the hand).

[0120] In some implementations, the adjustment process also includes rebuilding the predetermined 3D representation data 1210 to use the same space as the frame-specific representation data 1220. In some implementations, rebuilding the first 3D representation includes generating a texture parameterization of the first 3D representation. In some implementations, the sample 3D representation includes 2D height field data representing the 3D appearance of the second portion, and rebuilding the predetermined 3D representation data 1210 includes constructing a bounding volume hierarchy (BVH) of 2D UV positions and indices to map 2D coordinates of the height field of the sample 3D representation to a sub-portion of the first 3D representation, and based on the construction, representing the first 3D representation in the 2D height field (e.g., using "PIFu-uvBVH" data - 2D UV position and index BVH). For example, the uvBVH can be used to map 2D UV coordinates to 3D triangle indices. Mapping 2D UV coordinates to 3D triangle indices allows finding triangles that reference valid portions of texture data of the first 3D representation. Additionally, the triangle indices may be utilized to map to UV coordinates of the sample 3D representation and reconstruct a 2D image of the first 3D representation (eg, PIFu representation) aligned with the sample 3D representation.

[0121] Fig.13 13 is a flow chart illustrating an exemplary method 1300. In some implementations, a device (e.g., Figure 1 Device 10 or Fig. 9The method 1300 is performed on a device 965 of the present invention to adjust one or more colors in a 3D representation for generating a combined 3D representation. In some implementations, the method 1300 is performed on a mobile device, a desktop computer, a laptop computer, an HMD, or a server device. In some implementations, the method 1300 is performed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 1300 is performed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).

[0122] In some implementations, method 1300 is implemented at a processor of a device (e.g., a viewing device) that renders the combined 3D representation. Fig. 9 Device 910 renders a 3D representation 940 (avatar) of user 960 based on data obtained from device 965 .

[0123] At block 1302, method 1300 obtains a first 3D representation of at least a first portion of a user, wherein the first 3D representation is generated via a first technique. For example, the first 3D representation may represent an upper torso and a head. In some implementations, the first 3D representation may be generated using a PIFu technique that locally aligns pixels of a 2D registration image with a global background to form the first 3D representation.

[0124] At block 1304, method 1300 obtains a sample 3D representation of at least a second portion of the user, wherein the sample 3D representation is generated via a second technique different from the first technique, and the first portion represents at least a portion of the user (e.g., neck, clothing, etc.) different from the second portion of the user (e.g., face and hair only). For example, the sample 3D representation can be a sample parameterized-based representation (e.g., parameterized image 320) from enrollment, such as the frame-specific 3D representation data discussed herein.

[0125] In some implementations, the first portion of the user represents the user's face, hair, neck, upper torso, and / or clothing, and the second portion represents only the user's face and / or hair.

[0126] At block 1306, method 1300 adjusts the first 3D representation by adjusting one or more colors of the first 3D representation based on the colors in the first 3D representation and the colors in the sample 3D representation. For example, the adjustment can be based on a transformation (e.g., determined using a Monge-Kanorovich color transfer technique). In some implementations, determining such a transformation can be facilitated by reconstructing the first representation into the same space as the sample 3D reconstruction.

[0127] In some embodiments, adjusting the first 3D representation includes generating a transformation based on the first 3D representation and the sample 3D representation. In some embodiments, generating the transformation includes calculating the average color of the first 3D representation corresponding to a portion of the user, calculating the second average color of the sample 3D representation corresponding to the portion of the user (e.g., the same portion), and deriving the transformation based on the first average color and the second average color, wherein the transformation is a linear transformation matrix. For example, generating the transformation may include applying the Monge-Kantorovich color transfer technique to derive a linear matrix, wherein the matrix maps colors from the PIFu image to a parameterized image while retaining the details of the PIFu image. The use of the transformation as described herein provides a result of a 3D representation including more consistent hair and skin color (e.g., a more realistic avatar).

[0128] In some implementations, adjusting the first 3D representation includes reconstructing the first 3D representation into the same space as the sample 3D representation (e.g., a 2D height field space), determining a transformation based on the reconstructed first 3D representation and the sample 3D representation, and adjusting the first 3D representation based on the transformation. In some implementations, adjusting the first 3D representation includes applying the transformation to change each sub-portion (e.g., texel) of the first 3D representation. For example, applying the transformation can be used to correct the color of all parts of the 3D representation (e.g., correcting a neck, clothing, and other elements to avoid color discontinuities, given the assumption that there are additional portions that may also require color correction).

[0129] At block 1308, the method 1300 provides the adjusted first 3D representation for use in generating a combined 3D representation that combines the adjusted first 3D representation with one or more second 3D representations, wherein the second 3D representations are generated via a second technique. For example, the adjusted PIFu data from the registration is provided for use during a live communication session, wherein the adjusted PIFu data is combined with frame-specific, live parameterization-based representation data (such as frame-specific 3D representation data as discussed herein).

[0130] In some implementations, method 1300 also includes reconstructing the first 3D representation to use the same space as the sample 3D representation. In some implementations, reconstructing the first 3D representation includes generating a texture parameterization of the first 3D representation. In some implementations, the sample 3D representation includes 2D height field data representing the 3D appearance of the second portion, and reconstructing the first 3D representation includes constructing a bounding volume hierarchy of 2D UV positions and indices to map 2D coordinates of the height field of the sample 3D representation to a sub-portion of the first 3D representation, and representing the first 3D representation in the 2D height field based on the construction (e.g., using "PIFu-uvBVH" data - a BVH of 2D UV positions and indices).

[0131] Fig.14 1400 is a flowchart illustrating an exemplary method 1400. In some implementations, a device (e.g., Figure 1 Device 10 or Fig. 9 The method 1400 is performed on a device 965 of the embodiment of the present invention to adjust one or more colors in a 3D representation for generating a combined 3D representation based on filtering the identified parts (e.g., filtering hands and clothing from the user's head / face). In some implementations, the method 1400 is performed on a mobile device, a desktop computer, a laptop computer, an HMD, or a server device. In some implementations, the method 1400 is performed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 1400 is performed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).

[0132] In some implementations, method 1400 is implemented at a processor of a device (e.g., a viewing device) that renders the combined 3D representation. Fig. 9 Device 910 renders a 3D representation 940 (avatar) of user 960 based on data obtained from device 965 .

[0133] At block 1402, method 1400 obtains a first 3D representation of at least a first portion of a user (e.g., a hand). In some implementations, the first 3D representation is generated via a first technique. For example, the first 3D representation may represent an upper torso, a head, and a hand. In some implementations, the first 3D representation may be generated using a PIFu technique that locally aligns pixels of a 2D registration image with a global background to form the first 3D representation.

[0134] At block 1404, method 1400 obtains a sample 3D representation of at least a second portion of the user, wherein the first portion represents at least a portion of the user (e.g., only hands) that is different from the second portion of the user (e.g., only face and hair). In some implementations, the sample 3D representation is generated via a second technique that is different from the first technique. For example, the sample 3D representation can be a sample parameterization-based representation (e.g., parameterized image 320) from enrollment, such as the frame-specific 3D representation data discussed herein.

[0135] In some implementations, the first portion includes the user's hands and the second portion includes the user's face and / or hair. In some implementations, the first portion and the second portion of the user do not overlap. For example, the first portion includes only the user's hands and the second portion includes only the user's face and / or hair.

[0136] At block 1406, method 1400 filters the sample 3D representation based on identifying portions of the sample 3D representation that correspond to non-skin features (e.g., hair, clothing, etc.). In some implementations, filtering the sample 3D representation includes identifying portions of the sample 3D representation that correspond to hair or clothing of the user. In some implementations, filtering the sample 3D representation includes excluding non-skin features from the filtered sample 3D representation. In some implementations, filtering the sample 3D representation includes excluding portions of the sample 3D representation based on brightness (e.g., taking the top 25% to resolve any shadow issues).

[0137] At block 1408, method 1400 adjusts the first 3D representation by adjusting one or more colors of the first 3D representation based on the colors in the first 3D representation and the colors in the filtered sample 3D representation. For example, the adjustment can be based on a transformation (e.g., determined using a Monge-Kanorovich color transfer technique). In some implementations, determining such a transformation can be facilitated by reconstructing the first representation into the same space as the sample 3D reconstruction.

[0138] In some implementations, adjusting the first 3D representation includes generating a transformation based on the first 3D representation and the filtered sample 3D representation. In some implementations, adjusting the first 3D representation includes applying the transformation to change each sub-portion (e.g., texel) of the first 3D representation (e.g., correcting the color of all parts of the hand).

[0139] At block 1408, method 1400 provides the adjusted first 3D representation for generating a combined 3D representation that combines the adjusted first 3D representation with one or more second 3D representations. In some implementations, the second 3D representation is generated via a second technique. For example, the adjusted PIFu data from the enrollment is provided for use during a live communication session, where the adjusted PIFu data is combined with frame-specific, live parameterized representation data (such as frame-specific 3D representation data as discussed herein) (e.g., combining hands and faces for each frame).

[0140] In some implementations, method 1400 also includes reconstructing the first 3D representation to use the same space as the sample 3D representation. In some implementations, reconstructing the first 3D representation includes generating a texture parameterization of the first 3D representation. In some implementations, the sample 3D representation includes 2D height field data representing the 3D appearance of the second portion, and reconstructing the first 3D representation includes constructing a bounding volume hierarchy of 2D UV positions and indices to map 2D coordinates of the height field of the sample 3D representation to a sub-portion of the first 3D representation, and representing the first 3D representation in the 2D height field based on the construct (e.g., using "PIFu-uvBVH" data - a BVH of 2D UV positions and indices).

[0141] Fig.15 1 is a block diagram of an exemplary device 1500. Device 1500 illustrates an exemplary device configuration for a device described herein (e.g., device 10, device 965, etc.). Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the specific implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the device 1500 includes one or more processing units 1502 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 1506, one or more communication interfaces 1508 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 1510, one or more displays 1512, one or more internal-facing and / or external-facing image sensor systems 1514, memory 1520, and one or more communication buses 1504 for interconnecting these and various other components.

[0142] In some implementations, the one or more communication buses 1504 include circuits that interconnect system components and control communications between system components. In some implementations, the one or more I / O devices and sensors 1506 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, or one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0143] In some implementations, one or more displays 1512 are configured to present a view of a physical or graphical environment to a user. In some implementations, one or more displays 1512 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS), and / or similar display types. In some implementations, one or more displays 1512 correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. For example, device 10 includes a single display. As another example, device 10 includes a display for each eye of the user.

[0144] In some implementations, the one or more image sensor systems 1514 are configured to obtain image data corresponding to at least a portion of the physical environment 105. For example, the one or more image sensor systems 1514 include one or more RGB cameras (e.g., with complementary metal oxide semiconductor (CMOS) image sensors or charge coupled device (CCD) image sensors), monochrome cameras, IR cameras, depth cameras, event-based cameras, etc. In various implementations, the one or more image sensor systems 1514 also include an illumination source that emits light, such as a flash. In various implementations, the one or more image sensor systems 1514 also include an on-camera image signal processor (ISP) that is configured to perform multiple processing operations on the image data.

[0145] Memory 1520 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, memory 1520 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. Memory 1520 optionally includes one or more storage devices remotely located from one or more processing units 1502. Memory 1520 includes non-transitory computer-readable storage media.

[0146] In some implementations, the memory 1520 or a non-transitory computer-readable storage medium of the memory 1520 stores an optional operating system 1530 and one or more instruction sets 1540. The operating system 1530 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the instruction set 1540 includes executable software defined by binary information stored in the form of electrical charge. In some implementations, the instruction set 1540 is software that can be executed by one or more processing units 1502 to implement one or more of the techniques described herein.

[0147] The instruction set 1540 includes a registration instruction set 1542, a predetermined representation instruction set 1544, a frame specific representation instruction set 1546, and a combined representation instruction set 1548. The instruction set 1540 may be embodied as a single software executable file or as a plurality of software executable files.

[0148] In some implementations, the registration instruction set 1542 can be executed by the processing unit 1502 to generate registration data from the image data. The registration instruction set 1542 (e.g., Figure 8 The enrollment instruction set 820 of the embodiment can be configured to provide instructions to the user to collect image information to generate an enrollment avatar (e.g., enrollment image data 824) and determine whether additional image information is needed to generate an accurate enrollment avatar to be used by the avatar display process. For these purposes, in various specific implementations, the instructions include instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0149] In some implementations, the predetermined representation instruction set 1544 (e.g., Figure 8 The predetermined representation instruction set 830) can be executed by the processing unit 1502 to generate a 3D representation of the user (e.g., PIFu information) based on the registration information through one or more of the techniques discussed herein or other possible appropriate techniques. For these purposes, in various specific implementations, the instructions include instructions and / or logic for the instructions and heuristics and metadata for the heuristics.

[0150] In some implementations, the frame-specific representation instruction set 1546 (e.g., Figure 8 The frame-specific representation instruction set 840) can be executed by the processing unit 1502 to parameterize the user's facial features and eye gaze characteristics (e.g., generate appearance values ​​and depth values) by using one or more of the techniques discussed herein or other techniques that may be appropriate. For these purposes, in various specific implementations, the instructions include instructions and / or logic for the instructions as well as heuristics and metadata for the heuristics.

[0151] In some implementations, the combined representation instruction set 1548 (e.g., Figure 8 The combined representation instruction set 850 of the embodiment can be executed by the processing unit 1502 to generate and display a combined representation (e.g., a 3D avatar) of the user's face based on the predetermined representation (e.g., PIFu data) and the second data set (e.g., parameterized data). For these purposes, in various specific implementations, the instructions include instructions and / or logic for the instructions as well as heuristics and metadata for the heuristics.

[0152] Although the instruction set 1540 is shown as residing on a single device, it should be understood that in other implementations, any combination of elements may be located in separate computing devices. Fig.15 Rather, it serves as a functional description of various features present in a particular implementation that are different from the structural schematics of the implementations described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. The actual number of instruction sets and how features are distributed among them will vary depending on the implementation and may depend in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.

[0153] Fig.16 A block diagram of an exemplary head mounted device 1600 according to some implementations is shown. The head mounted device 1600 includes a housing 1601 (or shell) that houses various components of the head mounted device 1600. The housing 1601 includes (or is coupled to) eye pads (not shown) disposed at the proximal (of the user 25) end of the housing 1601. In various implementations, the eye pads are plastic or rubber pieces that comfortably and snugly hold the head mounted device 1600 in place on the face of the user 25 (e.g., around the eyes of the user 25).

[0154] Housing 1601 houses display 1610, which displays images, emits light toward or onto the eyes of user 25. In various implementations, display 1610 emits light through an eyepiece having one or more optical elements 1605, which refract the light emitted by display 1610, so that the display appears to user 25 as a virtual distance farther than the actual distance from the eye to display 1610. For example, optical element 1605 may include one or more lenses, waveguides, other diffractive optical elements (DOEs), etc. In order to enable user 25 to focus on display 1610, in various implementations, the virtual distance is at least greater than the minimum focal length of the eye (e.g., 7 cm). In addition, in order to provide a better user experience, in various implementations, the virtual distance is greater than 1 meter.

[0155] Housing 1601 also houses a tracking system that includes one or more light sources 1622, camera 1624, camera 1632, camera 1634, and controller 1680. One or more light sources 1622 emit light onto the eyes of user 25, which is reflected as a light pattern (e.g., a flashing circle) that can be detected by camera 1624. Based on the light pattern, controller 1680 can determine the eye tracking characteristics of user 25. For example, controller 1680 can determine the gaze direction and / or blink state (eyes open or closed) of user 25. As another example, controller 1680 can determine the center of the pupil, the size of the pupil, or the focus point. Therefore, in various specific implementations, light is emitted by one or more light sources 1622, reflected from the eyes of user 25, and detected by camera 1624. In various specific implementations, the light from the eyes of user 25 reflects from a hot mirror or passes through an eyepiece before reaching camera 1624.

[0156] The display 1610 emits light within a first wavelength range, and the one or more light sources 1622 emit light within a second wavelength range. Similarly, the camera 1624 detects light within the second wavelength range. In various implementations, the first wavelength range is a visible wavelength range (e.g., a wavelength range of approximately 400 nm-700 nm within the visible spectrum), and the second wavelength range is a near infrared wavelength range (e.g., a wavelength range of approximately 700 nm-1400 nm within the near infrared spectrum).

[0157] In various implementations, eye tracking (or specifically, determined gaze direction) is used to enable user interaction (e.g., user 25 selects an option on display 1610 by looking at it), provide perforated rendering (e.g., presenting a higher resolution in the area of ​​display 1610 that user 25 is looking at and presenting a lower resolution elsewhere on display 1610), or correct distortion (e.g., for images to be provided on display 1610). In various implementations, one or more light sources 1622 emit light toward the eyes of user 25, which is reflected in the form of multiple flashes.

[0158] In various implementations, camera 1624 is a frame / shutter based camera that generates images of the user's 25 eyes at a particular point in time or multiple points in time at a frame rate. Each image includes a matrix of pixel values ​​corresponding to pixels of the image, the pixels corresponding to locations of the camera's light sensor matrix. In a specific implementation, each image is used to measure or track pupil dilation by measuring changes in pixel intensity associated with one or both of the user's pupils.

[0159] In various implementations, camera 1624 is an event camera comprising multiple light sensors (e.g., a light sensor matrix) at multiple corresponding positions, which generates an event message indicating a specific position of a specific light sensor in response to the specific light sensor detecting a change in light intensity.

[0160] In various implementations, camera 1632 and camera 1634 are frame / shutter based cameras that can generate images of the face of user 25 at a specific point in time or multiple points in time at a frame rate. For example, camera 1632 captures an image of the user's face below the eyes, and camera 1634 captures an image of the user's face above the eyes. The images captured by camera 1632 and camera 1634 may include light intensity images (e.g., RGB) and / or depth image data (e.g., time of flight, infrared, etc.).

[0161] It should be understood that the specific implementations described above are cited by way of example, and the present disclosure is not limited to what has been particularly shown and described above. On the contrary, the scope includes both combinations and sub-combinations of the various features described above, as well as variations and modifications of the various features that would occur to a person skilled in the art upon reading the foregoing description and that are not disclosed in the prior art.

[0162] As described above, one aspect of the present technology is to collect and use physiological data to improve the user's electronic device experience in interacting with electronic content. The present disclosure contemplates that, in some cases, the collected data may include personal information data that uniquely identifies a specific person or can be used to identify the interests, characteristics, or tendencies of a specific person. Such personal information data may include physiological data, demographic data, location-based data, phone numbers, email addresses, home addresses, device characteristics of personal devices, or any other personal information.

[0163] The present disclosure recognizes that the use of such personal information data in the present invention technology can be used to benefit users. For example, personal information data can be used to improve the interaction and control capabilities of electronic devices. Therefore, the use of such personal information data enables planned control of electronic devices. In addition, the present disclosure also anticipates other uses of personal information data that benefit users.

[0164] The disclosure also contemplates that entities responsible for the collection, analysis, disclosure, transmission, storage or other purposes of such personal information and / or physiological data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to the use of privacy policies and practices that are recognized as meeting or exceeding the industry or government requirements for the privacy and security of maintaining personal information data. For example, personal information from users should be collected for the legal and reasonable purposes of entities, and not shared or sold outside these legal purposes. In addition, such collection should only be carried out after the user's informed consent. In addition, such entities should take any required steps to safeguard and protect access to such personal information data, and ensure that other people who can access personal information data comply with their privacy policies and procedures. In addition, such entities can subject themselves to third-party assessments to prove that they comply with widely accepted privacy policies and practices.

[0165] Regardless of the foregoing, the present disclosure also contemplates specific implementations in which users selectively prevent the use or access of personal information data. That is, the present disclosure contemplates that hardware elements or software elements may be provided to prevent or block access to such personal information data. For example, with respect to a content delivery service customized for a user, the technology of the present invention may be configured to allow a user to choose to "opt in" or "opt out" of participating in the collection of personal information data during registration for the service. In another example, a user may choose not to provide personal information data for a target content delivery service. In yet another example, a user may choose not to provide personal information but allow anonymous information to be transmitted for use in improving the functionality of the device.

[0166] Thus, while the present disclosure broadly covers the use of personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may also be implemented without access to such personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data. For example, content may be selected and delivered to a user by inferring preferences or settings based on non-personal information data or an absolute minimum amount of personal information, such as content requested by a device associated with the user, other non-personal information available to a content delivery service, or publicly available information.

[0167] In some embodiments, data is stored using a public / private key system that only allows the owner of the data to decrypt the stored data. In some other implementations, data can be stored anonymously (e.g., without identification and / or personal information about the user, such as legal name, username, time and location data, etc.). In this way, other users, hackers, or third parties cannot determine the identity of the user associated with the stored data. In some implementations, a user can access his or her stored data from a user device that is different from the user device used to upload the stored data. In these cases, the user may need to provide login credentials to access the data they store.

[0168] Numerous specific details are set forth herein to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will appreciate that the claimed subject matter may be practiced without these specific details. In other instances, methods, devices, or systems known to those of ordinary skill are not described in detail so as not to obscure the claimed subject matter.

[0169] Unless otherwise specifically noted, it should be understood that throughout the specification, discussions utilizing terms such as "process," "compute," "calculate," "determine," and "identify" refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.

[0170] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the subject matter of the present invention. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software for programming or configuring a computing device.

[0171] The specific implementation of the method disclosed herein can be performed in the operation of such a computing device. The order of the boxes presented in the above examples can be changed, for example, the boxes can be reordered, combined or divided into sub-boxes. Some boxes or processes can be executed in parallel.

[0172] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. In addition, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbers included herein are for ease of explanation only and are not intended to be limiting.

[0173] It will also be understood that, although the terms "first", "second", etc. may be used to describe various objects in this article, these objects should not be limited by these terms. These terms are only used to distinguish one object from another object. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0174] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "a" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the terms "including" or "comprising" when used in this specification specify the presence of stated features, integers, steps, operations, objects or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, objects, components or their groupings.

[0175] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

[0176] The foregoing description and summary of the present invention should be understood to be illustrative and exemplary in every aspect, rather than restrictive, and the scope of the present invention disclosed herein is determined not only by the detailed description of the exemplary specific implementations, but by the full breadth allowed by the patent law. It should be understood that the specific implementations shown and described herein are only illustrative of the principles of the present invention, and that various modifications can be implemented by those skilled in the art without departing from the scope and essence of the present invention.

Claims

1. A method comprising: At the device's processor: obtaining a first three-dimensional (3D) representation of a first portion of a user; obtaining a sequence of frame-specific second 3D representations over a time period, wherein each of the frame-specific second 3D representations represents a second portion of the user; and By modifying the first 3D representation and the corresponding frame-specific second 3D representation, a combined 3D representation of the user during the time period is generated. 2 . The method of claim 1 , wherein modifying the first 3D representation and the corresponding frame-specific second 3D representation comprises adjusting a sub-portion of the first 3D representation. 3 . The method of claim 1 , wherein modifying the first 3D representation and the corresponding frame-specific second 3D representation comprises adjusting positions of vertices of the first 3D representation and applying a texture based on each of the frame-specific second 3D representations. 4 . The method of claim 1 , wherein modifying the first 3D representation and the corresponding frame-specific second 3D representation comprises blending predetermined vertex positions of the first 3D representation and frame-specific data vertex positions on each of the frame-specific second 3D representations. 5 . The method of claim 1 , wherein each of the frame-specific second 3D representations is defined using a height value specified within a height field positionally associated with the 3D reference point. 6 . The method of claim 5 , wherein vertices of the first 3D representation are mapped to positions on the height field and associated with height values, wherein the vertices are mapped based on the 3D reference point. 7 . The method of claim 6 , wherein modifying a first 3D representation and a corresponding frame-specific second 3D representation comprises adjusting at least some of the height values ​​associated with the vertices based on the height values ​​specified for the corresponding frame-specific second 3D representation. 8 . The method of claim 7 , wherein adjusting the vertices comprises blending at least some of the height values ​​associated with the vertices with the height values ​​specified for the corresponding frame-specific second 3D representation using a specified alpha value. 9 . The method of claim 8 , wherein adjusting the vertices comprises blending at least some texture values ​​associated with the vertices with texture values ​​specified for the corresponding frame-specific second 3D representation using the specified alpha value.

10. The method of claim 1, wherein the second portion of the user is a sub-portion of the first portion.

11. The method according to claim 10, wherein: The first portion of the user includes a facial portion and an additional portion of the user; and The second portion of the user includes the facial portion but does not include the additional portion of the user.

12. The method of claim 1 , wherein generating the combined 3D representation of the user is based on an alignment associated with 3D reference points.

13. The method of claim 12, wherein the 3D reference point is associated with a 3D position of an atlas joint of a skeletal representation of the user.

14. The method of claim 13, wherein the 3D reference point is associated with a center point associated with an eye of the user, wherein the center point is defined at a location offset from the atlas joint.

15. The method of claim 1, wherein the first portion represented by the first 3D representation represents an upper torso and a head of the user. 16 . The method of claim 1 , wherein the first 3D representation is generated using a Pixel Aligned Implicit Function (PIFu) technique that locally aligns pixels of a 2D registration image with a global background to form the first 3D representation.

17. The method of claim 1, wherein each of the frame-specific second 3D representations is generated based on sensor data from an inward-facing or downward-facing camera and based on enrollment data, wherein the enrollment data includes an image of a second portion of the user exhibiting a plurality of different facial expressions.

18. The method of claim 1, wherein each of the frame-specific second 3D representations comprises a configuration of a skeletal representation of the user, wherein 3D reference points associated with the user are defined based on nodes of the skeletal representation.

19. The method of claim 1, further comprising: A view of the combined 3D representation is presented.

20. The method of claim 19, wherein the rendering occurs during a communication session corresponding to the time period, in which a second device captures sensor data and provides a sequence of frame-specific second 3D representations based on the sensor data. The method of claim 20 , wherein a second device provides a first 3D representation of the user during the communication session.

22. The method of claim 20, wherein a second device generates the combined 3D representation of the user and provides the combined 3D representation to the device during the communication session.

23. The method of claim 1, wherein the views of the combined 3D representation are displayed on the device in real time relative to the time period.

24. The method according to claim 1, wherein the device is a head mounted device (HMD).

25. An apparatus comprising: a non-transitory computer-readable storage medium; and One or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations including: obtaining a first three-dimensional (3D) representation of a first portion of a user; obtaining a sequence of frame-specific second 3D representations over a time period, wherein each of the frame-specific second 3D representations represents a second portion of the user; and By modifying the first 3D representation and the corresponding frame-specific second 3D representation, a combined 3D representation of the user during the time period is generated.

26. A non-transitory computer-readable storage medium storing program instructions executable on a device comprising one or more processors to perform operations comprising: obtaining a first three-dimensional (3D) representation of a first portion of a user; obtaining a sequence of frame-specific second 3D representations over a time period, wherein each of the frame-specific second 3D representations represents a second portion of the user; and By modifying the first 3D representation and the corresponding frame-specific second 3D representation, a combined 3D representation of the user during the time period is generated.

27. A method comprising: At the device's processor: obtaining a first three-dimensional (3D) representation of the object; obtaining a sequence of frame-specific second 3D representations of at least a portion of the object over a time period; as well as A combined 3D representation of the object within the time period is generated based on an alignment between the first 3D representation and a corresponding frame-specific second 3D representation, the alignment corresponding to the reference point.

Images