Synthetic hand texture
By combining facial and hand information, using sample hand texture to generate consistent hand 3D representations, the problem of accuracy and consistency of hand skin tones in the prior art is solved, and efficient and privacy protection in 3D representations is achieved .
Patent Information
- Application Number
- CN202410696233.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-05-21
- Filing Date
- 2024-05-31
- Publication Date
- 2024-12-03
AI Technical Summary
Existing 3D representation technology is difficult to provide uniform texture, especially the accuracy and consistency of the skin tone of the hand, resulting in the feeling that the hands and face do not belong to the same person in the representation, and it is difficult to lose, obstruct and inaccurate data. compensate.
By combining user's face and hand information, using sample hand textures to determine hand textures, identify a small number of skin tones, and generate or find textures based on these tones, using statistical processes such as PCA and VAE to compress sample hand textures, remove Details to protect privacy, generating 3D representations of the hand that are consistent with the face.
Ensure hand texture is consistent with facial skin tone, reduce feelings that are not part of the same person, improve efficiency, compensate for missing and inaccurate data, protect privacy, and provide accurate hand representation in real-time communication.
Smart Images

Figure CN119068108A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure generally relates to systems, methods, and devices that provide a three-dimensional (3D) representation of a user. Background Art
[0002] Existing 3D representations allow viewing of user representations. Existing 3D representation rendering techniques may not provide uniform texture, for example, with respect to adequately generating a 3D representation of a hand that includes consistent skin tone with the face and accurate skin tone with respect to the palm and back of the hand. Summary of the invention
[0003] Various specific implementations disclosed herein include devices, systems, and methods for creating a hand texture (e.g., a grid of pixel values representing the appearance of different parts of a user's hand) for providing a 3D user representation of a user's hand. The hand texture can be determined using information about the user's face and hands (e.g., the back of the hand and the palm) and sample hand textures. Using both face and hand input can help ensure that the hand texture is consistent with the face and accurate relative to the palm. Using sample hand textures can protect privacy, improve efficiency, and compensate for missing, occluded, and / or inaccurate hand data. The hand texture process can identify a relatively small number (e.g., 2, 3, 4) of skin tones (e.g., representing the colors of different surfaces of the user's skin, such as the back of the hand and the palm), and use these skin tones to generate, lookup, or interpolate textures based on a collection of sample hand textures.
[0004] Some specific implementations determine hand texture (or tone) for providing an accurate 3D user representation of a user's hand during an enrollment process. The hand texture can be determined based on data (e.g., images) associated with the user's face and hands (e.g., the dorsal (dorsal) side of the hand and the palm side of the hand) in combination with sample hand textures (e.g., real hand scans from multiple different individuals). Determining the hand texture via analyzing the texture or tone of the user's face and hands (with respect to generating a user representation such as, among other things, an avatar) can ensure that the hand texture is consistent with respect to the skin tone of the face and accurate with respect to the skin tone of the user's palm, thereby reducing or eliminating the perception that the user's hand and face do not belong to the same person. Furthermore, determining the hand texture via analyzing the texture or tone of the user's face and hands can account for different lighting conditions present during the enrollment process, thereby causing the hand texture to include inaccurate skin tone. Similarly, using sample hand textures in combination with analysis of the texture or tone of the user's face and hands (to determine the hand texture) can protect user privacy (e.g., by removing identifying user features such as, among other things, scars, tattoos, fingerprint patterns, etc.), improve efficiency, and compensate for missing, occluded, and / or inaccurate hand registration data.
[0005] Various implementations may identify one or more skin tones from different surfaces of a user. For example, two (e.g., a set of) skin tones may include a first skin tone representing the dorsal (back) side of a hand and a second skin tone representing the palm side of a hand. The two skin tones may be used to generate and / or find textures based on sample hand textures via any type of statistical process 210 (e.g., numerical optimization). Alternatively, the two skin tones may be used to generate and / or find textures based on sample hand textures via an interpolation process. Various implementations generate a set of skin tones based on a piece of facial texture and a hand registration image from pleasant facial registration data.
[0006] In some embodiments, any type of statistical process that captures the distribution of hand textures (e.g., principal component analysis (PCA), variational autoencoder (VAE), etc.) can be used to compress sample hand textures and remove details (of the sample hand textures) to protect sample data privacy and reduce storage requirements for the sample hand textures.
[0007] The hand texture may be combined to provide a hand structure associated with the physical properties of the hand (eg, bone length) to generate a 3D representation of the hand.
[0008] Various implementations determine hand texture and hand structure during the enrollment process. In this case, the hand texture and hand structure can be stored for future use, such as, in particular, during a real-time communication session.
[0009] During display of the user hand representation, the pixel values of the generated hand texture may be mapped to the vertices of a 3D hand mesh (which may be based on the current 3D position / configuration of the hand) to provide the appearance of a 3D user hand representation. The 3D hand mesh may be based on the current 3D position and / or configuration of the hand.
[0010] Various implementations enable 3D hand representations to be rendered simultaneously with (and positioned relative to) 3D head and / or face representations. In some implementations, a single process (e.g., a machine learning (ML) model) is used to generate representations of the hands and head / face.
[0011] In some implementations, a device has a processor (e.g., one or more processors) that executes instructions stored in a non-transitory computer-readable medium to perform a method. The method performs one or more steps or processes. In some implementations, at least two skin tones are obtained. The at least two skin tones correspond to colors representing at least two different parts of a user's hand. In some implementations, a hand texture representing a spatial arrangement of colors representing the appearance of the hand is determined. The hand texture can be determined based on the at least two skin tones and hand texture sample data, the hand texture sample data corresponding to a sample hand texture, the sample hand texture corresponding to at least some individuals other than the user. In some implementations, a hand structure associated with a physical attribute of the hand is determined. In some implementations, the hand structure and hand texture are provided for generating a three-dimensional (3D) representation of the hand.
[0012] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to perform or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and means for performing or causing the execution of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some illustrative implementations, some of which are shown in the accompanying drawings.
[0014] Figure 1 An example environment of a real-world environment including a device with a display is shown according to some implementations.
[0015] Figure 2 An example system is shown that generates a synthesized hand texture using skin tones of a user's palm side and back side of the hand in combination with a pleasing facial texture and a sample hand texture, in accordance with some implementations.
[0016] Figure 3 An example process is shown that uses a neural network to estimate a nominal skin tone of a user's hands during an enrollment process for generating a user representation, according to some implementations.
[0017] Figure 4 is a flow chart illustrating an exemplary method for determining a hand texture for providing a 3D representation of a user's hand and / or body, according to some implementations.
[0018] Figure 5 is a block diagram of an electronic device according to some specific implementations.
[0019] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. DETAILED DESCRIPTION
[0020] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail so as not to obscure more relevant aspects of the example implementations described herein.
[0021] Figure 1 An example environment 100 of a real-world environment 105 (e.g., a room) is shown that includes a device 10 (i.e., an HMD) having a display 15. In addition, the example environment 100 includes a device / system 104 (e.g., a server, an intermediary device, a receiving / viewing device, etc.) in communication with the device 10. In some implementations, the device 10 displays content to the user 25.
[0022] The device 10 obtains image data, depth data, motion data, hue data, and / or physiological data (e.g., pupil data, facial feature data, upper body features, arms, hands (e.g., dorsal (back) side of the hands and palm side of the hands), etc.) from the user 25 via multiple sensors (e.g., sensor 35). For example, the device 10 may obtain feature characteristic data 40a and 40b of the arms and hands (e.g., hue or color) via sensor 35 (e.g., downward-facing sensor / camera of the HMD).
[0023] In some implementations, the plurality of sensors (e.g., sensor 35) may include any number of sensors that collect data related to the appearance of the user 25. For example, when a head mounted device (HMD) is worn, one or more sensors (e.g., a camera inside the HMD) may collect pupil data for eye tracking, and one or more sensors (e.g., sensor 35) may be located on the HMD, but on an outer surface of the HMD facing the user's head / face / upper torso / arms / hands to capture facial / body feature data (e.g., facial feature feature data and / or feature feature data 40a and 40b of the upper torso, arms, hands, etc.).
[0024] The user data (eg, facial feature characteristic data, eye gaze characteristic data, and upper body, arm, and hand characteristic data 40a and 40b) may change over time and device 10 may use the user data to generate and / or provide a representation of the user.
[0025] In some implementations, the user data includes texture data of facial features, such as eyebrow movement, jaw movement, nose movement, cheek movement, etc. For example, when a person (e.g., user 25) smiles, the facial features may include a number of muscle movements that can be repeated by a representation of the user (e.g., an avatar) based on captured data from sensor 35.
[0026] In some embodiments, the user data (e.g., hand characteristic data 40a and 40b) includes texture data of facial features, such as eyebrow movement, chin movement, nose movement, cheek movement, etc. For example, when a person (e.g., user 25) smiles, the facial features may include a plethora of muscle movements that can be replicated by a representation of the user (e.g., an avatar) based on captured data from sensor 35.
[0027] In some implementations, device 10 captures user data about the appearance of user 25 and provides the user data (or a 3D user representation derived from the data) to another device so that the other device can present the 3D user representation of the user to a second user. For example, during a communication session, two users can wear HMD devices that transmit user representation data to each other so that each user can view a corresponding 3D representation of the other user, for example, using a 3D environment such as a shared XR environment. In some implementations, the 3D user representations are presented in real time (e.g., on device 10 or another device). In some implementations, the 3D user representations are recorded (e.g., as a 3D video including multiple frames of height field and appearance data) for later playback.
[0028] In some implementations, the device 10 presents an XR environment and / or a graphical user interface (GUI). In some implementations, the user 25 interacts within the XR environment and / or interacts with the GUI. In some implementations, the XR environment and / or GUI provides functions including but not limited to: image editing, drawing, rendering, word processing, website creation, disk editing, spreadsheet creation, playing games, calling, video conferencing, emailing, instant messaging, exercise support, digital photography, digital video, web browsing, digital music playback, and / or digital video playback. Executable instructions for performing these functions may be included in a computer-readable storage medium or other computer program product configured for execution by one or more processors.
[0029] Figure 2 An example system 200 is shown that uses skin tone estimation data 205 representing skin tone of a palm side 205a (e.g., including color, including multiple regional palm samples (taken from various regions of the palm) to characterize the palm with multiple colors, etc.) and skin tone of a dorsal (dorsal) side 205b of a user's hand (e.g., including color, including multiple regional dorsal (back, back of hand, etc.) side hand samples (taken from various regions of the dorsal side of the hand) to characterize the palm with multiple colors, etc.) in combination with a pleasant facial texture block 224 (representing the user's facial tone) and a sample hand texture 207 to generate an accurate synthesized hand texture 218 for providing a safe and accurate (3D) hand / user representation 228 (e.g., including the user's hand, face and / or body). The synthesized hand texture 218 can be generated during a user enrollment process. The skin tone estimation data 205 and the pleasant facial texture block 224 can be generated as follows with respect to Figure 3 The sample hand texture 207 may include a plurality (e.g., 500) of compressed hand textures 207a...207n. The compressed hand textures 207a-207n may be any suitable texture of a human hand, such as real hand image scans of hands of multiple individuals different from the user, synthetically generated images of hands, etc. Using the sample hand textures 207 to generate the synthesized hand texture 218 may compensate for missing, occluded, and / or inaccurate hand registration data retrieved during the registration process and captured by the scene-facing camera (e.g., as described in detail in the example of FIG. 2 ). Figure 3 As shown, hand images 304a and 304b are captured by one or more image sensors on the electronic device.
[0030] For example, various implementations may enable any type of statistical process 210 (e.g., numerical optimization) to use the skin tone estimation data 205 and the sample hand texture 207 to generate and / or find a reconstructed texture 214 that includes similarities to the skin tone estimation data 205. For example, the hand texture (e.g., the reconstructed texture 214) may be reconstructed relative to the distribution of hand textures in the compression model (e.g., the sample hand texture 207) that matches the color input.
[0031] Various implementations enable an interpolation process to generate a reconstructed texture 214 using the skin tone estimation data 205 and the sample hand texture 207 .
[0032] Some specific implementations enable the matching component 216 to generate a synthesized hand texture 218 from a pleasant facial texture 224 block (e.g., a facial texture block of a user's face without any lighting effects) and a reconstructed texture 214. The synthesized hand texture 218 includes a texture based on the pleasant facial texture 224 block and the reconstructed texture 214. The synthesized hand texture 218 can be applied to a hand structure / mesh 226 for generating a 3D hand / user representation 228 (e.g., an avatar including a hand representation) so that the 3D hand / user representation 228 includes a skin tone that is consistent with the facial skin tone of the actual user. The hand structure / mesh 226 can represent physical attributes of the hand (e.g., bone length). The consistent skin tone allows reducing or eliminating the feeling that the user's hand and face (in the generated user representation) do not belong to the same person. Likewise, the synthesized hand texture 218 can ensure that the synthesized hand texture 218 includes an accurate skin tone with respect to the skin tone of the palm side of the user's hand.
[0033] In some specific implementations, a principal component analysis (PCA) algorithm, a VAE algorithm, etc. may be performed to compress the sample hand texture 207 and remove the details of the sample hand texture 207 to protect the sample data privacy of the individual providing the sample hand texture and reduce the storage requirements of the sample hand texture. For example, user privacy may be protected by removing identifying user features (such as, in particular, scars, tattoos, fingerprint patterns, etc.) from the sample hand texture 207.
[0034] Various implementations may generate a hand structure associated with the physical attributes of the user's hand (e.g., bone length), which is used in conjunction with the synthesized hand texture 218 to generate a (3D) representation of the hand (e.g., 3D hand / user representation 228). The synthesized hand texture 218 and hand structure may be generated during the enrollment process and may be stored for later use, such as, inter alia, during a real-time communication session.
[0035] During display of the user's hand representation, the pixel values of the generated hand texture can be mapped to the vertices of the 3D hand mesh (which can be based on the current 3D position / configuration of the hand) to provide the appearance of a 3D user hand representation. The 3D hand mesh can be based on the current 3D position and / or configuration of the user's hand. In some implementations, the 3D mesh is modified for the user based on the physical properties of the user's hand (e.g., bone length). For example, the hand tracking system can be configured to determine the size of each finger of the user's hand, and the mesh can be warped to fit the user's hand.
[0036] Various implementations enable the 3D hand / user representation 228 to present a 3D hand representation rendered simultaneously with and positioned relative to a 3D head and / or face representation. In some implementations, a single process (e.g., a machine learning model, such as a trained neural network) is used to generate representations of the hands and head / face.
[0037] Figure 3 An example process 300 is shown that uses a neural network 308 to estimate a nominal skin tone of a user's hands regardless of lighting conditions that occur during an enrollment process for capturing data representing the user's hands to generate a representation of the user, such as an avatar. Figure 3 As a result of the process 300, skin tone estimation data 205 and pleasant facial texture blocks 224 (such as those regarding Figure 2 300 masks a portion 302a of a face in an image 302. Portion 302a represents an area that will not be used to provide a pleasing facial texture patch for generating a synthesized hand texture. The process also receives a patch 302b of unilluminated facial texture from an area above the eyes but below the hair of the user's image 302 (e.g., Figure 2 The skin tone estimation neural network 308 is used to generate (via execution of the skin tone estimation neural network 308) a skin tone estimation neural network representing the palm side 310a of the user's hand (e.g., the skin tone estimation neural network 308). Figure 2 of the palm side 205a) and the back (dorsal) side 310b (e.g., Figure 2 Skin tone estimation data 310 (eg, Figure 2The block 302b is generated by masking the upper facial region (of the face of the image 302) and extracting the facial skin tone of the region of the face that statistically has suitable light for sampling the skin tone. The block 302b can be generated by masking a portion of any facial region of the face of the image 302 that statistically has suitable light for sampling the skin tone.
[0038] The skin tone estimation neural network 308 is configured to monitor hand images 304a and 304b of the hand, which may be represented via non-ideal lighting conditions (e.g., lighting conditions that may have been dimmed during image capture and may have caused shadows). Therefore, the estimation neural network 308 can be trained on real scene-oriented sensor data (hand images 304a and 304b) and synthetic data (e.g., block 302a) to analyze the hand images 304a and 304b under non-ideal lighting conditions and generate as outputs (e.g., skin tone estimation data 310 representing the skin tone of the palm side 310a and the skin tone of the back (dorsal) side 310b of the hand) a pleasing skin tone of the back side and palm of the hand for use in generating a user representation representing the user's hand, face, and / or body (e.g., Figure 2 3D hand / user representation 228).
[0039] Figure 4 4 is a flowchart illustrating an exemplary method 400 for determining a hand texture for providing a 3D representation of a user's hand and / or body. In some implementations, the method 400 is performed by a device, such as a mobile device (e.g., Figure 1 Device 10), desktop computer, laptop computer, HMD, server / mediator device, receiving / viewing device, etc. In some specific implementations, the device has a screen for displaying images and / or a screen for viewing stereoscopic images, such as a head-mounted display (HMD, such as, for example Figure 1 In some implementations, the method 400 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 400 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., a memory). Each of the blocks in the method 400 may be enabled and executed in any order.
[0040] At block 402, the method 400 obtains at least two skin tones corresponding to colors representing at least two different portions of a user's hand. For example, the method 400 may obtain a skin tone of a palm side 205a of the hand and a skin tone of a back side 205b of the hand, such as Figure 2In some implementations, the at least two skin tones include a first skin tone (eg, color) representing a back portion of a hand of the user and a second skin tone (eg, color) representing a palm portion of a hand of the user.
[0041] In some implementations, the at least two skin tones are generated based on information representing a portion of the user's face in combination with an image of the user's hand. In some implementations, determining the at least two skin tones occurs during an enrollment process.
[0042] In some implementations, the portion of the user's face is a patch of facial texture retrieved from the pleasant face enrollment data.
[0043] At block 404, method 400 determines a hand texture, such as Figure 2 A reconstructed texture 214 is provided that represents a spatial arrangement of colors that represent the appearance of a hand. The hand texture can be determined based on at least two skin tones and hand texture sample data that corresponds to sample hand textures that correspond to, in particular, a synthetic hand image, at least some individuals other than the user, etc. In some implementations, the spatial arrangement of colors includes pixels representing multiple colors associated with different parts of the user's hand.
[0044] In some implementations, determining the hand texture can include selecting the texture from the hand texture sample data via any type of statistical process with respect to at least two skin tones.
[0045] In some implementations, determining the hand texture includes selecting a texture from the hand texture sample data via an interpolation process with respect to at least two skin tones.
[0046] In some embodiments, determining hand texture may include: compressing sample hand textures via execution of principal component analysis (PCA), VAE, etc.; and removing identifying details (e.g., scars, tattoos, etc.) from the sample hand textures via execution of PCA or VAE.
[0047] In some implementations, the hand texture is determined during the enrollment process and stored for future use. The future use includes a real-time communication session between the user and the additional user. Alternatively, the future use includes displaying a 3D representation of the user using the 3D representation of the hand.
[0048] At block 406, the method 400 determines a hand structure associated with a physical attribute of the hand (eg, bone length), such as Figure 2The hand structure / mesh 226 is shown. In some implementations, the pixel values of the hand texture are mapped to the vertices of the 3D hand mesh representing the hand structure to provide the appearance of a 3D representation of the hand. The vertices of the 3D hand mesh can be based on the current 3D position of the hand.
[0049] At block 408, the method 400 provides the hand structure and the hand texture for use in generating a three-dimensional (3D) representation of the hand as part of the 3D hand / user representation 228, such as Figure 2 The hand texture and hand structure may be determined during the enrollment process and stored for later use, such as during a real-time communication session. During display of the user representation, the pixel values of the generated hand texture may be mapped to the vertices of a 3D hand mesh (which may be based on the current 3D position / configuration of the hand) to provide the appearance of a 3D user hand representation.
[0050] Figure 5 is a block diagram of an example device 500. The device 500 shows Figure 1 Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and in order not to obscure more relevant aspects of the specific implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the device 500 includes one or more processing units 502 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 504, one or more communication interfaces 508 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE802.14x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZIGBEE, SPI, I2C, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 510, output devices (e.g., one or more displays) 512, one or more internal-facing and / or external-facing image sensor systems 514, memory 520, and one or more communication buses 504 for interconnecting these and various other components.
[0051] In some implementations, the one or more communication buses 504 include circuits that interconnect and control communications between system components. In some implementations, the one or more I / O devices and sensors 505 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), one or more cameras (e.g., an internally facing camera and an externally facing camera of the HMD), one or more infrared sensors, one or more thermal map sensors, etc.
[0052] In some implementations, one or more displays 512 are configured to present a view of a physical environment, a graphical environment, an extended reality environment, etc. to the user. In some implementations, one or more displays 512 are configured to present content (determined based on the determined user / object position of the user within the physical environment) to the user. In some implementations, one or more displays 512 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS) and / or similar display types. In some implementations, one or more displays 512 correspond to diffraction, reflection, polarization, holographic, etc. waveguide displays. In one example, the device 500 includes a single display. As another example, the device 500 includes a display for each eye of the user.
[0053] In some implementations, the one or more image sensor systems 514 are configured to obtain image data corresponding to at least a portion of the physical environment 100. For example, the one or more image sensor systems 514 include one or more RGB cameras (e.g., with complementary metal oxide semiconductor (CMOS) image sensors or charge coupled device (CCD) image sensors), monochrome cameras, IR cameras, depth cameras, event-based cameras, etc. In various implementations, the one or more image sensor systems 514 also include an illumination source that emits light, such as a flash. In various implementations, the one or more image sensor systems 514 also include an on-camera image signal processor (ISP) that is configured to perform multiple processing operations on the image data.
[0054] In some implementations, sensor data may be provided by a device (e.g., Figure 1The sensor data may include a 3D point cloud and a sequence of 2D images corresponding to a view of the room captured during the scan of the room. In some implementations, the sensor data includes image data (e.g., from an RGB camera), depth data (e.g., a depth image from a depth camera), ambient light sensor data (e.g., from an ambient light sensor), and / or motion data from one or more motion sensors (e.g., an accelerometer, a gyroscope, an IMU, etc.). In some implementations, the sensor data includes visual inertial odometry (VIO) data determined based on the image data. The 3D point cloud may provide semantic information about one or more elements of the room. The 3D point cloud may provide information about the location and appearance of a surface portion within the physical environment. In some implementations, the 3D point cloud is obtained over time (e.g., during a scan of the room), and the 3D point cloud may be updated, and an updated version of the 3D point cloud is obtained over time. For example, the 3D representation may be obtained (and analyzed / processed) as it updates / adjusts over time (eg, as a user scans a room).
[0055] In some implementations, the sensor data may be positioning information, and some implementations include VIO to determine equivalent ranging information using sequence camera images (e.g., light intensity image data) and motion data (e.g., obtained from an IMU / motion sensor) to estimate the driving distance. Alternatively, some implementations of the present disclosure may include a simultaneous localization and mapping (SLAM) system (e.g., a position sensor). The SLAM system may include a multi-dimensional (e.g., 3D) laser scanning and range measurement system that is independent of GPS and provides real-time simultaneous localization and mapping. The SLAM system may generate and manage very accurate point cloud data generated by reflections from laser scans of objects in the environment. Over time, the movement of any point in the point cloud is accurately tracked so that the SLAM system can use points in the point cloud as reference points for location, maintaining an accurate understanding of its position and orientation as it travels through the environment.
[0056] In some implementations, the device 500 includes an eye tracking system for detecting eye position and eye movement (e.g., eye gaze detection). For example, the eye tracking system may include one or more infrared (IR) light emitting diodes (LEDs), an eye tracking camera (e.g., a near infrared (NIR) camera), and an illumination source (e.g., a NIR light source) that emits light (e.g., NIR light) toward the user's eyes. In addition, the illumination source of the device 500 may emit NIR light to illuminate the user's eyes, and the NIR camera may capture images of the user's eyes. In some implementations, the images captured by the eye tracking system may be analyzed to detect the position and movement of the user's eyes, or to detect other information about the eyes such as pupil dilation or pupil diameter. In addition, the gaze point estimated from the eye tracking image may enable gaze-based interaction with content shown on the near-eye display of the device 500.
[0057] The memory 520 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some specific implementations, the memory 520 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 520 optionally includes one or more storage devices remotely located from the one or more processing units 502. The memory 520 includes a non-transitory computer-readable storage medium.
[0058] In some implementations, the memory 520 or a non-transitory computer-readable storage medium of the memory 520 stores an optional operating system 530 and one or more instruction sets 540. The operating system 530 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the instruction set 540 includes executable software defined by binary information stored in the form of electrical charge. In some implementations, the instruction set 540 is software that can be executed by one or more processing units 502 to implement one or more of the techniques described herein.
[0059] The instruction set 540 includes a hand texture determination instruction set 542 and a 3D representation generation instruction set 544. The instruction set 540 may be embodied as a single software executable file or as multiple software executable files.
[0060] The hand texture determination instruction set 542 is configured with instructions executable by the processor to determine a hand texture representing a spatial arrangement of colors that represents the appearance of a hand.
[0061] The 3D representation generation instruction set 544 is configured with instructions executable by the processor to generate a 3D representation of the user's hand based on the determined hand structure and hand texture.
[0062] Although the instruction set 540 is shown as residing on a single device, it should be understood that in other implementations, any combination of elements may be located in separate computing devices. Figure 5 Rather, it serves as a functional description of various features present in a particular implementation that are different from the structural schematics of the implementations described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. The actual number of instruction sets and how features are distributed among them will vary depending on the implementation and may depend in part on the specific combination of hardware, software, and / or firmware selected for a particular implementation.
[0063] Those of ordinary skill in the art will appreciate that well-known systems, methods, components, devices, and circuits are not described in detail to avoid obscuring more relevant aspects of the example implementations described herein. In addition, other effective aspects and / or variations do not include all of the specific details described herein. Therefore, several details are described in order to provide a thorough understanding of the example aspects shown in the accompanying drawings. In addition, the accompanying drawings only illustrate some example embodiments of the present disclosure and therefore should not be considered as limiting.
[0064] Although this specification includes many specific implementation details, these specific implementation details should not be understood as limitations on the scope of any invention or possible content to be protected, but should be understood as descriptions of features specific to a particular embodiment of a particular invention. Certain features described in the context of different embodiments in this specification may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in the form of any suitable sub-combination. In addition, although certain features may be described above as acting in certain combinations and even initially claimed in this way, one or more features of the claimed combination may be removed from the combination in some cases, and the claimed combination may involve a sub-combination or a variation of the sub-combination.
[0065] Similarly, although the operations are shown in a specific order in the accompanying drawings, this should not be understood as requiring such operations to be performed in a sequential order or in the specific order shown, or requiring the execution of all the operations shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the division of each system component in the above-mentioned embodiment should not be understood as requiring such division in all embodiments, and it should be understood that the program components and the system can be generally integrated together in a single software product or encapsulated in multiple software products.
[0066] Thus, specific embodiments of the subject matter have been described. Other embodiments are also within the scope of the following claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired results. In addition, the processes shown in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some specific implementations, multitasking and parallel processing may be advantageous.
[0067] The embodiments of the subject matter and operations described in this specification may be implemented in digital electronic circuits or in computer software, firmware or hardware (including the structures disclosed in this specification and their structural equivalents) or in a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, for example, one or more modules of computer program instructions encoded on a computer storage medium, for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or in addition, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical signal, optical signal or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them, or included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device. In addition, although the computer storage medium is not a propagation signal, the computer storage medium may be the source or destination of the computer program instructions encoded in the artificially generated propagation signal. The computer storage medium may also be, or be included in, one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices).
[0068] The term "data processing device" covers all kinds of devices, equipment and machines for processing data, such as programmable processors, computers, systems on chip, or multiple or combinations of the foregoing. The device may include a dedicated logic circuit (e.g., FPGA (field programmable gate array) or ASIC (application-specific integrated circuit)). In addition to hardware, the device may also include a code for creating an execution environment for the computer program under consideration, such as a code constituting a processor firmware, a protocol stack, a database management system, an operating system, a cross-platform operating environment, a virtual machine, or a combination of one or more of them. The device and the execution environment can implement various different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures. Unless otherwise specifically stated, it should be understood that throughout the specification, discussions using terms such as "processing", "calculating", "calculating", "determining", and "identifying" refer to the actions or processes of computing devices, such as one or more computers or similar electronic computing devices, which manipulate or convert data represented as physical electronic quantities or magnetic quantities in the memory, registers, or other information storage devices, transmission devices, or display devices of the computing platform.
[0069] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the subject matter of the present invention. Any suitable programming, scripting, or other type of language or combination of languages may be used to implement the teachings contained herein in software for programming or configuring a computing device.
[0070] The specific implementation of the method disclosed herein can be performed in the operation of such a computing device. The order of the boxes presented in the above examples can be changed, for example, the boxes can be reordered, combined and / or divided into sub-blocks. Some boxes or processes can be executed in parallel. The operations described in this specification can be implemented as operations performed by a data processing device on data stored on one or more computer-readable storage devices or received from other sources.
[0071] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. In addition, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more of the stated conditions or values may in practice be based on additional conditions or values beyond those stated. The headings, lists, and numbers included herein are for ease of explanation only and are not intended to be limiting.
[0072] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.
[0073] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "one", "a kind of" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their groupings.
[0074] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.
Claims
1. A method, comprising: At an electronic device having a processor: obtaining at least two skin tones corresponding to colors representing at least two different portions of a hand of a user; determining a hand texture comprising a spatial arrangement of colors representing an appearance of the hand, the hand texture being determined based on the at least two skin tones and the hand texture sample data; determining a hand structure associated with physical attributes of the hand; and The hand structure and the hand texture are provided for use in generating a three-dimensional (3D) representation of the hand. 2 . The method of claim 1 , wherein the at least two skin tones include a first skin tone representing a back portion of the hand of the user and a second skin tone representing a palm portion of the hand of the user. 3 . The method of claim 1 , wherein the at least two skin tones are generated based on information representing a portion of the user's face in combination with an image of the user's hand.
4. The method of claim 3, wherein determining the at least two skin tones occurs during an enrollment process, and wherein the portion of the face of the user is a patch of facial texture retrieved from pleasant facial enrollment data. 5 . The method of claim 1 , wherein the spatial arrangement of colors includes pixels representing multiple colors associated with different parts of the hand of the user.
6. The method of claim 1, wherein the determining the hand texture comprises selecting a texture from the hand texture sample data via a statistical process with respect to the at least two skin tones.
7. The method of claim 1, wherein the determining the hand texture comprises selecting a texture from the hand texture sample data via an interpolation process with respect to the at least two skin tones.
8. The method of claim 1, wherein the determining the hand texture comprises: compressing the sample hand texture via performance of principal component analysis (PCA); as well as Identifying details are removed from the sample hand texture via the execution of the PCA.
9. The method of claim 1, wherein the hand texture is determined during an enrollment process and stored for future use.
10. The method of claim 9, wherein the future use comprises a real-time communication session between the user and an additional user.
11. The method of claim 9, wherein the future use comprises displaying the 3D representation of the user using the 3D representation of the hand.
12. The method of claim 1, wherein pixel values of the hand texture are mapped to vertices of a 3D hand mesh representing the hand structure to provide the appearance of the 3D representation of the hand. The method of claim 12 , wherein the vertices of the 3D hand mesh are based on a current 3D position of the hand.
14. The method of claim 1, wherein the hand texture sample data corresponds to sample hand textures corresponding to at least some individuals other than the user.
15. The method of claim 1, wherein the at least two skin tones are determined based on one or more images of the hands and face of the user.
16. The method of claim 1, wherein the hand structure and the attributes are determined during an enrollment process, and wherein a size of a hand texture is modified and applied to the hand structure for generating the 3D representation of the hand based on the attributes.
17. An electronic device, comprising: a non-transitory computer-readable storage medium; as well as one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the apparatus to perform operations including: obtaining at least two skin tones corresponding to colors representing at least two different portions of a hand of a user; determining a hand texture representing a spatial arrangement of colors representing an appearance of the hand, the hand texture being determined based on the at least two skin tones and the hand texture sample data; determining a hand structure associated with physical attributes of the hand; and The hand structure and the hand texture are provided for use in generating a three-dimensional (3D) representation of the hand. 18 . The electronic device of claim 17 , wherein the at least two skin tones include a first skin tone representing a back portion of the hand of the user and a second skin tone representing a palm portion of the hand of the user.
19. The electronic device of claim 17, wherein the at least two skin tones are generated based on information representing a portion of the user's face in combination with an image of the user's hand.
20. A non-transitory computer-readable storage medium storing program instructions executable via one or more processors of an electronic device to perform operations including: obtaining at least two skin tones corresponding to colors representing at least two different portions of a hand of a user; determining a hand texture representing a spatial arrangement of colors representing an appearance of the hand, the hand texture being determined based on the at least two skin tones and the hand texture sample data; determining a hand structure associated with physical attributes of the hand; and The hand structure and the hand texture are provided for use in generating a three-dimensional (3D) representation of the hand.
