Electronic device having body composition analysis circuit

Abstract

To provide an electronic device that includes body composition analysis circuitry.SOLUTION: Body composition analysis circuitry estimates body composition based on captured images of a face, neck, and / or body (e.g., depth map images captured by a depth sensor, visible light and infrared images captured by image sensors, and / or other suitable images). The body composition analysis circuitry may analyze the image data and may extract portions of the image data that strongly correlate with body composition, such as portions of the cheeks, neck, waist, etc. The body composition analysis circuitry may encode the image data into a latent space. The latent space may be based on a deep learning model that accounts for facial expression and neck pose in face / neck images and that accounts for breathing and body pose in body images. The body composition analysis circuitry may output an estimated body composition based on the image data and based on user demographic information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] This relates generally to electronic devices, and in particular to electronic devices having sensors. (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Patent Application No. 17 / 865,194, filed July 14, 2022, U.S. Provisional Patent Application No. 63 / 238,714, filed August 30, 2021, and U.S. Provisional Patent Application No. 63 / 242,357, filed September 9, 2021, which are hereby incorporated by reference in their entireties. [Background technology]

[0002] Electronic devices such as mobile phones, watches, and other appliances are often equipped with sensors such as fingerprint sensors, facial recognition cameras, and heart rate sensors.

[0003] Using devices such as these can be difficult. Users may want to obtain different types of health-related information that traditional electronic devices cannot provide. Users may need to rely on multiple electronic devices to obtain the desired health-related information, which can be inconvenient and cumbersome. Summary of the Invention

[0004] The electronic device may include a body composition analysis circuit that estimates body composition based on captured images of the face, neck, and / or body (e.g., a three-dimensional depth map captured by a depth sensor, visible light and infrared images, and / or other suitable images). In some arrangements, the depth sensor in the electronic device may include an infrared light emitter that illuminates the face and neck with structured infrared light and an infrared light detector that detects the infrared light reflected from the face and neck. The depth sensor may generate depth map image data that captures the three-dimensional structured data based on the reflected infrared light. Other types of depth sensing technologies and / or visible light cameras may be used as needed to capture the face and neck image data. In some arrangements, the image may be a full body image or an image of a portion of the user's body, such as the torso or biceps.

[0005] In some arrangements, the body composition analysis circuitry can use a user study training model to map the images to body composition information. The model may be trained on images of specific body parts and / or on images of the entire body. The body composition information may describe how fat is distributed throughout the body and / or may describe the relative amount of fat in the visceral and subcutaneous compartments of the body. The body composition analysis circuitry can use the facial image to scale the body image to determine the body dimensions. The electronic device can be a head-mounted device or any other suitable electronic device worn or used by a first user while capturing an image of a second user. The electronic device can also be self-operating while capturing an image of the user. If desired, the electronic device can capture an image of the user while attached to a fixed mounting.

[0006] The body composition analysis circuit can analyze the image data and extract portions of the image data that are highly correlated with body composition, such as portions of the cheeks, neck, face, chest, waist, hips, thighs, and other areas. The body composition analysis circuit can encode the image data into a latent space. The latent space can be based on a deep learning model trained on user study data.

[0007] When using facial images, the latent space may include a first latent space representing a user identity, a second latent space representing a facial expression, and a third latent space representing a neck pose. The body composition analysis circuit can compensate for the facial expression and neck pose by using the facial and neck image data in the user identity latent space to output an estimated body composition.

[0008] When using body images, the latent space may include a first latent space representing a user identity, a second latent space representing a respiratory state, and a third latent space representing a body posture. The body composition analysis circuit can compensate for the respiratory state and body posture by using the body image data in the user identity latent space to output an estimated body composition.

[0009] The body composition analysis circuitry can use a user study training model to map the image to body composition information. The model may be trained on images of specific body parts and / or on images of the entire body. The body composition information may describe how fat is distributed throughout the body and / or may describe the relative amount of visceral and subcutaneous fat in specific body parts. The body composition analysis circuitry can use the facial image to scale the body image to determine the body dimensions. The electronic device can be a head mounted device or any other suitable electronic device worn by a first user while capturing an image of a second user. [Brief description of the drawings]

[0010] [Figure 1] 1 is a schematic diagram of an exemplary electronic device according to one embodiment. [Diagram 2] 1 is a perspective view of an exemplary electronic device having a display with an optical component window overlying an optical component, according to one embodiment. [Diagram 3] 1 is a cross-sectional side view of an exemplary electronic device having optical components such as a light source and an image sensor, according to one embodiment. [Figure 4] 1 is a cross-sectional side view of an exemplary light source including a diffuser, according to one embodiment. [Diagram 5] 5 is a front view of an exemplary object onto which a dot pattern is projected using a light source of the type shown in FIG. 4, according to one embodiment. [Figure 6] FIG. 2 is a schematic diagram of an exemplary body composition analysis circuit used to analyze face and neck images, according to one embodiment. [Figure 7] FIG. 2 is a schematic diagram of an exemplary body composition analysis circuit used to analyze body images, according to one embodiment. [Figure 8] 1 is a diagram of exemplary three-dimensional depth map image data associated with a face and neck, according to one embodiment. [Figure 9] 9 illustrates how relevant portions of three-dimensional depth map image data of the type shown in FIG. 8 can be extracted for body composition analysis, according to one embodiment. [Figure 10] 1 is a diagram of exemplary image data corresponding to a front body view, according to one embodiment. [Figure 11] 1 is a diagram of exemplary image data corresponding to a body side view, according to one embodiment. [Figure 12] 12A-12C illustrate how relevant portions of image data of the type shown in FIGS. 10 and 11 can be extracted for body composition analysis, according to one embodiment. [Figure 13] FIG. 1 illustrates how facial image data captured at different times can be aligned for body composition analysis, according to one embodiment. [Figure 14]FIG. 1 illustrates how body image data captured at different times can be registered for body composition analysis, according to one embodiment. [Figure 15] FIG. 2 is a diagram of exemplary user study data that may be collected over a period of time, according to one embodiment. [Figure 16] 1 is a flowchart of exemplary steps involved in estimating body composition based on captured image data, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] A schematic diagram of an exemplary type of electronic device in which optical components may be provided is shown in Figure 1. Electronic device 10 may be a computing device such as a laptop computer, a computer monitor with an embedded computer, a tablet computer, a mobile phone, a media player, or other handheld or portable electronic device, a smaller device such as a speaker (e.g., a voice controlled assistant or other suitable speaker), a watch device, a pendant device, a headphone or earphone device, a device integrated into glasses or other equipment worn on a user's head, or other wearable or miniature device, a television, a computer display without an embedded computer, a gaming device, a navigation device, an embedded system such as a system in which electronic equipment having a display is mounted in a kiosk or automobile, an equipment implementing the functions of two or more of these devices, or other electronic equipment.

[0012] As shown in FIG. 1, electronic device 10 may have control circuitry 16. Control circuitry 16 may include storage and processing circuitry to support operation of device 10. The storage and processing circuitry may include storage devices such as hard disk drive storage, non-volatile memory (e.g., flash memory or other electrically programmable read-only memory configured to form a solid-state drive), volatile memory (e.g., static or dynamic random access memory), etc. Processing circuitry in control circuitry 16 may be used to control operation of device 10. The processing circuitry may be based on one or more microprocessors, microcontrollers, digital signal processors, baseband processors, power management units, audio chips, application specific integrated circuits, etc.

[0013] Device 10 may have input / output circuitry, such as input / output device 12. Input / output device 12 may include user input devices that collect user input and output components that provide output to a user. Device 12 may also include communications circuitry that receives data for device 10 and provides data from device 10 to external devices. Device 12 may also include sensors that collect information from the environment.

[0014] Input / output device 12 may include one or more displays, such as display 14. Display 14 may be a touchscreen display that includes a touch sensor for collecting touch input from a user, or display 14 may not be touch-sensitive. The touch sensor for display 14 may be based on an array of capacitive touch sensor electrodes, acoustic touch sensor structures, resistive touch components, force-based touch sensor structures, optically-based touch sensors, or other suitable touch sensor devices. Display 14 may be a liquid crystal display, a light-emitting diode display (e.g., an organic light-emitting diode display), an electrophoretic display, or other display.

[0015] The input / output device 12 may include optical components 18. The optical components 18 may include light emitting diodes and other light sources. By way of example, the optical components 18 may include one or more visible light sources, such as light source 20 (e.g., light emitting diodes). The light emitting diodes 20 may provide constant illumination (e.g., to implement a flashlight function for the device 10) and / or may emit pulses of flash illumination for a visible light camera, such as a visible light image sensor 26. The optical components 18 may also include an infrared light source, such as an infrared light source 22 (e.g., a laser, a lamp, an infrared light emitting diode, an array of Vertical-Cavity Surface-Emitting Lasers (VCSELs), etc.). The infrared light source 22 may provide constant and / or pulsed illumination at an infrared wavelength, such as 940 nm, a wavelength in the range of 800-1100 nm, etc. For example, the infrared light source 22 may provide constant illumination for an infrared camera, such as an infrared image sensor 28. Infrared image sensor 28 may, by way of example, be configured to capture iris scan information from a user's eye and / or may be used to capture images for a facial recognition process implemented in control circuitry 16.

[0016] If desired, the infrared light source 22 may be used to provide flood illumination (e.g., diffuse infrared light that uniformly covers a given area) and to provide structured light (e.g., a pattern of collimated dots). The flood illumination may be used to capture infrared images of external objects (e.g., to detect the user's face and / or to create a depth map), and the structured light may be projected onto the external objects to perform depth mapping operations (e.g., to obtain a three-dimensional map of the user's face). This is merely exemplary. Other types of depth sensors (e.g., indirect time-of-flight sensors, stereo cameras, etc.) may be used as desired.

[0017] To enable the light source 22 to provide both flood illumination and structured light, the light source 22 may include a switchable diffuser and an array of collimated light sources, such as lasers, or vertical cavity surface emitting lasers. When flood illumination is desired, the diffuser may be turned on to diffuse the light from the light source. When structured illumination is desired, the diffuser may be turned off to allow the collimated light to pass through the diffuser uninhibited. The diffuser, such as the diffuser in the light source 22, may be formed from a liquid crystal material, an electrophoretic material, or other switchable light modulator. In some implementations, the light source 22 projects light through a Diffractive Optical Element (DOE) to create a replica of the dot pattern. However, this is merely exemplary. If desired, the infrared light source 22 may include a first light source that provides flood illumination and a second light source that provides structured light.

[0018] The optical components 18 may also include an optical proximity detector 24 and an ambient light sensor 30 .

[0019] The optical proximity detector 24 may include an infrared light source, such as an infrared emitting diode, and a corresponding light detector, such as an infrared photodetector, for detecting when an external object illuminated by the infrared light from the light emitting diode is in the vicinity of the device 10.

[0020] Ambient light sensor 30 may be a monochrome ambient light sensor that measures ambient light intensity, or it may be a color ambient light sensor that measures ambient light color and intensity by taking light measurements with multiple photodetectors, each of which is provided with a corresponding color filter (e.g., a color filter that passes red, blue, yellow, green, or other color light) such that each of the multiple photodetectors is responsive to ambient light in a different wavelength band.

[0021] In addition to optical components 18, input / output device 12 may include buttons, joysticks, scroll wheels, touch pads, keypads, keyboards, microphones, speakers, tone generators, vibrators, cameras, light emitting diodes and other status indicators, non-optical sensors (e.g., temperature sensors, microphones, capacitive touch sensors, force sensors, gas sensors, pressure sensors, sensors that monitor device orientation and movement, such as inertial measurement units formed from accelerometers, compasses, and / or gyroscopes), data ports, etc. A user can control the operation of device 10 by providing commands via input / output device 12, and can receive status information and other output from device 10 using the output resources of input / output device 12.

[0022] The device 10 may have a housing. The housing may form a laptop computer housing, a housing for a watch, a mobile phone housing, a tablet computer housing, or other suitable device housing. A perspective view of a portion of an exemplary electronic device is shown in FIG. 2. In the example of FIG. 2, the device 10 includes a display, such as display 14, mounted within a housing 32. The housing 32, which may often be referred to as a housing or case, may be formed from plastic, glass, ceramic, fiber composite, metal (e.g., stainless steel, aluminum, etc.), other suitable materials, or a combination of any two or more of these materials. The housing 32 may be formed using a unibody configuration in which some or all of the housing 32 is machined or molded as a single structure, or may be formed using multiple structures (e.g., an internal frame structure, one or more structures forming an external housing surface, etc.). The housing 32 may have any suitable shape. In the example of FIG. 2, the housing 32 has a rectangular exterior (footprint when viewed from above) and has four perimeter edges (e.g., opposing top and bottom edges, and opposing left and right edges). The sidewalls may extend along a perimeter of the housing 32. If desired, a strap may be coupled to a main portion of the housing 32 (e.g., in configurations in which the device 10 is a watch or a head-mounted device).

[0023] The display 14 may be protected using a display cover layer, such as a layer of clear glass, clear plastic, sapphire, or other clear layer (e.g., a transparent planar member forming part or all of the front surface of the device 10 or mounted elsewhere on the device 10). Openings may be formed in the display cover layer. For example, openings may be formed in the display cover layer to accommodate buttons, speaker ports such as speaker port 34, or other components. Openings may be formed in the housing 32 to form communication ports (e.g., audio jack ports, digital data ports, etc.), form openings for buttons, etc. In some configurations, the housing 32 may have a rear housing wall formed from a planar glass member or other transparent layer (e.g., a planar member formed on the rear surface of the device 10 opposite the front surface of the device 10 that includes the display cover layer).

[0024] Display 14 may have an array of pixels 38 (e.g., liquid crystal display pixels, organic light emitting diode pixels, electrophoretic display pixels, etc.) in active area AA. The pixels 38 of active area AA may display an image for a user of device 10. Active area AA may be rectangular, may have a notch along one or more of the edges of active area AA, may be circular, may be oval, may be rectangular with rounded corners, and / or may have other suitable shapes.

[0025] Inactive portions of the display 14, such as an inactive border area IA, may be formed along one or more edges of the active area AA. The inactive border area IA may overlie circuitry, signal lines, and other structures that do not emit light to form an image. To conceal the inactive circuitry and other components within the border area IA from view to a user of the device 10, the underside of the outermost layer (e.g., a display cover layer or other display layer) of the display 14 may be coated with an opaque masking material, such as a layer of black ink (e.g., a polymer containing black dye and / or black pigment, other colored opaque materials, etc.) and / or other layers (e.g., metals, dielectrics, semiconductors, etc.). Opaque masking materials such as these may also be formed on the interior surface of a planar rear housing wall formed from glass, ceramics, polymers, crystalline transparent materials such as sapphire, or other transparent materials.

[0026] 2, the speaker port 34 is formed from an elongated opening (e.g., a strip-shaped opening) that extends along a dimension parallel to the top periphery of the housing 32. A speaker may be mounted within the device housing 32 in alignment with the opening for the speaker port 34. During operation of the device 10, the speaker port 34 functions as an ear speaker port for a user of the device 10 (e.g., a user may place the opening 34 adjacent to the user's ear during a phone call).

[0027] Optical components 18 (e.g., visible digital image sensors, infrared digital image sensors, light-based proximity sensors, ambient light sensors, visible and / or infrared light emitting diodes providing constant and / or pulsed illumination, etc.) may be mounted under one or more optical component windows, such as optical component window 40. In the example of FIG. 2, four of the windows 40 have circular outlines (e.g., circular footprints when viewed from above) and one of the windows 40 has an elongated strip-shaped opening (e.g., elongated strip-shaped footprints when viewed from above). The elongated window 40 is mounted between the sidewall along the top periphery of the device 10 and the speaker port 34, and extends parallel to the top periphery of the housing 32. If desired, windows such as optical window 40 may have shapes other than circular and rectangular shapes. The example of FIG. 2 is merely illustrative.

[0028] An optical component window such as window 40 may be formed in an inactive area IA of display 14 (e.g., an inactive border area in the display cover layer, such as an inactive display area extending along the top periphery of housing 32) or in other portions of device 10, such as a rear housing wall portion formed from a transparent member coated with an opaque masking material, a metal housing wall portion, a polymer wall structure, etc. In the example of FIG. 2, window 40 is formed adjacent the top periphery of housing 32 between a speaker port opening 34 in the display cover layer for display 14 and a sidewall along the top edge of housing 32. In some configurations, an opaque masking layer is formed on the underside of the display cover layer in the inactive area IA, and optical window 40 is formed from an opening in the opaque masking layer. To help optical window 40 visually blend with the opaque masking layer, a thin film interference filter formed from a stack of dark ink layers, metal layers, dielectric layers, and / or other structures may be overlapping window 40.

[0029] The infrared emitters and infrared detectors in device 10 may be used to form a three-dimensional depth sensor. FIG. 3 is a side view of an exemplary depth sensor 36 in device 10 that may be used to generate a three-dimensional depth map, such as eye scan information, face images (e.g., an image of a user's face for use in performing face recognition operations to authenticate a user of device 10, an image of a user's face and neck for generating animoji, etc.), body images (e.g., an image of a user's body for use in performing motion tracking or body segmentation), and / or other three-dimensional depth mapping information. Depth sensor 36 may include an infrared light emitter 22 and an infrared light detector 28. Device 10 may use an infrared light source 22 (e.g., an infrared light emitting diode, an infrared laser, etc.) to generate infrared light 48. Light 48 may illuminate an external object in the vicinity of device 10, such as an external object 44 (e.g., a user's face and / or eyes). Infrared light 46 reflected from external objects 44 may be received and imaged using infrared digital image sensor 28 to generate an infrared image (e.g., a three-dimensional depth map) of the face and / or eyes. Depth information may also be captured by applying appropriate software algorithms to the visible and / or near infrared video and / or by using any other suitable depth sensor in the device.

[0030] The infrared light source 22 can operate in different modes depending on the type of infrared information collected by the infrared camera 28. For example, in a flood lighting mode, the light source 22 can emit diffuse light that evenly covers a desired target area. In a structured light mode, the light source 22 can emit a known pattern of light onto the desired target area.

[0031] Figure 4 illustrates illumination from light source 22 when light source 22 operates in flood illumination mode. As shown in Figure 4, light source 22 can emit diffuse infrared light 56 ​​that continuously covers a given area of ​​external object 44. Infrared camera 28 can capture an infrared image of the diffusely illuminated external object 44. In some arrangements, the flood illumination from light source 22 can be used to detect a user's face during a facial identification operation.

[0032] FIG. 5 illustrates illumination from the light source 22 when the light source 22 operates in a structured light mode. In the structured light mode, the light source 22 can project a known pattern of infrared light 56 ​​onto the external object 44. In the example of FIG. 5, the infrared light 56 ​​forms a pattern of dots on the external object 44. The dots may be in a regular grid array (e.g., uniformly spaced from each other), or the dots may be projected in a random speckle pattern. However, this is merely exemplary. If desired, the light source 22 can emit structured light in other patterns (e.g., horizontal lines, vertical lines, a grid of horizontal and vertical lines, or other suitable predetermined patterns). The structured infrared light 56 ​​of FIG. 5 may be based on laser interference or on a projected display element that emits infrared light through a spatial light modulator to create the desired pattern.

[0033] In some arrangements, light source 22 may include one light source that provides flood lighting and another light source that provides structured light. In other arrangements, the same light source may be used to provide both flood lighting and structured light. This may be accomplished using a switchable diffuser element that selectively diffuses the light emitted from the light source.

[0034] Data collected using optical components 18 may be used for one or more health-related applications, such as body composition assessment. For example, control circuitry 16 may use optical components 18 to capture images (e.g., visible images, infrared images, three-dimensional depth map images, etc.) of the user's face, neck, and / or body, which images may then be analyzed to provide user-specific body composition information, such as body mass index, body fat percentage (e.g., total body fat percentage, fat percentage in individual body parts, and / or fat percentage in different fat storage compartments, such as subcutaneous and visceral compartments), bone mass, and / or other health-related information.

[0035] The control circuitry 16 can store one or more models for mapping user image data to body composition information. The models may be statistical models, machine learning models, models based on a combination of statistical modeling and machine learning, or a combination of multiple machine learning models. Models trained using machine learning may be implemented using principal component analysis, autoencoders, and / or any other suitable data compression techniques.

[0036] An autoencoder is an artificial neural network that learns to encode data into a latent space by reducing the dimension of the data. Autoencoders are trained to encode the distribution of the input into the latent space to minimize the loss between the output and the input. Principal component analysis reduces the dimensionality of the input data by removing redundant information and capturing the most important features of the input data (e.g., the features with the highest variance). Principal component analysis is generally restricted to linear mappings, and encoders do not have linearity constraints.

[0037] 6 is a schematic diagram of a body composition analysis circuit 58 used to determine body composition from face and / or neck images. The body composition analysis circuit 58 may be part of the control circuit 16 and / or may be implemented as a stand-alone circuit. The body composition analysis circuit 58 may receive information such as face and neck image data (e.g., three-dimensional depth map data of the user's face and / or neck from the depth sensor 36, a visible image of the user's face and / or neck from the visible image sensor 26, etc.), and optional additional user data (e.g., user-specific demographic information such as gender, height, weight, age, ethnicity, and / or other user data stored in the device 10 and / or otherwise provided to the circuit 58). Based on the received face and neck image data and the received optional user demographic data, the body composition analysis circuit 58 may output estimated body composition information such as body mass index, body fat percentage, face and neck fat percentage, bone mass, and / or other health-related information. Optionally, the user demographic information may be omitted and the body composition analysis circuit 58 may estimate the user's body composition based solely on the captured face and neck image data.

[0038] If desired, the face and neck image data may be collected as part of a dedicated body composition analysis (e.g., when the depth sensor 36 is being used specifically to obtain face and neck images for a body composition analysis) and / or may be collected when the depth sensor 36 is already being used for some other purpose (e.g., when the depth sensor 36 is already being used for facial recognition and user authentication purposes, when the depth sensor 36 is already being used to create Animoji or other virtual reality applications that involve capturing a user's facial expressions, etc.). The face and neck image data may include one or more images of the face and neck captured at different times and / or across multiple days.

[0039] User demographic information may be received from the user as part of a dedicated body composition analysis questionnaire and / or may be received from the user as part of some other health-related application.

[0040] The body composition analysis circuit 58 can store models that are trained using data from user studies. For example, data may be collected from a group of participants (e.g., 1 month, 2 months, 3 months, 6 months, 8 months, 10 months, 1 year, more than 1 year, less than 1 year, etc.) over a given period of time (e.g., 10 participants, 50 participants, 100 participants, 1000 participants, and / or any other suitable number of participants). At each of the data collection points during the study, the face and neck shape and size of the study participants may be measured and the user's body composition may be measured. The face and neck shape and size may be correlated with body measurements (e.g., 100 participants, 500 participants, 1000 participants, and / or any other suitable number of participants) over a given period of time (e.g., 1 month, 2 months, 3 months, 6 months, 8 months, 10 months, 1 year, more than 1 year, less than 1 year, etc.). At each of the data collection points during the study, the face and neck shape and size of the study participants may be measured and the user's body composition may be measured. The face and neck shape and size may be correlated with body measurements (e.g., 100 participants, 500 participants, 1000 participants, and / or any other suitable number of participants). , body landmarks and measurements) and / or any other suitable measurement device (e.g., a 3-D body scanner). Body composition may be measured using any suitable body composition tracking technique, such as magnetic resonance imaging, dual energy x-ray absorptiometry, air displacement plethysmography, underwater weighing, etc. Alternatively, a model may be trained to predict fat percentage in the face and neck. Data collected during a user study may serve as training data for training a model stored in body composition analysis circuit 58 in device 10.

[0041] The body composition analysis circuit 58 can use principal component analysis, autoencoders, and / or any other suitable data compression techniques to reduce the dimensionality of the input data in the latent space. For example, the latent space can include an identity latent space describing the subject's identity, an expression latent space describing the subject's facial expression, and a pose latent space describing the subject's neck pose. By including a facial expression latent space and a neck pose latent space, the body composition analysis circuit 58 can compensate for the effects of the facial expression and neck pose by using the identity latent space only to output the subject's estimated body composition. In addition, transfer learning methods can be used to selectively enhance pre-trained machine learning models using other data.

[0042] FIG. 7 is a schematic diagram of a body composition analysis circuit 58 for determining body composition from a face and / or neck image. The body composition analysis circuit 58 may be part of the control circuit 16 and / or may be implemented as a stand-alone circuit. The body composition analysis circuit 58 may receive information such as body image data (e.g., three-dimensional depth map data of the user's body from the depth sensor 36, a visible image of the user's body from the visible image sensor 26, etc.) and optional additional user data (e.g., user-specific demographic information such as gender, height, weight, age, ethnicity, and / or other user data stored in the device 10 and / or otherwise provided to the circuit 58). Based on the received body image data and the received optional user demographic data, the body composition analysis circuit 58 may output estimated body composition information such as body mass index, body fat percentage, bone mass, and / or other health-related information. If desired, the user demographic information may be omitted and the body composition analysis circuit 58 may estimate the user's body composition based solely on the captured body image data.

[0043] The body composition analysis circuit 58 may analyze the body composition using any suitable model. In a two-compartment model, the body is assumed to be composed of two compartments, a first compartment corresponding to fat and a second compartment corresponding to everything other than fat (e.g., muscle, bone, etc.). In a three-compartment model, the body is assumed to be composed of visceral fat, subcutaneous fat, and non-fat. If desired, the body composition analysis circuit 58 may use a three-compartment model and may estimate the amount of visceral fat, subcutaneous fat, and non-fat in the user based on an image of the user. The body composition analysis circuit 58 may estimate the body composition of a specific region of the body (e.g., how much visceral fat and subcutaneous fat is located in the user's torso) or may estimate the body composition over the entire body (e.g., how the total amount of visceral fat and subcutaneous fat is distributed over the user's body).

[0044] If desired, the body image data may be collected as part of a dedicated body composition analysis (e.g., when the depth sensor 36 is specifically used to obtain body images for body composition analysis) and / or may be collected when the depth sensor 36 is already being used for some other purpose (e.g., when the depth sensor 36 is already being used for some other body scanning purpose). The body image data may include one or more images of the body captured from different views (e.g., front view, side profile view, rear view, etc.) at different times and / or across multiple days. The image data may include a series of images, such as images from footage taken while the subject is breathing and / or moving.

[0045] User demographic information may be received from the user as part of a dedicated body composition analysis questionnaire and / or may be received from the user as part of some other health-related application.

[0046] The body composition analysis circuit 58 can store models that are trained using data from user studies. For example, data may be collected from a group of participants (e.g., 1 month, 2 months, 3 months, 6 months, 8 months, 10 months, 1 year, more than 1 year, less than 1 year, etc.) over a given period of time (e.g., 10 participants, 50 participants, 100 participants, 1000 participants, and / or any other suitable number of participants). At each of the data collection points during the study, the body shape and size of the study participants may be measured and the user's body composition may be measured. Body shape and size may be measured using a three-dimensional depth sensor of the type shown in FIG. 3, body measurements (e.g., Body composition may be measured using body landmarks and measurements, and / or any other suitable measurement device (e.g., a 3-D body scanner). Body composition may be measured using any suitable body composition tracking technique, such as magnetic resonance imaging, dual energy x-ray absorptiometry, air displacement plethysmography, underwater weighing, etc. Alternatively, a model may be trained to predict fat percentage in the body. Data collected during a user study may serve as training data for training a model stored in body composition analysis circuit 58 in device 10.

[0047] The body composition analysis circuit 58 can use principal component analysis, autoencoder, and / or any other suitable data compression techniques to reduce the dimensionality of the input data in the latent space. For example, the latent space can include an identity latent space describing the subject's identity, a respiratory state latent space describing the subject's respiratory state, and a posture latent space describing the subject's body posture. By including a respiratory state latent space and a body posture latent space, the body composition analysis circuit 58 can compensate for the effects of breathing and body posture by using the identity latent space only to output the subject's estimated body composition. In addition, transfer learning methods can be used to selectively enhance pre-trained machine learning models using other data.

[0048] The model that the body composition analysis circuit 58 uses to map the image data to body composition can take into account various factors to help distinguish fat from fluid. The body composition analysis circuit 58 can use known areas of fat and water storage to distinguish between fat and fluid accumulation. For example, bags under the eyes can be an indicator of fluid retention rather than fat storage. Areas around the joints, legs, and arms tend to be areas of fluid retention rather than areas of fat storage.

[0049] 8 is a diagram illustrating example data that may be used to determine body composition when using face and / or neck image data. As shown in FIG. 8, the captured image data 60 (e.g., the captured face and neck image data) may include a three-dimensional depth map of the user's face and neck. The face and neck image data 60 may include an array of data points representing depths to different locations across the user's face and neck. The image data 60 may be captured by the depth sensor 36 of FIG. 3, if desired.

[0050] Optionally, all of the image data 60 may be used during the body composition analysis operation, or only a portion of the image data 60 may be used during the body composition analysis operation. Because body fat tends to be stored in certain fat pockets, such as areas in the cheeks and neck, these areas may be more indicative of body composition than other areas. For example, the shape of a user's forehead may exhibit slight variations as the user's body fat changes, while the cheek and neck areas may exhibit detectable changes that directly correlate to changes in body composition. Optionally, the body composition analysis circuit 58 may select a portion of the data 60, such as the data in area 62 for body composition analysis, and may delete the remaining data from the device 10. After selecting the data in area 62 and deleting the remaining data, the body composition analysis circuit 58 may proceed with the body composition analysis using the data 60' of FIG. 9.

[0051] 10 and 11 are diagrams illustrating exemplary data that may be used to determine body composition when using body images. As shown in FIG. 10, the captured image data 60 (e.g., the captured body image data 60) may include one or more three-dimensional depth maps of the user's body captured from a front view. FIG. 11 illustrates how the captured body image data 60 may include one or more three-dimensional depth maps of the user's body captured from a side profile view. The body image data 60 may include an array of data points representing depths to different locations across the user's body. The image data 60 may be captured by the depth sensor 36 of FIG. 3, if desired. However, this is merely exemplary. If desired, the image data 60 may be depth image data captured by a different type of depth sensor (e.g., a depth sensor that does not use structured light), visible light image data captured using a visible light camera, infrared image data captured by an infrared sensor, or other suitable image data.

[0052] In some arrangements, the data 60 may be collected using a sensor in the device 10 that is placed far enough away from the user to capture a full body image. For example, the device 10 may be a television with a sensor that captures image data 60 while the user is standing far enough away from the user to capture a full body image, or the device 10 may be a portable electronic device such as a mobile phone, laptop, tablet computer, or other electronic device that may be propped up in one place to capture a full body image of a user while the user is standing at a distance. If desired, the device 10 may be a head-mounted device or any other suitable electronic device that a first user (e.g., a fitness trainer) wears while looking at a second user (e.g., a client of the trainer) at a distance. The head-mounted device may have a sensor that captures image data 60 of a second user while the second user stands at a distance from the first user wearing the device 10. The electronic device may be self-operating while capturing an image of the user. If desired, the electronic device may be attached to a fixed mounting while capturing an image of the user.

[0053] In some arrangements, the data 60 may be collected by a handheld electronic device held in the user's hand (e.g., using a front image sensor in the device 10). Image distortion may be corrected using pincushion correction, keystone correction, and / or any other suitable distortion compensation technique. If necessary, an image of the user's face that does not exhibit distortion may be used to remove distortion in the full-body image. For example, the dimensions of the user's face may be determined from the undistorted face image, and the determined dimensions of the user's face may then be used to scale the full-body image so that the control circuitry 16 can determine the dimensions of the user's body based on the full-body image. Orientation information from motion sensors (e.g., accelerometers, gyroscopes, compasses, etc.) in the device 10 may also be used to remove distortion from the full-body image to obtain a more accurate picture of the size of the user's body. An arrangement in which the body composition analysis circuitry 58 stitches together multiple pictures of different parts of the body may also be used.

[0054] In some arrangements, image data 60 may include images of only a portion of the user's body. For example, image data 60 may be torso image data including images of only the user's torso, biceps image data including images of the user's biceps, legs image data including images of only the user's legs, and / or other suitable image data. An image of a portion of the user's body may be used to determine body composition in that particular portion of the user's body (e.g., to track visceral and / or subcutaneous fat in the torso, biceps, etc.).

[0055] Optionally, all of the image data 60 may be used during the body composition analysis operation, or only a portion of the image data 60 may be used during the body composition analysis operation. Because body fat tends to be stored in certain fat pockets, such as areas in the face, neck, waist, hips, and thighs, these areas may be more indicative of body composition than other areas. For example, the shape of a user's forehead may exhibit slight variations as the user's body fat changes, while areas of the cheeks, neck, and waist may exhibit detectable changes that directly correlate to changes in body composition. Optionally, the body composition analysis circuit 58 may determine which portions of the data 60 correspond to areas of the body that are strongly correlated with body composition, such as the data in region 62, and may delete the remaining data from the device 10. After selecting the data in region 62 and deleting the remaining data, the body composition analysis circuit 58 may proceed with the body composition analysis using the data 60' of FIG. 12.

[0056] Optionally, the body composition analysis circuit 58 can track changes in body composition over time by comparing face, neck, and / or body images captured at different times. FIG. 13 is a diagram illustrating how the body composition analysis circuit 58 can compare face images captured at different times. As shown in FIG. 13, image 64 can represent an image captured by the depth sensor 36 at a first time, and image 64' can represent an image captured by the depth sensor 36 at a second time. To track body composition changes between the first time and the second time, the body composition analysis circuit 58 can align portions of image 64 and image 64' that are expected to change least over time. For example, a user's eyes, nose, ears, and / or other facial features may exhibit slight changes over time and therefore can serve as good anchors for aligning images captured at different times. 13, for example, body composition analysis circuitry 58 can align eye 64E in image 64 with eye 64E' in image 64', thereby allowing body composition analysis circuitry 58 to more accurately track changes in shape and size of other areas of the user's face, such as cheeks and neck. Optionally, body composition analysis circuitry 58 can store models that map changes in face and neck shape and size to changes in body composition (e.g., optionally, body composition analysis circuitry 58 can map differences between image 64 and image 64' to corresponding changes in body fat).

[0057] FIG. 14 is a diagram illustrating how the body composition analysis circuit 58 may compare body images captured at different times. As shown in FIG. 14, image 64 may represent an image captured by the depth sensor 36 at a first time, and image 64' may represent an image captured by the depth sensor 36 at a second time. To track body composition changes between the first time and the second time, the body composition analysis circuit 58 may align portions of the image 64 and portions of the image 64' that are expected to change least over time. For example, the user's eyes, nose, ears, other facial features, and limb and / or skeletal length may exhibit slight changes over time and therefore serve as good anchors for aligning images captured at different times. As shown in FIG. 14, for example, the body composition analysis circuit 58 may align the eye 64E of the image 64 with the eye 64E' of the image 64', thereby allowing the body composition analysis circuit 58 to more accurately track changes in the shape and size of other areas of the body, such as the user's face, neck, and waist. Optionally, body composition analysis circuitry 58 can store a model that maps changes in body shape and size to changes in body composition (e.g., optionally, body composition analysis circuitry 58 can map differences between images 64 and 64' to corresponding changes in body fat).

[0058] FIG. 15 is a diagram illustrating how data may be collected during one or more user studies to train a model stored in the body composition analysis circuit 58 in the device 10. As shown in FIG. 15, data 66 may be collected from a given user population at time t0, time t1, time t2, etc. through time tn. The data 66 may include measurements of the participants' face, neck, and / or body shape and size, as well as measurements of the participants' body fat. The data 66 may be collected once a day, once a week, once a month, or at any other suitable cadence throughout the user study. The study may span a period of one month, two months, three months, six months, eight months, ten months, one year, more than one year, less than one year, etc.

[0059] At each of the data collection time points during the study (e.g., at times t0, t1, t2, ... tn), the face, neck, and / or body shape and size of each participant may be measured, and the participant's body composition may be measured. The training data may include total body measurements and / or may include body segment measurements (e.g., biceps measurements, torso measurements, leg measurements, etc.). Training a model stored in device 10 based on the body segment data may enable a user to track changes in specific body parts. For example, a user may use device 10 to take a picture of the user's biceps, and body composition analysis circuitry 58 may map the image of the biceps to a muscle mass value based on the biceps training data included in data 66.

[0060] Face, neck, and / or body shape and size may be measured using a three-dimensional depth sensor of the type shown in FIG. 3, using anthropometric measurements (e.g., body landmarks and measurements), and / or using any other suitable measurement device (e.g., a three-dimensional body scanner). Body composition may be measured using any suitable body composition tracking technique, such as magnetic resonance imaging, dual energy x-ray absorptiometry, air displacement plethysmography, underwater weighing, etc. Fat data may be measured using localized methods, such as, for example, magnetic resonance imaging or dual energy x-ray absorptiometry (e.g., to obtain body fat for body parts such as the limbs, torso, lower abdomen, upper abdomen, chest, neck, head, and face). Data 66 collected during a user study may serve as training data for training a model stored in body composition analysis circuitry 58 in device 10. If desired, images of a given participant's face, neck, and / or body captured at different times throughout the study may be aligned and compared using techniques of the type described in connection with FIGS. 13 and 14.

[0061] FIG. 16 is a flowchart of exemplary steps involved in estimating a user's body composition based on images captured during operation of device 10.

[0062] During the operation of block 100, the body composition analysis circuitry 58 may use one or more optical components 18 in the device 10 to capture one or more images of the user's face, neck, and / or body. For example, the depth sensor 36 may capture a three-dimensional depth map image of the user's face, neck, and / or body, the visible image sensor 26 may capture a visible image of the user's face, neck, and / or body, and / or other optical components 18 in the device 10 may be used to collect image data of the user's face, neck, and / or body.

[0063] For body images, image data may be captured by scanning the body from head to toe, capturing the entire body in one image frame, and / or capturing multiple image frames of different parts of the body, such as the face, neck, waist, legs, etc. The body image data may include a body front view and a body side view, as shown in Figures 10 and 11. Image data may also be captured in a series of images, such as images from a video taken while the subject is breathing and / or moving.

[0064] The face, neck, and / or body image data may be collected as part of a dedicated body composition analysis (e.g., when the depth sensor 36 is specifically used to obtain face, neck, and / or body images for a body composition analysis) and / or may be collected when the depth sensor 36 is already being used for some other purpose (e.g., when the depth sensor 36 is already being used for facial recognition and user authentication purposes, when the depth sensor 36 is already being used to create Animoji or other virtual reality applications that involve capturing a user's facial expressions, etc.). The face, neck, and / or body image data may include one or more images of the face, neck, and / or body captured from different perspectives at different times and / or across multiple days.

[0065] During the operation of block 102, body composition analysis circuitry 58 may analyze the image captured during block 100 and may identify which regions of the captured image are relevant for body composition analysis. This may include identifying which regions of the image data correspond to regions that are strongly correlated with body composition (e.g., regions 62 of FIGS. 8, 9, 10, 11, and 12). For example, image data corresponding to face regions (e.g., cheek regions), neck regions, and / or waist regions may be saved for body composition analysis. Regions of the image data that are not strongly correlated with body composition may be deleted or otherwise unused, if desired.

[0066] During the operation of block 104, body composition analysis circuitry 58 may encode the relevant image data identified during block 102 into a latent space. This may include reducing the dimensionality of the image data using autoencoders, principal component analysis, and / or other data compression techniques. For example, if the relevant image data for body composition includes thousands of data points, body composition analysis circuitry 58 may compress the relevant image data to (by way of example) a few hundred data points.

[0067] For face and neck images, the latent space may include an identity latent space that describes the subject's identity, an expression latent space that describes the subject's facial expression, and a pose latent space that describes the subject's neck pose. The latent space may be based on statistical modeling, deep learning techniques (e.g., autoencoders, primary component analysis, etc.), and / or a combination of statistical modeling and deep learning.

[0068] For body images, the latent space may include an identity latent space that describes the subject's identity, a respiratory state latent space that describes the subject's respiratory state, and a body posture latent space that describes the subject's body posture. The latent space may be based on statistical modeling, deep learning techniques (e.g., autoencoders, primary component analysis, etc.), and / or a combination of statistical modeling and deep learning.

[0069] During the operations of block 106, the body composition analysis circuit 58 can compensate for the effects of facial expressions and neck posture in the face / neck image by extracting only the identity latent space (e.g., removing the expression latent space and the neck posture latent space). For the body image, the body composition analysis circuit 58 can compensate for the effects of breathing and body posture by extracting only the identity latent space (e.g., removing the breathing state latent space and the body posture latent space).

[0070] During the operation of block 108, the body composition analysis circuit 58 can estimate body composition based on the image data in the identity latent space. For example, using a model trained on one or more user studies (e.g., as described in connection with FIGS. 6 and 7), the body composition analysis circuit 58 can map the compressed image data (e.g., a compressed data set representing the size and / or shape of the user's cheeks and neck, a compressed data set representing the size and / or shape of the user's waist, etc.) to body composition information such as body mass index, body fat percentage (e.g., total body fat percentage, fat percentage in individual body parts, and / or fat percentage in different fat storage compartments such as subcutaneous and visceral compartments), bone mass, and / or other health-related information. The body composition information provided by the body composition analysis circuit 58 can be an estimated current body composition value (e.g., a body fat percentage value, a body mass index value, or a bone mass value) and / or an estimated change in a body composition parameter (e.g., the amount of increase or decrease in a given body composition parameter such as body mass index, body fat percentage, bone mass, etc.). Optionally, the body composition analysis circuitry 58 may also take into account any available user demographic information (e.g., gender, height, weight, age, ethnicity, and / or other user data stored on the device 10 and / or otherwise provided to the circuitry 58) to determine the user's body composition.

[0071] Body composition analysis circuitry 58 may analyze body composition using any suitable model. In a two-compartment model, the body is assumed to be composed of two compartments, a first compartment corresponding to fat and a second compartment corresponding to everything other than fat (e.g., muscle, bone, etc.). In a three-compartment model, the body is assumed to be composed of visceral fat, subcutaneous fat, and non-fat. If desired, body composition analysis circuitry 58 may use a three-compartment model and estimate the amount of visceral fat, subcutaneous fat, and non-fat in the user based on an image of the user.

[0072] During the operation of block 108, the body composition analysis circuit 58 may estimate the body composition of a specific region of the body (e.g., how much visceral fat and subcutaneous fat is located in the user's torso, biceps, or other body parts), or may estimate the body composition across the entire body (e.g., how the visceral fat and subcutaneous fat are distributed across the user's body).

[0073] The operations of block 108 may include removing distortions (e.g., perspective distortion created when a user points a front camera downward to capture the entire body in one frame) from the body image. The body composition analysis circuit 58 may also use an image of the user's face (e.g., a face image collected during a user identification operation and / or a previously collected face image, such as a face image captured specifically for body composition analysis) to scale the entire body image (e.g., body dimensions may be determined based on the entire body image and the face image using the face image for scaling). Orientation information from motion sensors (e.g., accelerometers, gyroscopes, compasses, etc.) in the device 10 may also be used to remove distortions from the entire body image to obtain a more accurate picture of the size of the user's body. An arrangement in which the body composition analysis circuit 58 stitches together multiple pictures of different parts of the body may also be used.

[0074] During the operations of block 110, the control circuitry 16 can take actions in response to the analysis results. For example, the device 10 can provide the assessment results to a user of the device 10 and / or issue an alert for the user of the device 10 (e.g., if the assessment results suggest a risk of disease). In general, notifications can be issued, databases can be updated, recommendations can be provided, and / or other actions can be taken based on the results of the sensor processing operations of block 108. For example, the display 14 can display an estimated body fat percentage value, a body mass index value, a bone mass value, and / or other information determined by the body composition analysis circuitry 58. The notifications can include text notifications, audible alerts, email messages, annotated images, other on-screen notification content on the display 14, and / or other notification content.

[0075] As noted above, one aspect of the present technology is the collection and use of information, such as information from input / output devices. The present disclosure contemplates that in some cases, data may be collected that includes personal information data that uniquely identifies a particular person or that can be used to contact or locate a particular person. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, Twitter IDs, home addresses, data or records regarding a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), birth date, username, password, biometric information, or any other identifying or personal information.

[0076] This disclosure recognizes that the use of such personal information in the present technology may be used to the benefit of the user. For example, personal information data may be used to deliver targeted content that is more interesting to the user. Thus, use of such personal information data allows the user greater control over the content that is delivered. Additionally, other uses of personal information data that benefit the user are also contemplated by this disclosure. For example, health and fitness data may be used to provide insight into the user's overall wellness, or as proactive feedback to individuals using the technology to pursue wellness goals.

[0077] This disclosure contemplates that entities involved in the collection, analysis, disclosure, transmission, storage, or other use of such personal information data will establish and adhere to privacy policies and / or practices. In particular, such entities should implement and consistently use privacy policies and practices that are generally recognized as meeting or exceeding industry or government requirements for keeping personal information data private and secure. Such policies should be easily accessible by users and should be updated as data collection and / or use changes. Personal information from users should be collected for legitimate and reasonable uses of the entity and should not be shared or sold outside of these legitimate uses. Furthermore, such collection / sharing should occur after receiving the user's informed consent. In addition, such entities should consider taking the steps necessary to protect and secure access to such personal information data and to ensure that others having access to the personal information data adhere to their own privacy policies and procedures. Furthermore, such entities may undergo a third-party assessment to attest to the entity's adherence to widely accepted privacy policies and practices. In addition, policies and practices should be tailored to the particular type of personal information data collected and / or accessed and should comply with applicable laws and standards, including jurisdiction-specific considerations. For example, in the United States, the collection of or access to certain health data may be governed by federal and / or state laws, such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to, and should be handled accordingly, other regulations and policies. Thus, different privacy practices should be maintained for different personal data types in each country.

[0078] Notwithstanding the above, the present disclosure also contemplates embodiments in which a user selectively prevents use of or access to personal information data. That is, the present disclosure contemplates that hardware and / or software elements may be provided to prevent or block access to such personal information data. For example, the present technology may be configured to allow a user to select "opt-in" or "opt-out" of participating in the collection of personal information data during registration for the service or at any time thereafter. In another example, a user may choose not to provide certain types of user data. In yet another example, a user may choose to limit the amount of time that user-specific data is maintained. In addition to providing "opt-in" and "opt-out" options, the present disclosure contemplates providing notifications regarding access or use of personal information. For example, a user may be notified upon downloading an application ("app") in which the user's personal information data will be accessed, and then may be reminded again just before the personal information data is accessed by the app.

[0079] It is also contemplated in this disclosure that personal information data should be managed and processed to minimize the risk of unintended or unauthorized access or use. Risk may be minimized by limiting collection of data and deleting data when it is no longer needed. Additionally and when applicable, including in certain health-related applications, data anonymization may be used to protect user privacy. Anonymization may be facilitated when appropriate by removing specific identifiers (e.g., birthdate, etc.), controlling the amount or specificity of data stored (e.g., collecting location data at the city level rather than the address level), controlling how data is stored (e.g., aggregating data across users), and / or other methods.

[0080] Thus, while the present disclosure broadly includes using information, which may include personal information data, to implement one or more various disclosed embodiments, the present disclosure also contemplates that the various embodiments may also be implemented without requiring access to personal information data, i.e., various embodiments of the present technology are not rendered inoperable due to the absence of all or a portion of such personal information data.

[0081] According to one embodiment, an electronic device is provided that includes a sensor for capturing an image of the face and neck, and a body composition analysis circuit for encoding a portion of the image into a latent space and mapping the portion of the image to a fat percentage value.

[0082] According to another embodiment, the latent space is based on a statistical model.

[0083] According to another embodiment, the latent space is based on a deep learning model.

[0084] According to another embodiment, the sensor includes a depth sensor and the image includes a three-dimensional depth map image, the depth sensor including an infrared light source that illuminates the face and neck with structured light and an infrared light detector that detects the structured light reflected from the face and neck.

[0085] According to another embodiment, the latent space includes a first latent space representing a user identity, a second latent space representing a facial expression, and a third latent space representing a neck pose.

[0086] According to another embodiment, the body composition analysis circuit compensates for facial expression and neck pose by using the first latent space to map portions of the image to percentage fat values.

[0087] According to another embodiment, the portion of the image includes a cheek portion and a neck portion.

[0088] According to another embodiment, the body composition analysis circuitry removes remaining portions of the image that are not encoded in the latent space.

[0089] According to another embodiment, the body composition analysis circuitry maps a portion of the image to a percentage fat value based at least in part on user demographic data.

[0090] According to another embodiment, the sensor includes a visible light camera and the image includes a visible light image.

[0091] According to one embodiment, an electronic device is provided that includes an infrared light source that illuminates a face with structured light, an infrared light detector that detects the structured light reflecting from the face and outputs corresponding depth information, a body composition analysis circuit that maps the depth information to body composition information, and a display that displays the body composition information.

[0092] According to another embodiment, the body composition analysis circuitry encodes depth information into a latent space.

[0093] According to another embodiment, the latent space is based on a deep learning model that is trained on user study data.

[0094] According to another embodiment, the body composition analysis circuitry compensates for facial expression and neck pose.

[0095] According to another embodiment, the body composition analysis circuitry discards unused portions of the depth information.

[0096] According to one embodiment, an electronic device is provided that includes a three-dimensional depth sensor that captures face and neck image data, and a control circuit that receives the face and neck image data, receives user demographic information, and determines an estimated body composition based on the face and neck image data and the user demographic information.

[0097] According to another embodiment, the control circuitry uses a portion of the face and neck image data to determine the estimated body composition and discards remaining portions of the face and neck image data.

[0098] According to another embodiment, the electronic device includes a display that displays the estimated body composition.

[0099] According to another embodiment, the control circuitry encodes the face and neck image data into a latent space.

[0100] According to another embodiment, the latent space is based on a deep learning model that is trained on user study data.

[0101] According to one embodiment, an electronic device is provided that includes a sensor that captures an image of the body and a body composition analysis circuit that uses a user study training model to map the image of the body to body composition information, where the user study training model accounts for variations due to breathing and posture.

[0102] According to another embodiment, the sensor captures an image of the face and the body composition analysis circuit uses the image of the face to scale the image of the body to determine body dimensions.

[0103] According to another embodiment, the body composition information indicates how body fat is distributed throughout the body.

[0104] According to another embodiment, the body composition information includes visceral fat percentage and subcutaneous fat percentage.

[0105] According to another embodiment, the user study training model is selected from the group consisting of a statistical model and a deep learning model.

[0106] According to another embodiment, an electronic device includes a head mounted support structure.

[0107] According to another embodiment, a sensor captures images of a body of a first user while a head-mounted support structure is worn on the head of a second user.

[0108] According to another embodiment, the sensor includes an infrared depth sensor and the image of the face and body is an infrared depth map image.

[0109] According to another embodiment, the sensor includes a visible light camera and the images of the face and body are visible light images.

[0110] According to another embodiment, the image of the body is an image of only a portion of the body, and the body composition information describes the body composition of the portion of the body.

[0111] The above is merely illustrative and various modifications may be made to the described embodiments. The above embodiments may be implemented individually or in any combination.

Claims

1. A sensor for capturing an image of the face and neck; encoding a portion of the image into a latent space including a first latent space representing a user identity, a second latent space representing a facial expression, and a third latent space representing a neck pose; determining a percent fat value based on the portion of the image a body composition analysis circuit that uses the first latent space to determine the percent fat value based on the portion of the image, thereby compensating for the facial expression and the neck pose; 1. An electronic device comprising:

2. The electronic device of claim 1 , wherein the latent space is based on a statistical model.

3. The electronic device of claim 1 , wherein the latent space is based on a deep learning model.

4. The sensor includes a depth sensor, the image includes a three-dimensional depth map image, and the depth sensor includes: an infrared light source that illuminates the face and neck with a pattern of light; and an infrared light detector that detects the pattern of light reflected from the face and neck.

5. The electronic device of claim 1 , wherein the portion of the image includes a cheek portion and a neck portion.

6. The electronic device of claim 5 , wherein the body composition analysis circuitry removes remaining portions of the image that are not encoded in the latent space.

7. The electronic device of claim 1 , wherein the body composition analysis circuitry maps the portion of the image to the percentage fat value based at least in part on user demographic data.

8. The electronic device of claim 1 , wherein the sensor comprises a visible light camera and the image comprises a visible light image.

9. an infrared light source that illuminates the face with a light pattern; an infrared light detector that detects the pattern of light reflected from the face and outputs corresponding depth information; a body composition analysis circuit that encodes the depth information into a latent space including a first latent space representing a user identity, a second latent space representing a facial expression, and a third latent space representing a neck pose, and maps the depth information to body composition information, the body composition analysis circuit compensating for the facial expression and the neck pose by using the first latent space to map the depth information to the body composition information; a display for displaying the body composition information; and 1. An electronic device comprising:

10. The electronic device of claim 9 , wherein the latent space is based on a deep learning model trained on user study data.

11. The electronic device of claim 9 , wherein the body composition analysis circuitry removes unused portions of the depth information.

12. a sensor for capturing an image of the body; encoding the image into a latent space including a first latent space representing a user identity, a second latent space representing a respiratory state, and a third latent space representing a body posture; Using a user study training model to map the image of the body to body composition information. A body composition analysis circuit, An electronic device comprising: The body composition analysis circuitry uses the first latent space to map the image of the body to the body composition information, thereby compensating for variations due to breathing and posture.

13. 13. The electronic device of claim 12, wherein the sensor captures an image of a face, and the body composition analysis circuitry uses the image of the face to scale the image of the body to determine dimensions of the body.

14. 13. The electronic device of claim 12, wherein the body composition information indicates how body fat is distributed throughout the body, the sensor includes an infrared depth sensor, the image of the body is an infrared depth map image, the image of the body is an image of only a portion of the body, and the body composition information describes the body composition of the portion of the body.

15. 13. The electronic device of claim 12, wherein the body composition information includes a visceral fat percentage and a subcutaneous fat percentage, and the user study training model is selected from the group consisting of a statistical model and a deep learning model.

16. 13. The electronic device of claim 12, further comprising a head-mounted support structure, the sensor capturing the image of the body of a first user while the head-mounted support structure is worn on a head of a second user.

Images