Electronic device with body composition analysis circuit

Abstract

The present disclosure relates to an electronic device with a body composition analysis circuit. An electronic device may include a body composition analysis circuit that estimates body composition based on captured images of a face, neck, and / or body (e.g., a depth map image captured by a depth sensor, a visible light and infrared image captured by an image sensor, and / or other suitable images). The body composition analysis circuit may analyze image data and may extract portions of the image data that are strongly related to body composition, such as portions of the cheeks, neck, waist, etc. The body composition analysis circuit may encode the image data into a latent space. The latent space may be based on a deep learning model that considers facial expressions and neck posture in face / neck images and considers breathing and body posture in body images. The body composition analysis circuit may output an estimated body composition based on the image data and based on user demographic information.

Description

[0001] This patent application claims priority to U.S. Patent Application No. 17 / 865,194 filed on July 14, 2022, U.S. Provisional Patent Application No. 63 / 238,714 filed on August 30, 2021, and U.S. Provisional Patent Application No. 63 / 242,357 filed on September 9, 2021, which are hereby incorporated by reference in their entirety. Technical Field

[0002] The present disclosure relates generally to electronic devices, and more particularly, to electronic devices having sensors. Background Art

[0003] Electronic devices such as cell phones, watches, and other equipment are sometimes provided with sensors such as fingerprint sensors, facial recognition cameras, and heart rate sensors.

[0004] Using devices such as these can be challenging. Users may wish to obtain different types of health-related information that are not available from traditional electronic devices. Users may need to rely on more than one piece of electronic equipment to obtain the desired health-related information, which can be inconvenient and cumbersome. Summary of the invention

[0005] The electronic device may include body composition analysis circuitry that estimates body composition based on captured images of the face, neck, and / or body (e.g., three-dimensional depth maps, visible light and infrared images, and / or other suitable images captured by a depth sensor). In some arrangements, the depth sensor in the electronic device may include an infrared light emitter that illuminates the face and neck with structured infrared light and an infrared light detector that detects infrared light reflected from the face and neck. The depth sensor may generate depth map image data that captures three-dimensional structure data based on the reflected infrared light. If desired, other types of depth sensing technology and / or visible light cameras may be used to capture facial and neck image data. In some arrangements, these images may be full-body images, or may be images of a portion of the user's body, such as a torso or biceps.

[0006] In some arrangements, the body composition analysis circuitry may map images to body composition information using a model trained through user studies. The model may be trained on images of specific body parts and / or may be trained on images of the entire body. The body composition information may describe how fat is distributed throughout the body and / or may describe the relative amounts of fat in the visceral and subcutaneous compartments of the body. The body composition analysis circuitry may use an image of the face to scale an image of the body in order to determine the dimensions of the body. The electronic device may be a head mounted device or any other suitable electronic device worn or used by a first user when capturing images of a second user. The electronic device may also operate autonomously when capturing images of a user. If desired, the electronic device may capture images of a user while attached to a stationary fixture.

[0007] The body composition analysis circuitry may analyze the image data and may extract portions of the image data that are strongly related to body composition, such as portions of the cheeks, neck, face, chest, waist, buttocks, thighs, and other areas. The body composition analysis circuitry may encode the image data into a latent space. The latent space may be based on a deep learning model trained on user study data.

[0008] When a facial image is used, the latent space may include a first latent space representing the user identity, a second latent space representing the facial expression, and a third latent space representing the neck posture. The body composition analysis circuit may output an estimated body composition by compensating for the facial expression and the neck posture using the facial and neck image data in the user identity latent space.

[0009] When a body image is used, the latent space may include a first latent space representing the user identity, a second latent space representing the breathing state, and a third latent space representing the body posture. The body composition analysis circuit may output an estimated body composition by compensating for the breathing state and the body posture using the body image data in the user identity latent space.

[0010] The body composition analysis circuitry may use a model trained through user studies to map an image to body composition information. The model may be trained on images of specific body parts and / or may be trained on images of the entire body. The body composition information may describe how fat is distributed throughout the body and / or may describe the relative amounts of visceral and subcutaneous fat in specific body parts. The body composition analysis circuitry may use an image of the face to scale an image of the body in order to determine the dimensions of the body. The electronic device may be a head mounted device or any other suitable electronic device worn by a first user when capturing images of a second user. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a schematic diagram of an illustrative electronic device according to one embodiment.

[0012] Figure 2 is a perspective view of an exemplary electronic device having a display with an optical assembly window overlapping an optical assembly according to an embodiment.

[0013] Figure 3 is a cross-sectional side view of an exemplary electronic device having optical components such as a light source and an image sensor according to an embodiment.

[0014] Figure 4 is a cross-sectional side view of an exemplary light source including a diffuser according to an embodiment.

[0015] Figure 5 For use according to the implementation scheme Figure 4 A front view of an exemplary object onto which a light source of the type shown projects a dot pattern.

[0016] Figure 6 is a schematic diagram of an illustrative body composition analysis circuit for analyzing face and neck images according to an embodiment.

[0017] Figure 7 is a schematic diagram of an illustrative body composition analysis circuit for analyzing a body image according to an embodiment.

[0018] Figure 8 is a diagram of exemplary three-dimensional depth map image data associated with a face and neck according to an embodiment.

[0019] Fig. 9 To illustrate the extractable Figure 8 A diagram of the manner in which relevant portions of three-dimensional depth map image data of the type shown are used for body composition analysis.

[0020] Fig.10 is a diagram of exemplary image data corresponding to a frontal body view according to an embodiment.

[0021] Fig.11 is a diagram of exemplary image data corresponding to a side body view according to an embodiment.

[0022] Fig.12 To illustrate the extractable Fig.10 and Fig.11 A diagram of the type of image data shown in the figure in a manner that provides for body composition analysis.

[0023] Fig.13 is a schematic diagram illustrating the manner in which facial image data captured at different times may be aligned for body composition analysis according to an embodiment.

[0024] Fig.14is a schematic diagram illustrating the manner in which body image data captured at different times may be aligned for body composition analysis according to an embodiment.

[0025] Fig.15 is a graph of illustrative user research data that may be collected over a period of time according to an embodiment.

[0026] Fig.16 Flowchart of illustrative steps involved in estimating body composition based on captured image data according to an embodiment. DETAILED DESCRIPTION

[0027] Figure 1 Schematic diagrams of illustrative electronic devices of the type that may be provided with optical components are shown in . Electronic device 10 may be a computing device such as a laptop computer, a computer monitor including an embedded computer, a tablet computer, a cellular phone, a media player, or other handheld or portable electronic device, a speaker (e.g., a voice-controlled assistant or other suitable speaker), a smaller device (such as a watch device), a hanging device, a headset or earpiece device, a device embedded in glasses or other equipment worn on a user's head, or other wearable or miniature devices, a television, a computer display that does not include an embedded computer, a gaming device, a navigation device, an embedded system (such as a system in which an electronic device with a display is installed in a kiosk or a car), an apparatus that implements the functionality of two or more of these devices, or other electronic equipment.

[0028] like Figure 1 As shown, electronic device 10 may have control circuitry 16. Control circuitry 16 may include storage and processing circuitry for supporting the operation of device 10. Storage and processing circuitry may include storage devices such as hard drive storage devices, non-volatile memory (e.g., flash memory or other electrically programmable read-only memory configured to form a solid-state drive), volatile memory (e.g., static or dynamic random access memory), etc. Processing circuitry in control circuitry 16 may be used to control the operation of device 10. The processing circuitry may be based on one or more microprocessors, microcontrollers, digital signal processors, baseband processors, power management units, audio chips, application specific integrated circuits, etc.

[0029] Device 10 may have input-output circuitry such as input-output device 12. Input-output device 12 may include a user input device that collects user input and an output component that provides output to the user. Device 12 may also include communication circuitry that receives data from device 10 and provides data from device 10 to external devices. Device 12 may also include sensors that collect information from the environment.

[0030] Input-output devices 12 may include one or more displays, such as display 14. Display 14 may be a touch screen display including a touch sensor for acquiring touch input from a user, or display 14 may be insensitive to touch. The touch sensor of display 14 may be based on an array of capacitive touch sensor electrodes, an acoustic touch sensor structure, a resistive touch component, a force-based touch sensor structure, a light-based touch sensor, or other suitable touch sensor arrangements. Display 14 may be a liquid crystal display, a light emitting diode display (e.g., an organic light emitting diode display), an electrophoretic display, or other display.

[0031] Input-output device 12 may include optical assembly 18. Optical assembly 18 may include light emitting diodes and other light sources. For example, optical assembly 18 may include one or more visible light sources, such as light source 20 (e.g., light emitting diodes). Light emitting diode 20 may provide constant illumination (e.g., to implement a flash function of device 10) and / or may emit pulses of flash illumination for a visible light camera such as visible light image sensor 26. Optical assembly 18 may also include an infrared light source (e.g., a laser, a lamp, an infrared light emitting diode, an array of vertical cavity surface emitting lasers (VCSELs), etc.), such as infrared light source 22. Infrared light source 22 may provide constant illumination and / or pulsed illumination at infrared wavelengths such as 940 nm, wavelengths within the range of 800 nm-1100 nm, etc. For example, infrared light source 22 may provide constant illumination for an infrared camera such as infrared image sensor (or infrared light detector) 28. For example, infrared image sensor 28 may be configured to capture iris scan information from a user's eye and / or may be used to capture images for a facial recognition process implemented on control circuit 16.

[0032] If desired, infrared light source 22 can be used to provide flood lighting (e.g., diffuse infrared light that evenly covers a given area) and to provide structured light (e.g., a pattern of collimated dots). Flood lighting can be used to capture infrared images of external objects (e.g., to detect a user's face and / or create a depth map), while structured light can be projected onto external objects to perform depth mapping operations (e.g., to obtain a three-dimensional map of the user's face). This is merely illustrative. Other types of depth sensors (e.g., indirect time-of-flight sensors, stereo cameras, etc.) can be used if desired.

[0033] In order to enable the infrared light source 22 to provide both flood lighting and structured light, the infrared light source 22 may include a switchable diffuser and an array of collimated light sources such as lasers or vertical cavity surface emitting lasers. When flood lighting is required, the diffuser can be turned on to diffuse the light from the light source. When structured lighting is required, the diffuser can be turned off to allow collimated light to pass through the diffuser without inhibition. Diffusers such as the diffuser in the infrared light source 22 can be formed by liquid crystal materials, electrophoretic materials, or other switchable light modulators. In some specific implementations, the infrared light source 22 projects light through a diffractive optical element (DOE) to produce a replica of a dot pattern. However, this is merely exemplary. If desired, the infrared light source 22 may include a first light source that provides flood lighting and a second light source that provides structured light.

[0034] The optical assembly 18 may also include an optical proximity detector 24 and an ambient light sensor 30 .

[0035] Optical proximity detector 24 may include an infrared light source, such as an infrared light emitting diode, and a corresponding light detector, such as an infrared photodetector, for detecting when an external object illuminated by the infrared light from the light emitting diode is within proximity of device 10 .

[0036] The ambient light sensor 30 may be a monochrome ambient light sensor that measures the intensity of ambient light, or may be a color ambient light sensor that measures the color and intensity of ambient light by performing light measurements using multiple photodetectors, each photodetector being provided with a corresponding color filter (e.g., a filter that passes red light, blue light, yellow light, green light, or other colors of light), and each photodetector thereby responding to ambient light in a different wavelength band.

[0037] In addition to optical components 18, input-output devices 12 may include buttons, joysticks, scroll wheels, touch pads, keypads, keyboards, microphones, speakers, tone generators, vibrators, cameras, light-emitting diodes and other status indicators, non-optical sensors (e.g., temperature sensors, microphones, capacitive touch sensors, force sensors, gas sensors, pressure sensors, sensors that monitor device orientation and motion such as an inertial measurement unit formed by an accelerometer, compass, and / or gyroscope), data ports, etc. A user may control the operation of device 10 by providing commands through input-output devices 12, and may receive status information and other output from device 10 using the output resources of input-output devices 12.

[0038] Device 10 may have a housing. The housing may form a laptop computer housing, a housing for a watch, a cell phone housing, a tablet computer housing, or other suitable device housing. Figure 2 A perspective view of a portion of an exemplary electronic device is shown in FIG. Figure 2In the example of , device 10 includes a display, such as display 14, mounted in housing 32. Housing 32, which may sometimes be referred to as a case or housing, may be formed from plastic, glass, ceramic, fiber composite materials, metal (e.g., stainless steel, aluminum, etc.), other suitable materials, or combinations of any two or more of these materials. Housing 32 may be formed using a one-piece configuration in which a portion or all of housing 32 is machined or molded into a single structure, or may be formed using multiple structures (e.g., an internal frame structure, one or more structures forming an exterior housing surface, etc.). Housing 32 may have any suitable shape. Figure 2 In the example of FIG. 1 , housing 32 has a rectangular profile (footprint when viewed from above) and has four peripheral edges (e.g., opposing upper and lower edges and opposing left and right edges). The sidewalls may extend along the perimeter of housing 32. If desired, the strap may be coupled to the main portion of housing 32 (e.g., in a configuration where device 10 is a watch or head-mounted device).

[0039] Display 14 may be protected using a display cover layer such as a transparent glass layer, transparent plastic, sapphire, or other transparent layer (e.g., a transparent planar member forming part or all of the front face of device 10 or mounted in other portions of device 10). Openings may be formed in the display cover layer. For example, openings may be formed in the display cover layer to accommodate buttons, speaker ports such as speaker port 34, or other components. Openings may be formed in housing 32 to form communication ports (e.g., audio jack ports, digital data ports, etc.), to form openings for buttons, etc. In some configurations, housing 32 may have a rear housing wall formed by a planar glass member or other transparent layer (e.g., a planar member formed on the back face of device 10 opposite the front face of device 10 including the display cover layer).

[0040] Display 14 may have an array of pixels 38 (e.g., liquid crystal display pixels, organic light-emitting diode pixels, electrophoretic display pixels, etc.) in active area AA. Pixels 38 of active area AA may display images to a user of device 10. Active area AA may be rectangular, may have notches along one or more edges thereof, may be circular, may be oval, may be rectangular with rounded corners, and / or may have other suitable shapes.

[0041] Inactive portions of display 14, such as inactive border area IA, may be formed along one or more edges of active area AA. Inactive border area IA may overlap circuits, signal lines, and other structures that do not emit light to form an image. In order to hide inactive circuits and other components in border area IA from the perspective of a user of device 10, the underside of the outermost layer (e.g., a display cover layer or other display layer) of display 14 may be coated with an opaque masking material, such as a black ink layer (e.g., a polymer containing a black dye and / or a black pigment, an opaque material of other colors, etc.) and / or other layers (e.g., metal, dielectric, semiconductor, etc.). Opaque masking materials such as these may also be formed on the inner surface of a flat rear housing wall formed of glass, ceramic, polymer, crystalline transparent material such as sapphire, or other transparent material.

[0042] exist Figure 2 In the example of device 10, speaker port 34 is formed by an elongated opening (e.g., a strip-shaped opening) that extends along a dimension parallel to the upper peripheral edge of housing 32. A speaker may be mounted within device housing 32 in alignment with the opening of speaker port 34. During operation of device 10, speaker port 34 functions as an earphone speaker port for a user of device 10 (e.g., a user may place speaker port 34 near the user's ear during a phone call).

[0043] Optical components 18 (e.g., visible digital image sensors, infrared digital image sensors, light-based proximity sensors, ambient light sensors, visible light emitting diodes and / or infrared light emitting diodes that provide constant illumination and / or pulsed illumination, etc.) may be mounted below one or more optical component windows such as optical component window 40. Figure 2 In the example of FIG. 1 , four of the windows 40 have circular profiles (e.g., a circular footprint when viewed from above), and one of the windows 40 has an elongated strip-shaped opening (e.g., an elongated strip-shaped footprint when viewed from above). The elongated window 40 is mounted between the sidewall along the upper peripheral edge of the device 10 and the speaker port 34, and extends parallel to the upper peripheral edge of the housing 32. If desired, the window, such as the optical window 40, may have a shape other than circular and rectangular. Figure 2 The examples are exemplary only.

[0044] Optical assembly windows such as window 40 may be formed in inactive area IA of display 14 (e.g., an inactive border region in a display cover layer, such as an inactive display region extending along an upper peripheral edge of housing 32), or may be formed in other portions of device 10, such as portions of a rear housing wall formed from a transparent member coated with an opaque masking material, portions of a metal housing wall, polymer wall structures, etc. Figure 2As shown in the example in , window 40 is formed adjacent to the upper peripheral edge of housing 32, between speaker port 34 in the display cover layer of display 14 and the sidewall along the upper edge of housing 32. In some configurations, an opaque masking layer is formed on the underside of the display cover layer in inactive area IA, and optical window 40 is formed by an opening in the opaque masking layer. To help optical window 40 visually blend with the opaque masking layer, a dark ink layer, a metal layer, a thin film interference filter formed by a dielectric layer stack, and / or other structures may overlap window 40.

[0045] The infrared emitters and infrared detectors in device 10 may be used to form a three-dimensional depth sensor. Figure 3 1 is a side view of an exemplary depth sensor 36 in device 10 that may be used to generate a three-dimensional depth map, such as eye scan information, a face image (e.g., an image of a user's face for performing a facial recognition operation to authenticate a user of device 10, an image of a user's face and neck for generating animated expressions, etc.), a body image (e.g., an image of a user's body for performing motion tracking or body segmentation), and / or other three-dimensional depth mapping information. Depth sensor 36 may include infrared light source 22 and infrared image sensor 28. Device 10 may use infrared light source 22 (e.g., an infrared light emitting diode, an infrared laser, etc.) to generate infrared light 48. Light 48 may illuminate external objects near device 10, such as external object 44 (e.g., a user's face and / or eyes). Infrared image sensor 28 may be used to receive and image reflected infrared light 46 from external object 44 to generate an infrared image of the face and / or eyes (e.g., a three-dimensional depth map). Depth information may also be captured by applying appropriate software algorithms to visible light and / or near infrared video and / or using any other suitable depth sensor in the device.

[0046] Infrared light source 22 may operate in different modes depending on the type of infrared information to be collected by infrared image sensor 28. For example, in flood illumination mode, infrared light source 22 may emit diffuse light that uniformly covers the desired target area. In structured light mode, infrared light source 22 may emit a known light pattern onto the desired target area.

[0047] Figure 4 2 shows the illumination from the infrared light source 22 when the infrared light source 22 is operated in the flood illumination mode. Figure 4 As shown, infrared light source 22 may emit diffuse infrared light 56 ​​that continuously covers a given area of ​​external object 44. Infrared image sensor 28 may capture an infrared image of diffusely illuminated external object 44. In some arrangements, flood illumination from infrared light source 22 may be used to detect a user's face during facial recognition operations.

[0048] Figure 5 4 shows illumination from infrared light source 22 when infrared light source 22 is operated in structured light mode. In structured light mode, infrared light source 22 may project a known pattern of infrared light 56 ​​onto external object 44. Figure 5 In the example of , infrared light 56 ​​forms a pattern of dots on external object 44. The dots may be an ordered grid array (e.g., evenly spaced from each other), or the dots may be projected in a random speckled pattern. However, this is merely exemplary. If desired, infrared light source 22 may emit structured light in other patterns (e.g., horizontal lines, vertical lines, a grid of horizontal and vertical lines, or other suitable predetermined patterns). Figure 5 The structured infrared light 56 ​​may be based on laser interference or may be based on a projection display element that transmits infrared light through a spatial light modulator to produce a desired pattern.

[0049] In some arrangements, the infrared light source 22 may include one light source that provides flood illumination and another light source that provides structured light. In other arrangements, the same light source may be used to provide both flood illumination and structured light. This may be achieved using a switchable diffuser element that selectively diffuses light emitted from the light source.

[0050] Data collected using optical assembly 18 may be used for one or more health-related applications, such as body composition assessment. For example, control circuitry 16 may use optical assembly 18 to capture images of a user's face, neck, and / or body (e.g., visible images, infrared images, three-dimensional depth map images, etc.), which images may then be analyzed to provide user-specific body composition information, such as body mass index, body fat percentage (e.g., fat percentage of the entire body, fat percentage of individual body parts, and / or fat percentage in different fat storage compartments such as subcutaneous and visceral compartments), bone mass, and / or other health-related information.

[0051] The control circuit 16 may store one or more models for mapping the user image data to body composition information. The model may be a statistical model, a machine learning model, a model based on a combination of statistical modeling and machine learning, or a combination of multiple machine learning models. Models trained using machine learning may be implemented using principal component analysis, autoencoders, and / or any other suitable data compression techniques.

[0052] An autoencoder is an artificial neural network that learns to encode data into a latent space by reducing the dimensionality of the data. An autoencoder is trained to encode the input distribution within the latent space to minimize the loss between the output and the input. PCA reduces the dimensionality of the input data by removing redundant information and capturing the most important features of the input data (e.g., features with the highest variance). PCA is usually limited to linear mappings, while encoders do not have any linear constraints.

[0053] Figure 6 Schematic diagram of body composition analysis circuit 58 for determining body composition from face and / or neck images. Body composition analysis circuit 58 may be part of control circuit 16, and / or may be implemented as a stand-alone circuit. Body composition analysis circuit 58 may receive information such as: face and neck image data (e.g., three-dimensional depth map data of the user's face and / or neck from depth sensor 36, visible image of the user's face and / or neck from visible image sensor 26, etc.), and optional additional user data (e.g., user-specific demographic information, such as gender, height, weight, age, race, and / or other user data stored in device 10 and / or otherwise provided to circuit 58). Based on the received face and neck image data and optional user demographic data, body composition analysis circuit 58 may output estimated body composition information, such as body mass index, body fat percentage, face and neck fat percentage, bone mass, and / or other health-related information. If desired, user demographic information may be omitted, and body composition analysis circuit 58 may estimate the user's body composition based solely on the captured face and neck image data.

[0054] If desired, the face and neck image data may be acquired as part of a dedicated body composition analysis (e.g., when the depth sensor 36 is used specifically to obtain face and neck images for body composition analysis), and / or may be acquired when the depth sensor 36 is already being used for some other purpose (e.g., when the depth sensor 36 is already being used for facial recognition and user authentication purposes, when the depth sensor 36 is already being used to create animated expressions or other virtual reality applications involving capturing the facial expressions of a user). The face and neck image data may include one or more images of the face and neck captured at different times of the day and / or over multiple days.

[0055] User demographic information may be received from the user as part of a dedicated body composition analysis questionnaire, and / or may be received from the user as part of some other health-related application.

[0056] Body composition analysis circuitry 58 may store models trained using data from a study of users. For example, data may be collected from a group of participants (e.g., ten participants, fifty participants, one hundred participants, one thousand participants, and / or any other suitable number of participants) over a given period of time (e.g., one month, two months, three months, six months, eight months, ten months, one year, more than one year, less than one year, etc.). At each data collection point during the study, the face and neck shape and size of the study participants may be measured, and the body composition of the users may be measured. The data may be collected using Figure 3, using anthropometric measurements (e.g., body landmarks and measurements), and / or using any other suitable measuring device (e.g., a 3D body scanner) to measure face and neck shape and size. Body composition may be measured using any suitable body composition tracking technology such as magnetic resonance imaging, dual energy x-ray absorptiometry, air displacement plethysmography, underwater weighing, etc. Alternatively, a model may be trained to predict the percentage of fat in the face and neck. Data collected during the user study may be used as training data for training a model stored in body composition analysis circuitry 58 in device 10.

[0057] The body composition analysis circuit 58 may use principal component analysis, autoencoders, and / or any other suitable data compression techniques to reduce the dimensionality of the input data in the latent space. For example, the latent space may include an identity latent space describing the identity of the subject, an expression latent space describing the facial expression of the subject, and a pose latent space describing the neck posture of the subject. By including the facial expression latent space and the neck posture latent space, the body composition analysis circuit 58 may output an estimated body composition of the subject by compensating for the effects of facial expression and neck posture using only the identity latent space. In addition, transfer learning methods may be used to selectively enhance a pre-trained machine learning model using other data.

[0058] Figure 7 Schematic diagram of body composition analysis circuit 58 for determining body composition from facial and / or neck images. Body composition analysis circuit 58 may be part of control circuit 16, and / or may be implemented as a stand-alone circuit. Body composition analysis circuit 58 may receive information such as body image data (e.g., three-dimensional depth map data of the user's body from depth sensor 36, visible image of the user's body from visible image sensor 26, etc.), and optional additional user data (e.g., user-specific demographic information, such as gender, height, weight, age, race, and / or other user data stored in device 10 and / or otherwise provided to circuit 58). Based on the received body image data and optional user demographic data, body composition analysis circuit 58 may output estimated body composition information, such as body mass index, body fat percentage, bone mass, and / or other health-related information. If desired, user demographic information may be omitted, and body composition analysis circuit 58 may estimate the user's body composition based solely on the captured body image data.

[0059] The body composition analysis circuit 58 can use any suitable model to analyze body composition. In the two-compartment model, it is assumed that the body consists of two compartments, which are a first compartment corresponding to fat and a second compartment corresponding to everything except fat (e.g., muscle, bone, etc.). In the three-compartment model, it is assumed that the body consists of visceral fat, subcutaneous fat, and non-fat. If necessary, the body composition analysis circuit 58 can use a three-compartment model and can estimate the amount of visceral fat, subcutaneous fat, and non-fat in the user's body based on the user's image. The body composition analysis circuit 58 can estimate the body composition of a specific area of ​​the body (e.g., how much visceral fat and subcutaneous fat are present in the user's torso), or can estimate the body composition in the entire body (e.g., how the total amount of visceral fat and subcutaneous fat is distributed across the user's body).

[0060] If desired, body image data may be collected as part of a dedicated body composition analysis (e.g., when depth sensor 36 is used specifically to obtain body images for body composition analysis), and / or may be collected when depth sensor 36 is already being used for some other purpose (e.g., when depth sensor 36 is already being used for some other body scanning purpose). The body image data may include one or more images of the body in different views (e.g., front view, side profile view, back view, etc.) captured at different times of the day and / or over multiple days. The image data may include image sequences such as those from a video taken while the subject is breathing and / or moving.

[0061] User demographic information may be received from the user as part of a dedicated body composition analysis questionnaire, and / or may be received from the user as part of some other health-related application.

[0062] Body composition analysis circuitry 58 may store models trained using data from a study of users. For example, data may be collected from a group of participants (e.g., ten participants, fifty participants, one hundred participants, one thousand participants, and / or any other suitable number of participants) over a given period of time (e.g., one month, two months, three months, six months, eight months, ten months, one year, more than one year, less than one year, etc.). At each data collection point during the study, the body shape and size of the study participants may be measured, and the body composition of the users may be measured. Figure 3Body shape and size may be measured using a three-dimensional depth sensor of the type shown in , using anthropometric measurements (e.g., body landmarks and measurements), and / or using any other suitable measuring device (e.g., a three-dimensional body scanner). Body composition may be measured using any suitable body composition tracking technology such as magnetic resonance imaging, dual energy x-ray absorptiometry, air displacement plethysmography, underwater weighing, etc. Alternatively, the model may be trained to predict the percentage of fat in the body. The data collected during the user study may be used as training data for training the model stored in the body composition analysis circuit 58 in the device 10.

[0063] The body composition analysis circuit 58 may use principal component analysis, autoencoders, and / or any other suitable data compression techniques to reduce the dimensionality of the input data in the latent space. For example, the latent space may include an identity latent space describing the identity of the subject, a breathing state latent space describing the breathing state of the subject, and a posture latent space describing the body posture of the subject. By including the breathing state latent space and the body posture latent space, the body composition analysis circuit 58 can output the estimated body composition of the subject by compensating for the effects of breathing and body posture using only the identity latent space. In addition, transfer learning methods can be used to selectively enhance a pre-trained machine learning model using other data.

[0064] The model used by body composition analysis circuitry 58 to map image data to body composition may take into account various factors to help distinguish fat from fluid. Body composition analysis circuitry 58 may use known fat and water storage areas to differentiate between fat and fluid accumulation. For example, bags under the eyes may be an indicator of fluid retention rather than fat storage. Areas around joints, feet, and arms tend to be areas of fluid retention rather than fat storage.

[0065] Figure 8 FIG. 2 is a diagram showing exemplary data that may be used to determine body composition when using facial and / or neck image data. Figure 8 As shown, the captured image data 60 (e.g., captured face and neck image data) may include a three-dimensional depth map of the user's face and neck. The face and neck image data 60 may include an array of data points representing depth to different locations across the user's face and neck. If desired, the depth map may be obtained by Figure 3 The depth sensor 36 captures image data 60 .

[0066] If desired, all of image data 60 may be used during a body composition analysis operation, or only a portion of image data 60 may be used during a body composition analysis operation. Because body fat tends to be stored in certain pockets of fat, such as in the cheeks and neck, those areas may be more indicative of body composition than other areas. For example, as the user's body fat changes, the shape of the user's forehead may show little change, while portions of the cheeks and neck may show detectable changes that directly correlate to changes in body composition. If desired, body composition analysis circuitry 58 may select certain portions of data 60, such as data in area 62, for body composition analysis, and may delete the remaining data from device 10. After selecting the data in area 62 and deleting the remaining data, body composition analysis circuitry 58 may use Fig. 9 The data 60' was subjected to body composition analysis.

[0067] Fig.10 and Fig.11 is a diagram showing exemplary data that may be used to determine body composition when using a body image. Fig.10 As shown, the captured image data 60 (eg, captured body image data 60 ) may include one or more three-dimensional depth maps of the user's body captured from a frontal view. Fig.11 1 shows how the captured body image data 60 may include one or more three-dimensional depth maps of the user's body captured from a side profile view. The body image data 60 may include an array of data points representing depth to different locations across the user's body. Figure 3 The depth sensor 36 of the image data 60 is used to capture the image data 60. However, this is merely exemplary. If desired, the image data 60 may be depth image data captured by a different type of depth sensor (e.g., a depth sensor that does not use structured light), may be visible light image data captured using a visible light camera, may be infrared image data captured by an infrared sensor, or may be other suitable image data.

[0068] In some arrangements, data 60 may be collected using sensors in device 10 that are placed far enough from the user to capture a full-body image. For example, device 10 may be a television with a sensor that captures image data 60 when the user stands far enough from the user to capture a full-body image, or device 10 may be a portable electronic device such as a cellular phone, a laptop computer, a tablet computer, or other electronic device that can be supported in a position to capture a full-body image of the user when the user stands at a certain distance. If desired, device 10 may be a head-mounted device or any other suitable electronic device worn by a first user (e.g., a fitness trainer) when observing a second user (e.g., a client of the trainer) at a certain distance. The head-mounted device may have a sensor that captures the image data 60 of the second user when the second user stands at a certain distance from the first user wearing device 10. The electronic device may operate autonomously when capturing images of the user. If desired, the electronic device may be attached to a stationary fixture when capturing images of the user.

[0069] In some arrangements, data 60 may be collected by a handheld electronic device held in the user's hand (e.g., using a front-facing image sensor in device 10). Image distortion may be corrected using pincushion correction, keystone correction, and / or any other suitable distortion compensation technique. If desired, an image of the user's face that does not exhibit distortion may be used to remove distortion in the full-body image. For example, the size of the user's face may be determined from the facial image that does not have distortion, which in turn may be used to scale the full-body image so that the control circuit 16 may determine the size of the user's body based on the full-body image. Orientation information from a motion sensor (e.g., accelerometer, gyroscope, compass, etc.) in device 10 may also be used to remove distortion in the full-body image to obtain a more accurate picture of the size of the user's body. An arrangement may also be used in which the body composition analysis circuit 58 merges multiple photos of different parts of the body together.

[0070] In some arrangements, the image data 60 may include an image of only a portion of the user's body. For example, the image data 60 may be torso image data including only an image of the user's torso, biceps image data including images of the user's biceps, leg image data including only images of the user's legs, and / or other suitable image data. An image of a portion of the user's body may be used to determine body composition in that particular portion of the user's body (e.g., to track visceral and / or subcutaneous fat in the torso, biceps, etc.).

[0071] If desired, all of the image data 60 may be used during the body composition analysis operation, or only a portion of the image data 60 may be used during the body composition analysis operation. Because body fat tends to be stored in certain pockets of fat in areas such as the face, neck, waist, hips, and thighs, those areas may be more indicative of body composition than other areas. For example, as the user's body fat changes, the shape of the user's forehead may exhibit little change, while portions of the cheeks, neck, and waist may exhibit detectable changes that directly correlate to changes in body composition. If desired, the body composition analysis circuitry 58 may determine which portions of the data 60 correspond to areas of the body that are strongly correlated with body composition, such as the data in area 62, and may delete the remaining data from the device 10. After selecting the data in area 62 and deleting the remaining data, the body composition analysis circuitry 58 may use Fig.12 The data 60' was subjected to body composition analysis.

[0072] If desired, body composition analysis circuitry 58 may track changes in body composition over time by comparing facial, neck, and / or body images captured at different times. Fig.13 is a diagram illustrating the manner in which body composition analysis circuitry 58 may compare facial images captured at different times. Fig.13 As shown, image 64 may represent an image captured by depth sensor 36 at a first time, while image 64' may represent an image captured by depth sensor 36 at a second time. To track changes in body composition between the first time and the second time, body composition analysis circuitry 58 may align at least portions of image 64 and image 64' that are expected to change over time. For example, the user's eyes, nose, ears, and / or other facial features may exhibit little change over time and, therefore, may serve as good anchor points for aligning images captured at different times. Fig.13 As shown, for example, body composition analysis circuitry 58 may align eye 64E of image 64 with eye 64E' of image 64', thereby allowing body composition analysis circuitry 58 to more accurately track changes in shape and size for other areas of the face, such as the user's cheeks and neck. If desired, body composition analysis circuitry 58 may store a model that maps changes in face and neck shape and size to changes in body composition (e.g., body composition analysis circuitry 58 may map differences between image 64 and image 64' to corresponding changes in body fat, if desired).

[0073] Fig.14 is a diagram illustrating the manner in which body composition analysis circuitry 58 may compare body images captured at different times. Fig.14As shown, image 64 may represent an image captured by depth sensor 36 at a first time, while image 64' may represent an image captured by depth sensor 36 at a second time. To track changes in body composition between the first time and the second time, body composition analysis circuitry 58 may align at least portions of image 64 and image 64' that are expected to change over time. For example, a user's eyes, nose, ears, other facial features, and limb and / or bone lengths may exhibit little change over time and, therefore, may serve as good anchor points for aligning images captured at different times. Fig.14 As shown, for example, body composition analysis circuitry 58 may align eye 64E of image 64 with eye 64E' of image 64', thereby allowing body composition analysis circuitry 58 to more accurately track changes in shape and size for other areas of the body, such as the user's face, neck, and waist. If desired, body composition analysis circuitry 58 may store a model that maps changes in body shape and size to changes in body composition (e.g., body composition analysis circuitry 58 may map differences between image 64 and image 64' to corresponding changes in body fat, if desired).

[0074] Fig.15 is a diagram illustrating the manner in which data may be collected during one or more user studies used to train a model stored in body composition analysis circuitry 58 in device 10. Fig.15 As shown, data 66 may be collected from a given user population at time t0, time t1, time t2, etc. up to time tn. Data 66 may include measurements of the participant's face, neck, and / or body shape and size, and measurements of the participant's body fat. Data 66 may be collected daily, weekly, monthly, or at any other suitable cadence throughout the user study. The study may extend over a period of one month, two months, three months, six months, eight months, ten months, one year, more than one year, less than one year, etc.

[0075] At each data collection point during the study (e.g., at time t0, t1, t2, ... tn), each participant's face, neck, and / or body shape and size may be measured, and the participant's body composition may be measured. The training data may include full-body measurements and / or may include segmented body measurements (e.g., bicep measurements, torso measurements, leg measurements, etc.). Training a model stored in device 10 based on segmented body data may allow a user to track changes in a particular body part. For example, a user may use device 10 to take a picture of the user's biceps, and body composition analysis circuitry 58 may map the bicep image to a muscle mass value based on the bicep training data included in data 66.

[0076] Available Figure 3The face, neck, and / or body shape and size may be measured using a three-dimensional depth sensor of the type shown, using anthropometric measurements (e.g., body landmarks and measurements), and / or using any other suitable measurement device (e.g., a three-dimensional body scanner). Body composition may be measured using any suitable body composition tracking technology such as magnetic resonance imaging, dual-energy x-ray absorptiometry, air displacement plethysmography, underwater weighing, or the like. For example, fat data may be measured using a local method such as magnetic resonance imaging or dual-energy x-ray absorptiometry (e.g., to obtain body fat in body areas such as the limbs, torso, lower abdomen, upper abdomen, chest, neck, head, and face). The data 66 collected during the user study may be used as training data for training a model stored in the body composition analysis circuitry 58 in the device 10. If desired, the body composition analysis circuitry 58 may be used in conjunction with the user study. Fig.13 and Fig.14 Techniques of the type described are used to align and compare images of a given participant's face, neck, and / or body captured at different times throughout a study.

[0077] Fig.16 Flowchart of illustrative steps involved in estimating a user's body composition based on captured images during operation of device 10.

[0078] During operation of block 100, body composition analysis circuitry 58 may capture one or more images of the user's face, neck, and / or body using one or more optical components 18 in device 10. For example, depth sensor 36 may capture a three-dimensional depth map image of the user's face, neck, and / or body, visible image sensor 26 may capture a visible image of the user's face, neck, and / or body, and / or other optical components 18 in device 10 may be used to collect image data of the user's face, neck, and / or body.

[0079] For body images, the image data may be captured by scanning the body from head to toe, by capturing the entire body in one image frame, and / or by capturing multiple image frames of different parts of the body, such as the face, neck, waist, legs, etc. The body image data may include a frontal body view and a side body view, such as Fig.10 and Fig.11 The image data may also be captured in an image sequence such as those from a video captured while the subject is breathing and / or moving.

[0080] The face, neck, and / or body image data may be collected as part of a dedicated body composition analysis (e.g., when the depth sensor 36 is used specifically to obtain face, neck, and / or body images for body composition analysis), and / or may be collected when the depth sensor 36 is already being used for some other purpose (e.g., when the depth sensor 36 is already being used for facial recognition and user authentication purposes, when the depth sensor 36 is already being used to create animated expressions or other virtual reality applications involving capturing the user's facial expressions). The face, neck, and / or body image data may include one or more images of the face, neck, and / or body captured from different angles at different times of the day and / or over multiple days.

[0081] During operation at block 102, body composition analysis circuitry 58 may analyze the images captured during block 100 and may identify which regions of the captured images are relevant to body composition analysis. This may include identifying which regions of the image data correspond to regions that are strongly correlated with body composition (e.g., Figure 8 , Fig. 9 , Fig.10 , Fig.11 and Fig.12 For example, image data corresponding to a facial region (e.g., a cheek region), a neck region, and / or a waist region may be saved for body composition analysis. If desired, regions of the image data that are not strongly associated with body composition may be deleted or otherwise not used.

[0082] During operation at block 104, the body composition analysis circuitry 58 may encode the relevant image data identified during block 102 into a latent space. This may include using an autoencoder, principal component analysis, and / or other data compression techniques to reduce the dimensionality of the image data. For example, if the relevant image data for body composition includes thousands of data points, the body composition analysis circuitry 58 may compress the relevant image data to hundreds of data points (as an illustrative example).

[0083] For face and neck images, the latent space may include an identity latent space describing the identity of the subject, an expression latent space describing the facial expression of the subject, and a pose latent space describing the pose of the neck of the subject. The latent space may be based on statistical modeling, deep learning techniques (e.g., autoencoders, principal component analysis, etc.), and / or may be based on a combination of statistical modeling and deep learning.

[0084] For body images, the latent space may include an identity latent space describing the identity of the subject, a breathing state latent space describing the breathing state of the subject, and a body posture latent space describing the body posture of the subject. The latent space may be based on statistical modeling, deep learning techniques (e.g., autoencoders, principal component analysis, etc.), and / or may be based on a combination of statistical modeling and deep learning.

[0085] During the operation of block 106, the body composition analysis circuitry 58 may compensate for the effects of facial expression and neck pose in the face / neck image by extracting only the identity latent space (e.g., removing the expression latent space and the neck pose latent space). For body images, the body composition analysis circuitry 58 may compensate for the effects of respiration and body pose by extracting only the identity latent space (e.g., removing the respiration state latent space and the body pose latent space).

[0086] During operation at block 108, body composition analysis circuitry 58 may estimate body composition based on image data in an identity latent space. For example, using a model trained on one or more user studies (e.g., as combined with Figure 6 and Figure 7 ), the body composition analysis circuit 58 may map the compressed image data (e.g., a compressed data set representing the size and shape of the user's cheeks and neck, a compressed data set representing the size and shape of the user's waist, etc.) to body composition information, such as body mass index, body fat percentage (e.g., fat percentage for the entire body, fat percentage for individual body parts, and / or fat percentage in different fat storage compartments such as subcutaneous and visceral compartments), bone mass, and / or other health-related information. The body composition information provided by the body composition analysis circuit 58 may be an estimated current body composition value (e.g., a body fat percentage value, a body mass index value, or a bone mass value), and / or may be an estimated change in a body composition parameter (e.g., the amount of increase or decrease in a given body composition parameter such as body mass index, body fat percentage, bone mass, etc.). If desired, the body composition analysis circuit 58 may also consider any available user demographic information (e.g., gender, height, weight, age, race, and / or other user data stored in the device 10 and / or otherwise provided to the circuit 58) to determine the user's body composition.

[0087] The body composition analysis circuit 58 may use any suitable model to analyze the body composition. In a two-compartment model, the body is assumed to consist of two compartments, a first compartment corresponding to fat and a second compartment corresponding to everything other than fat (e.g., muscle, bone, etc.). In a three-compartment model, the body is assumed to consist of visceral fat, subcutaneous fat, and non-fat. If desired, the body composition analysis circuit 58 may use a three-compartment model and may estimate the amount of visceral fat, subcutaneous fat, and non-fat in the user's body based on the user's image.

[0088] During operation of box 108, the body composition analysis circuit 58 may estimate the body composition of a particular area of ​​the body (e.g., how much visceral fat and subcutaneous fat is present in the user's torso, biceps, or other body part), or may estimate the body composition of the entire body (e.g., how visceral fat and subcutaneous fat are distributed across the user's body).

[0089] The operation of block 108 may include removing distortion (e.g., perspective distortion created when a user points the front-facing camera downward to capture the entire body in one frame) from the image of the body. The body composition analysis circuitry 58 may also use an image of the user's face (e.g., a previously captured facial image, such as a facial image captured during a user identification operation and / or a facial image captured specifically for body composition analysis) to scale the full-body image (e.g., the facial image may be scaled to determine body size based on the full-body image and the facial image). Orientation information from motion sensors in the device 10 (e.g., accelerometers, gyroscopes, compasses, etc.) may also be used to remove distortion in the full-body image to obtain a more accurate picture of the size of the user's body. An arrangement may also be used in which the body composition analysis circuitry 58 merges together multiple photos of different parts of the body.

[0090] During the operation of block 110, control circuitry 16 may take action in response to the analysis results. For example, device 10 may provide the assessment results to a user of device 10 and / or may issue an alert to the user of device 10 (e.g., if the assessment results indicate, for example, a risk of disease). Generally speaking, based on the results of the sensor processing operations of block 108, notifications may be issued, databases may be updated, recommendations may be provided, and / or other actions may be taken. For example, display 14 may display an estimated body fat percentage value, body mass index value, bone mass value, and / or other information determined by body composition analysis circuitry 58. Notifications may include text notifications, audible alerts, email messages, annotated images, other on-screen notification content on display 14, and / or other notification content.

[0091] As described above, one aspect of the present technology is to collect and use information, such as information from input-output devices. The present disclosure contemplates that, in some cases, data including personal information that uniquely identifies or can be used to contact or locate a specific person may be collected. Such personal information data may include demographic data, location-based data, phone numbers, email addresses, twitter IDs, home addresses, data or records related to a user's health or fitness level (e.g., vital sign measurements, medication information, exercise information), date of birth, username, password, biometric information, or any other identifying information or personal information.

[0092] The present disclosure recognizes that the use of such personal information in the disclosed technology can be used to benefit the user. For example, the personal information data can be used to deliver targeted content that the user is more interested in. Therefore, the use of such personal information data enables the user to have planned control over the delivered content. In addition, the present disclosure also anticipates other uses of personal information data that benefit the user. For example, health and fitness data can be used to provide insights into the user's overall health status, or can be used as positive feedback to individuals who use technology to pursue health goals.

[0093] This disclosure envisions that entities responsible for collecting, analyzing, disclosing, transmitting, storing or otherwise using such personal information data will comply with established privacy policies and / or privacy practices. Specifically, such entities should implement and adhere to privacy policies and practices that are recognized as meeting or exceeding industry or government requirements for maintaining the privacy and security of personal information data. Such policies should be easily accessible to users and should be updated as the collection and / or use of data changes. Personal information from users should be collected for legitimate and reasonable purposes of the entity and not shared or sold outside of these legitimate uses. In addition, such collection / sharing should be performed after receiving informed consent from the user. In addition, such entities should consider taking any necessary steps to safeguard and protect access to such personal information data and ensure that others who have access to personal information data comply with their privacy policies and processes. In addition, such entities may subject themselves to third-party assessments to demonstrate their compliance with widely accepted privacy policies and practices. In addition, policies and practices should be adjusted to specific types of personal information data collected and / or accessed, and to applicable laws and standards including specific considerations of jurisdiction. For example, in the United States, the collection or access of certain health data may be governed by federal and / or state laws such as the Health Insurance Portability and Accountability Act (HIPAA), while health data in other countries may be subject to other regulations and policies and should be handled accordingly. Therefore, different privacy practices should be maintained in each country for different types of personal data.

[0094] Regardless of the foregoing, the present disclosure also contemplates implementation schemes in which users selectively block the use or access of personal information data. That is, the present disclosure contemplates that hardware elements and / or software elements may be provided to prevent or block access to such personal information data. For example, the present technology may be configured to allow users to choose to "opt in" or "opt out" of participating in the collection of personal information data at any time during or after registration for a service. For another example, a user may choose not to provide a particular type of user data. For another example, a user may choose to limit the length of time that user-specific data is retained. In addition to providing "opt-in" and "opt-out" options, the present disclosure contemplates providing notifications related to access or use of personal information. For example, a user may be informed that their personal information data will be accessed when downloading an application ("app"), and then be reminded again just before the personal information data is accessed by the app.

[0095] Furthermore, it is an object of the present disclosure that personal information data should be managed and processed to minimize the risk of unintentional or unauthorized access or use. Risks can be minimized by limiting data collection and deleting data once it is no longer needed. Additionally, and when applicable, including in certain health-related applications, data de-identification can be used to protect the privacy of users. Where appropriate, de-identification can be facilitated by removing specific identifiers (e.g., date of birth, etc.), controlling the amount or characteristics of stored data (e.g., collecting location data at a city level rather than an address level), controlling how data is stored (e.g., aggregating data across users), and / or other methods.

[0096] Thus, while the present disclosure broadly covers the use of information that may include personal information data to implement one or more of the various disclosed embodiments, the present disclosure also contemplates that various embodiments may be implemented without access to personal information data. That is, various embodiments of the present technology will not fail to function properly due to the lack of all or a portion of such personal information data.

[0097] According to one embodiment, an electronic device is provided, comprising: a sensor that captures an image of a face and neck; and a body composition analysis circuit that encodes a portion of the image into a latent space and maps the portion of the image to a fat percentage value.

[0098] According to another embodiment, the latent space is based on a statistical model.

[0099] According to another embodiment, the latent space is based on a deep learning model.

[0100] According to another embodiment, the sensor includes a depth sensor, and the image includes a three-dimensional depth map image, the depth sensor including an infrared light source and an infrared light detector, the infrared light source illuminating the face and neck with structured light, and the infrared light detector detecting the structured light reflected from the face and neck.

[0101] According to another embodiment, the latent space includes a first latent space representing the identity of the user, a second latent space representing the facial expression, and a third latent space representing the neck posture.

[0102] According to another embodiment, the body composition analysis circuit maps the portion of the image to the fat percentage value by compensating for facial expression and neck pose using the first latent space.

[0103] According to another embodiment, the portion of the image includes a cheek portion and a neck portion.

[0104] According to another embodiment, the body composition analysis circuit removes remaining portions of the image that were not encoded into the latent space.

[0105] According to another embodiment, the body composition analysis circuit maps the portion of the image to the fat percentage value based at least in part on user demographics.

[0106] According to another embodiment, the sensor comprises a visible light camera and the image comprises a visible light image.

[0107] According to one embodiment, an electronic device is provided, which includes: an infrared light source that illuminates a face with structured light; an infrared light detector that detects the structured light reflected from the face and outputs corresponding depth information; a body composition analysis circuit that maps the depth information to body composition information; and a display that displays the body composition information.

[0108] According to another embodiment, the body composition analysis circuit encodes depth information into a latent space.

[0109] According to another embodiment, the latent space is based on a deep learning model trained with user research data.

[0110] According to another embodiment, the body composition analysis circuit compensates for facial expression and neck posture.

[0111] According to another embodiment, the body composition analysis circuit deletes unused portions of the depth information.

[0112] According to one embodiment, an electronic device is provided, comprising: a three-dimensional depth sensor that captures facial and neck image data; and a control circuit that receives the facial and neck image data, receives user demographic information, and determines an estimated body composition based on the facial and neck image data and the user demographic information.

[0113] According to another embodiment, the control circuit uses a portion of the face and neck image data to determine the estimated body composition and deletes a remaining portion of the face and neck image data.

[0114] According to another embodiment, the electronic device includes: a display that displays the estimated body composition.

[0115] According to another embodiment, the control circuit encodes the face and neck image data into the latent space.

[0116] According to another embodiment, the latent space is based on a deep learning model trained with user research data.

[0117] According to one embodiment, an electronic device is provided, comprising: a sensor that captures an image of a body; and a body composition analysis circuit that uses a model trained through user studies to map the image of the body to body composition information, the model trained through user studies taking into account changes due to breathing and posture.

[0118] According to another embodiment, the sensor captures an image of the face, and the body composition analysis circuit uses the image of the face to scale the image of the body to determine the dimensions of the body.

[0119] According to another embodiment, the body composition information indicates how body fat is distributed throughout the body.

[0120] According to another embodiment, the body composition information includes visceral fat percentage and subcutaneous fat percentage.

[0121] According to another embodiment, the model trained by user studies is selected from the group consisting of a statistical model and a deep learning model.

[0122] According to another embodiment, the electronic device includes a head-mounted support structure.

[0123] According to another embodiment, the sensor captures images of the first user's body while the head mounted support structure is worn on the second user's head.

[0124] According to another embodiment, the sensor includes an infrared depth sensor and the images of the face and body are infrared depth map images.

[0125] According to another embodiment, the sensor includes a visible light camera and the images of the face and body are visible light images.

[0126] According to another embodiment, the image of the body is an image of only a portion of the body, and the body composition information describes the body composition of the portion of the body.

[0127] The foregoing is merely exemplary and various modifications may be made to the embodiments described. The foregoing embodiments may be implemented independently or in any combination.

Claims

1. An electronic device, comprising: a sensor that captures images of the face and neck; as well as A body composition analysis circuit, wherein the body composition analysis circuit: encoding a portion of the image into a latent space, wherein the latent space comprises a first latent space representing a user identity, a second latent space representing a facial expression, and a third latent space representing a neck posture; and A fat percentage value is determined based on the portion of the image, wherein the body composition analysis circuitry compensates for facial expression and neck pose by removing the second latent space and the third latent space and determining the fat percentage value based on the portion of the image using the first latent space. The electronic device of claim 1 , wherein the latent space is based on a statistical model.

3. The electronic device of claim 1, wherein the latent space is based on a deep learning model.

4. The electronic device of claim 1 , wherein the sensor comprises a depth sensor and the image comprises a three-dimensional depth map image, the depth sensor comprising: an infrared light source, the infrared light source illuminating the face and the neck with structured light; as well as An infrared light detector detects the structured light reflected from the face and the neck. The electronic device according to claim 1 , wherein the portion of the image includes a cheek portion and a neck portion.

6. The electronic device of claim 5, wherein the body composition analysis circuit deletes remaining portions of the image that are not encoded into the latent space.

7. The electronic device of claim 1, wherein the body composition analysis circuitry maps the portion of the image to the fat percentage value based at least in part on user demographic data.

8. The electronic device of claim 1, wherein the sensor comprises a visible light camera and the image comprises a visible light image.

9. An electronic device, comprising: an infrared light source that illuminates the face with structured light; an infrared light detector that detects the structured light reflected from the face and outputs corresponding depth information; a body composition analysis circuit that encodes the depth information into a latent space and maps the depth information to body composition information, wherein the latent space includes a first latent space representing a user identity, a second latent space representing a facial expression, and a third latent space representing a neck pose, and wherein the body composition analysis circuit compensates for the facial expression and the neck pose by removing the second latent space and the third latent space and mapping the depth information to the body composition information using the first latent space; as well as A display displays the body composition information.

10. The electronic device of claim 9, wherein the latent space is based on a deep learning model trained with user research data.

11. The electronic device of claim 9, wherein the body composition analysis circuit compensates for facial expression and neck posture.

12. The electronic device of claim 9, wherein the body composition analysis circuit deletes unused portions of the depth information.

13. An electronic device, comprising: a sensor that captures an image of the body; as well as A body composition analysis circuit, wherein the body composition analysis circuit: encoding the facial and neck image data into a latent space, wherein the latent space comprises a first latent space representing the identity of the user, a second latent space representing the facial expression, and a third latent space representing the neck posture; A model trained through user studies is used to map the image of the body to body composition information, wherein the model trained through user studies accounts for changes due to breathing and posture by removing the second latent space and the third latent space and using the first latent space to map the image of the body to body composition information.

14. The electronic device of claim 13, wherein the sensor captures an image of a face, and wherein the body composition analysis circuitry uses the image of the face to scale the image of the body to determine dimensions of the body.

15. An electronic device according to claim 13, wherein the body composition information indicates how body fat is distributed throughout the body, wherein the sensor includes an infrared depth sensor and the image of the body is an infrared depth map image, wherein the image of the body is an image of only a portion of the body, and wherein the body composition information describes the body composition of the portion of the body.

16. The electronic device of claim 13, wherein the body composition information includes visceral fat percentage and subcutaneous fat percentage, and wherein the model trained by user studies is selected from the group consisting of a statistical model and a deep learning model.

17. The electronic device of claim 13, further comprising a head-mounted support structure, wherein the sensor captures the image of the body of the first user when the head-mounted support structure is worn on the head of the second user.

Images