Evaluation for disease risk from data captured by user

Abstract

To provide a device that inexpensively, safely and accurately evaluates a chronic disease risk of a human being.SOLUTION: A disease risk evaluation system 100 includes a user interaction and display system, an image capture system, a first biomarker extraction system, a diabetes disease risk evaluation system, and a 3D body shape model. The image capture system captures one or more images of a human being. The first biomarker extraction system uses the 3D body shape model to estimate a body feature of the human being on the basis of the captured image. The diabetes disease risk evaluation system uses the body feature to generate a disease risk evaluation value related to diabetes and cardiovascular disease risks. The body feature includes at least one or more of a 3D body shape or a body shape indicator, a blood flow, a blood pressure, a heart rate, a respiration rate, a heart rate variation, a heart workload, and one or more of an irregular heartbeat or a stress index. The 3D body shape model is trained over a set of training images.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The described embodiments relate generally to machine learning systems, and more particularly to improving the accuracy of machine learning systems by combining multiple prediction techniques. [Background technology]

[0002] Despite advances in healthcare and modern medicine, accurate detection of chronic diseases such as cardiovascular disease, obesity, and type 2 diabetes remains a challenge. Early detection can save millions of lives and improve quality of life. However, detection of these chronic diseases remains complicated by the traditional need for expensive and inaccessible medical equipment, such as scanners and blood pressure monitors, and by the traditional inaccuracies of techniques that rely primarily on basic anthropometric measurements, such as body mass index (BMI) calculations, which are largely based on a person's weight, height, age, or other information.

[0003] Various statistical methods using physical measurements are performed with limited degrees of freedom in regression models. Some conventional machine learning systems with more degrees of freedom have also been explored. However, conventional machine learning systems may be limited by inaccurate inherent designs, statistical outliers, and / or relatively small datasets and experience. Therefore, it may be desirable to inexpensively, safely, and accurately assess human chronic disease risk using additional human data and data sense. Summary of the Invention

[0004] One aspect of the present disclosure relates to an apparatus including an input module configured to retrieve first and second input data, a processor, and a computer-readable medium including programming instructions that, when executed, cause the processor to: generate a 3D body shape model including biometrics from the first and second input data, generate a health risk indicator and assessment including a first disease risk assessment using the biometrics, generate a second risk assessment based on the second input data, and fuse the biometrics with the first and second input data and optionally the first and second disease risk assessments to generate a multi-category disease risk assessment.

[0005] In one example, the input data module is integrated with an image capture device configured to capture one or more images of the subject. In one example, the second input data includes one or more images of the subject. In one example, the diabetes risk assessment is based on the one or more images.

[0006] In one example, the first disease comprises diabetes, hi one example, the second disease comprises cardiovascular disease.

[0007] One aspect of the present disclosure relates to an apparatus that includes an image capture device configured to capture one or more images of a subject, a processor, and a computer-readable medium, wherein the computer-readable medium includes programming instructions that, when executed, cause the processor to: generate a diabetes risk assessment based on the one or more images using a 3D body shape model; generate a cardiovascular risk assessment based on the one or more images; and fuse the diabetes risk assessment and the cardiovascular risk assessment to generate a disease risk assessment.

[0008] According to some examples, the programming instructions are further configured to update the 3D body shape model based on an output of the fusion of the diabetes risk assessment value and the cardiovascular risk assessment value.

[0009] In some examples, the programming instructions are further configured to generate, from the one or more images using the 3D body shape model, first biomarkers representing body characteristics and generate a diabetes risk assessment based on the body characteristics. In some examples, the first biomarkers include one or more of a 3D body shape or a body shape indicator. According to one example, the diabetes risk assessment is indicative of risk associated with one or more of type 2 diabetes, obesity, central obesity, or metabolic syndrome.

[0010] According to one example, the programming instructions are further configured to generate second biomarkers representative of the cardiac disease-related biomarkers from the one or more images and generate a cardiovascular assessment based on the second biomarkers. The second biomarkers may include one or more of blood flow, blood pressure, heart rate, respiratory rate, heart rate variability, cardiac workload, irregular heartbeat, or stress index. According to one example, the cardiovascular risk assessment may indicate a risk associated with one or more of cardiovascular disease, heart attack, or stroke.

[0011] In one example, the programming instructions are further configured to generate a second biomarker from the facial image of the one or more images, the second biomarker including at least blood flow.

[0012] In another aspect of the present disclosure, an apparatus may include input data configured to retrieve input data, a processor, and a computer-readable medium. The computer-readable medium may include programming instructions that, when executed, cause the processor to generate body features from the input data using a disease risk model and generate a disease risk assessment based on the body features. According to one example, the body features include at least one of a 3D body shape or body shape indicator, and one or more of blood flow, blood pressure, heart rate, respiratory rate, heart rate variability, cardiac work, irregular heartbeat, or stress index.

[0013] In one example, the disease risk assessment indicates the risk associated with at least one or more of type 2 diabetes, obesity, central obesity, or metabolic syndrome, and one or more of cardiovascular disease, heart attack, or stroke.

[0014] In one example, the programming instructions can be further configured to use a machine learning network to generate a disease risk model from the one or more training images.

[0015] In one example, the input data module can include an image capture device configured to capture one or more images of the object. In one example, the input data can include one or more images of the object. [Brief explanation of the drawings]

[0016] The present disclosure will be readily understood by the following detailed description taken in conjunction with the accompanying drawings, in which like reference numerals indicate like structural elements, and in which:

[0017] [Figure 1] FIG. 1 is a block diagram of a system for assessing disease risk in a human, according to some examples described in this disclosure. [Figure 2] 1 is an exemplary process for assessing human disease risk using a machine learning network, according to some examples described in this disclosure. [Figure 3] FIG. 1 is a block diagram of a system for assessing disease risk in a human, according to some examples described in this disclosure. [Figure 4] 1 is an exemplary process for assessing human disease risk using a machine learning network, according to some examples described in this disclosure. [Figure 5] FIG. 1 is a block diagram of a computing device that may be used to implement with various systems or that may be integrated into one or more components of the systems, according to some examples described in this disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0018] Specific details are described below to provide a thorough understanding of various example embodiments of the present disclosure. However, it should be understood that the examples described herein may be practiced without these specific details. Furthermore, the specific examples of the present disclosure described herein should not be construed as limiting the scope of the disclosure to these specific examples. In other examples, well-known circuits, control signals, timing protocols, and software operations have not been shown in detail so as not to unnecessarily obscure the embodiments of the present disclosure. Furthermore, terms such as "couples" and "coupled" mean that two components can be electrically coupled directly or indirectly. Indirectly coupled may mean that two components are coupled via one or more intermediate components.

[0019] The systems and methods described herein can be configured to receive input data including images and infer 3D body models generated using artificial intelligence (AI) and machine learning models. These models can then be used to generate health risk and assessment indicators, such as cardiovascular risk assessments or diabetes risk indicators, based on one or more of the input data and images. Additionally, the systems and methods described herein can be configured to fuse or combine various risk assessments to generate more accurate and comprehensive multivariate-driven disease and health risk assessments.

[0020] 1 is a block diagram of a system for assessing disease risk in a human, according to some examples described in this disclosure. In some examples, a disease risk assessment system 100 can include an input data module with an image capture system 104, e.g., an input data module including an image capture device configured to capture one or more user images and additional human data. The image capture system can include an image capture device such as a camera (e.g., a mobile phone or another handheld computing device with a built-in or fixed camera). The data input module including the image capture device can be configured to capture one or more photographic images of the user in multiple views. The human data can be age, weight, ethnic group, or other historical health data.

[0021] In some examples, system 100 may include a user interaction and display system 102 coupled to an input data module, including coupled to image capture system 104. User interaction and display system 102 may include a computer display configured to provide visual and audio assistance to guide the user to capture an optimal image depending on whether the user is capturing the image themselves or whether another person is capturing the image. For example, during user image capture, user interaction and display system 102 may display a visual representation of the human body, such as a skeleton, a general body silhouette, a target including several alignment features, or other indicators configured to guide the user to move a body part to a desired position so that the captured body image is aligned with the visual representation.

[0022] In some examples, the visual representation may include a body outline, bounding box or bar, or other symbol to indicate the proposed position of one or more body parts or the entire body of the user. As used herein, the term “skeleton” should be broadly interpreted to include a body outline, a bounding box (or other geometric shape), a linear skeletal representation including joints, individual targets representing body and / or joint locations, etc. Furthermore, any number or combination of visual representations or skeletons may be used. For example, system 102 may display a visual representation of the skeletal form of an arm to guide the user to move the arm in a desired pose or extend the arm. Similarly, system 102 may display a skeletal or body structure or other visual representation of the entire body, which may include the head, arms, legs, chest, and / or other parts of the body. The skeleton or outline or joints may be generated based on first data input by the input data module, including a captured user image from an image capture device of the image capture system, such that the skeleton or outline or generally guidance aid is displayed on the display of user interaction and display system 102 in proportion to the image being captured. Guidance aids are displayed to guide the user to capture second and subsequent user images following the first captured user image. In some examples, the user interaction and display system 102 can be standalone and external to the image capture system 104. In other scenarios, the user interaction and display system 102 can be integrated with the image capture system 104, for example, in a mobile phone.

[0023] In some examples, the system 100 may include a first biomarker extraction system, which may be configured to extract first biomarkers based on captured user images from the image capture system 104. Examples of the first biomarkers may include 3D body shape. The first biomarkers, in some embodiments, may also include body shape indicators. Examples of body shape indicators may include body volume, body fat, bone mineral density, or other indicators. The first biomarkers may be extracted based on a 3D body shape model 110. The 3D body shape model may include a 3D representation of the human body. The 3D body shape model may also include body shape indicators such as body volume, body fat, bone mineral density, etc. In a non-limiting example, the 3D body shape indicators may include total body fat, waist-to-length ratio, waist-to-hip ratio, waist circumference, chest circumference, hip circumference, and thigh circumference of the subject.

[0024] The 3D body shape model can be trained from the user image in that the 3D body shape model represents a relationship between the user image and the 3D body shape and body shape indicators. For example, the 3D body shape model can be trained by a 3D representation system. The 3D representation system can also be configured to receive captured user images from the image capture system 104 and use the 3D body shape model to estimate the 3D body shape and body shape indicators of the human body based on the user image. In some examples, the 3D body shape model can include a collection of 3D body shape models, each representing an individual or a group of people.

[0025] 1 , in some examples, the 3D body shape model can also be trained based on body scan parameters in the body scan database 124. In some examples, the body scan parameters can be collected from DEXA scans of various parts of the human body. For example, the body scan parameters can include body fat and / or bone mineral density (measured in Z-scores and T-scores) for different parts of the body, such as the torso, thighs, or hips. In some examples, the 3D body shape model can be trained in a machine learning network. Details of 3D body shape and composition model training will be described further in this disclosure with reference to FIGS. 2 and 4.

[0026] In some examples, the system 100 may include a diabetes disease risk assessment system 114 configured to receive first biomarkers, such as estimated 3D body shape and body shape indicators, from the first biomarker extraction system 106. The diabetes disease risk assessment system 114 may use the estimated first biomarkers to generate a diabetes disease risk value. The diabetes disease risk value may include multiple values ​​representing diabetes risk, such as type 2 diabetes risk, obesity risk, central obesity risk, and metabolic syndrome risk. In some examples, the diabetes disease risk value may be obtained based on the first biomarkers, such as the 3D body shape and body shape indicators. Furthermore, the diabetes disease risk value may be obtained based on changes in one or more of the first biomarkers over time. For example, the system may determine changes in body fat and / or other body shape indicators over a period of time and determine an obesity risk value based on the changes in these indicators over time.

[0027] 1 , the system 100 may further include a second biomarker extraction system 108, which may be configured to extract a second biomarker based on the captured image from the image capture system 104. Examples of the second biomarker may include cardiac disease-related biomarkers such as blood pressure, heart rate, respiratory rate, etc. Examples of the second biomarker may also include heart rate variability, cardiac workload, irregular heartbeat, and stress index. In some examples, the second biomarker may be derived from a physiological characteristic 112.

[0028] With further reference to FIG. 1 , physiological characteristics can be obtained from health sensors or health data. For example, a patient may undergo a daily blood pressure test to obtain health data such as blood pressure, heart rate, and respiratory rate. In some examples, one or more of the biomarkers can be detected from a user image captured from the image capture system 104. Human skin is translucent, but light and its respective wavelengths can be reflected at different lengths beneath the skin, revealing blood flow information. This allows blood flow to be detected from one or more camera images. Similar to facial scanning, the image capture system 104 can be configured to capture thumb images / videos or facial images / videos or extracted data from a smart sensor, such as the user's smartwatch, and detect blood flow from the captured facial images. According to one embodiment, the captured facial images can be video and / or still images.

[0029] In some examples, the user interaction and display system 102 can be configured to guide the user during blood flow capture / detection. For example, the user interaction and display system 102 can provide a user interface on the display to guide the user to remain still while the image capture system determines blood flow from a sequence of facial images. In some examples, blood flow and / or other second biomarkers can be detected from images of other parts of the human body, such as the entire face or specific parts of the face, such as the nose, eyes, ears, cheeks, forehead, lips, etc. Additionally, other parts of the body, such as the hands, wrists, feet, etc., can be used to provide indicators.

[0030] In some examples, the system 100 can include a cardiovascular disease risk assessment system 116, which is configured to receive second biomarkers, such as blood pressure, heart rate, respiratory rate, heart rate variability, cardiac workload, irregular heartbeat, and stress index, from the second biomarker extraction system 108. Based on these biomarkers, the system can determine a risk assessment related to cardiac disease, such as cardiovascular disease risk, heart attack risk, and stroke risk. In some examples, the risk assessment can be expressed as one or more assessment values, such as a cardiovascular risk value, a heart attack risk value, etc.

[0031] With further reference to FIG. 1 , system 100 can include a biomarker fusion system 118 configured to fuse the diabetes risk assessment and cardiovascular risk assessment obtained from diabetes disease risk assessment system 114 and cardiovascular disease risk assessment system 116, respectively. For example, biomarker fusion system 118 can compare the diabetes risk assessment value with the heart attack risk value to determine whether there is a discrepancy between the two assessment values. Based on the comparison, fusion system 118 can adjust the diabetes risk assessment value or the heart attack risk value. This allows for a more accurate disease risk assessment than the diabetes disease risk assessment system or the cardiovascular disease risk assessment system alone. For example, convergence of the two assessment values ​​indicates an accurate assessment, and the confidence level of the risk assessment value can be increased due to validation and similarity between the two assessment values. Similarly, if the two assessment values ​​diverge or contradict each other, additional measurements can be taken or the outlier can be ignored. In some examples, the two assessment values ​​are determined independently, and their combination can be evaluated for an overall or composite risk assessment, with varying values ​​indicating different potential outcomes.

[0032] In some examples, the results of the fusion system 118 can be used to fine-tune the 3D body shape model 110. For example, if the fusion system 118 determines that the diabetes disease risk assessment value needs to be adjusted, one or more biological indicators of the 3D body shape model that contributed to the diabetes risk can also be adjusted. In some examples, if the 3D body shape model is learned from a machine learning network, the machine learning network can also be adjusted based on the results of the biomarker fusion system 118. The fine-tuned 3D body shape model is then used in future operations of the first biomarker extraction system 106. Alternatively and / or additionally, the first biomarker extraction system 106 can be configured to be re-run after the 3D body shape model 110 has been adjusted.

[0033] 2 is an exemplary process for assessing a human's disease risk using a machine learning network, according to some examples described in this disclosure. In some examples, exemplary process 200 can be implemented in disease risk assessment system 100 of FIG. 1. Referring to FIG. 2, process 200 can include prediction process 210. Prediction process 210 can include capturing a user image in operation 202. For example, operation 202 can be performed by image capture system 104 (FIG. 1) to obtain one or more user images, such as a front image, a side image, a rear image, and / or user images from different angles. The user image can be a face image, an upper body image, and / or a full body image.

[0034] In some examples, process 210 may further include extracting body shape features in operation 204. In some examples, operation 204 may be performed in the first biomarker extraction system 106 (FIG. 1). Examples of body shape features may include a 2D silhouette representing a foreground of a human body, 2D joints, or other body shape features. In some examples, the body shape features may be obtained based on a captured user image. Additionally and / or alternatively, process 210 may further include receiving user data in operation 206. For example, operation 206 may receive user input data, such as the user's weight, height, age, gender, ethnicity, etc. This operation may be implemented in the user interaction and display system 102 (FIG. 1). Operation 206 may further evaluate one or more databases to retrieve other user data, such as user health fitness data.

[0035] 2, process 210 may further include predicting a 3D body shape using a machine learning network at operation 212 based on the extracted body shape features (204) and / or the received user data (206). Operation 212 may, in some examples, be performed by first biomarker extraction system 106 (FIG. 1). As described with respect to the embodiment of FIG. 1, the predicted 3D body shape may include a 3D body shape and a first biomarker, such as a body shape indicator (e.g., body volume, body fat, bone mineral density, or other indicator). Similar to FIG. 1, the first biomarker may be obtained using a machine learning model 208. The machine learning model may include a 3D body shape model, such as 110 (FIG. 1).

[0036] 2, process 210 may further include assessing diabetes risk at operation 214. In some embodiments, operation 214 may be performed in diabetes disease risk assessment system 114 (FIG. 1) to generate a diabetes disease risk assessment. Process 210 may further include fusing the diabetes risk assessment and the cardiovascular risk assessment at operation 216. In some examples, the diabetes risk assessment may be obtained from diabetes disease risk assessment system 114 (FIG. 1). The cardiovascular risk assessment may be obtained from cardiovascular disease risk assessment system 116 (FIG. 1). Operation 216 may be performed in biomarker fusion system 118 (FIG. 1) to generate the disease risk assessment. In some examples, process 210 may further include fine-tuning a machine learning network at operation 218 based on the output of fusion operation 216.

[0037] 2 , process 200 may further include a training process 220 for training machine learning model 208. In some examples, process 220 may include acquiring user images in act 202′, extracting body shape features in act 204′, and acquiring user data in act 206′. Process 220 may use the images / features / data from acts 202′, 204′, and / or 206′ to train the machine learning model in act 222. Processes 202′, 204′, and 206′ may be performed in the same manner as processes 202, 204, and 206, respectively, except that the user images acquired from act 202′ differ from the user images acquired from act 202, and the user data acquired from 206′ differ from the user data acquired from 206.

[0038] In a non-limiting example, operation 202' may retrieve user images from a training dataset. For example, the training dataset may include a collection of previously captured or collected training user images and / or training user data, along with ground truth data associated with the training dataset. The ground truth data may include ground truth 3D body shapes and / or other body features.

[0039] In some examples, the training data can include multiple sets, each collected from a subject in a group of subjects, with each set including corresponding ground truth data. In some examples, operation 222 can train a machine learning network to generate a machine learning model 208 based on the collected training data. In some examples, training process 222 can generate a single machine learning model 208 based on the training data collected from the group of subjects. The collected data can be used to modify the weights and parameters of the machine learning model.

[0040] In some other examples, the training process 222 can generate multiple machine learning models 208, each based on training data from a subgroup of subjects or a single subject. For example, the training process can generate machine learning models for subgroups of graining subjects divided by ethnic group, by gender, by age, by height, or by other demographic measures such as occupation, education, etc. Thus, the machine learning models 208 can include one or more 3D body shape models (e.g., 110 in FIG. 1 ).

[0041] Returning to process 210, user images and user data (e.g., weight, height, age, etc.) may be acquired in real time from the user via image capture system 104 and / or user interaction and display system 102. User data may also be acquired from one or more sensors or databases (e.g., user fitness data), as described above. An operation to assess disease risk may be performed using machine learning model 208 learned from process 220. Output from the fusion operation may be used in operation 218 to fine-tune machine learning model 208.

[0042] FIG. 3 is a block diagram of a system for assessing a human's disease risk, according to some examples described in this disclosure. The disease risk assessment system 300 may include one or more components similar to those of the system 100 (FIG. 1). For example, the user interaction and display system 302 and the image capture system 304 may be similar to the user interaction and display system 102 and the image capture system 104, and therefore, their descriptions will not be repeated. In some examples, the system 100 may include a feature extraction system 306 configured to extract human body features from captured images from the image capture system 304. Examples of the human body features may include the 3D body shape and body shape indicators (e.g., first biomarkers) described above in the embodiment of FIG. 1. The human body features may also include heart disease-related biomarkers, such as those corresponding to the second biomarkers described in the embodiment of FIG. 1. In other words, the feature extraction system 306 may use a single disease risk model 310 to jointly extract both the human body features and the heart disease-related biomarkers.

[0043] Similar to the embodiment of FIG. 1 , the disease risk model 310 can be trained from user images by a machine learning network. For example, the disease risk model can be trained by a disease risk model training system. In some examples, the disease risk model can include weights and / or parameters that represent relationships between the user images (and / or user data) and various 3D body shapes and heart disease-related features. These weights / parameters are learned from a collection of training datasets. Training the disease risk model is described in further detail with reference to FIG. 4 .

[0044] Additionally and / or alternatively, the feature extraction system 306 can also extract physiological features directly from user-captured images. For example, some features, such as blood flow, can be extracted from one or more camera images based on image processing at the micropixel level. Similar to the embodiment of FIG. 1, the image capture system 304 can be configured to capture an image of the user's face and detect blood flow from the captured thumb or face image, or an appropriate body part, or a device such as a smartwatch sensor. In some examples, the user interaction and display system 302 can be configured to guide the user during blood flow capture / detection, similar to that described in the embodiment of 102 (FIG. 1). In some examples, blood flow and / or other physiological features can be detected from face images, partial face images, or images of other parts of the human body. The extracted physiological features can be provided to the disease risk model 310 to generate heart disease-related features, similar to that described in the embodiment of FIG. 1.

[0045] In some examples, the system 300 may include a disease risk assessment system 314 configured to receive human body features, such as a 3D body shape, a body shape indicator, and a heart disease-related biomarker, from the feature extraction system 306. The disease risk assessment system 314 may use the extracted human body features to generate one or more disease risk values. For example, the disease risk values ​​may include multiple values ​​representing risks of diabetes, such as type 2 diabetes risk, obesity risk, central obesity risk, and metabolic syndrome risk, and risks of heart disorders, such as cardiovascular disease risk, heart attack risk, and stroke risk. The disease risk assessment may be performed, for example, similarly to the diabetes disease risk assessment system 114 and the cardiovascular disease risk assessment system 116 of FIG. 1 . Details of assessing the risks associated with these diseases will not be repeated.

[0046] 4 is an exemplary process for assessing a human's disease risk using a machine learning network, according to some examples described in this disclosure. In some examples, exemplary process 400 may be implemented in disease risk assessment system 300 of FIG. 1. Referring to FIG. 4, process 400 may include a prediction process 410. Prediction process 410 may include capturing a user image at operation 402, extracting body shape features at operation 404, and / or receiving user data at 406. Operations 402, 404, and 406 may be performed similarly to operations 202, 204, and 206, respectively, in FIG. 2. Accordingly, a description of these operations will not be repeated.

[0047] 2, operations 404, 406 may extract or receive additional features or user data related to cardiac disease. For example, extracting body shape features 404 may further extract other features, such as blood flow, from the captured image. Receiving user data 406 may also receive additional user data related to cardiac disease, such as the user's blood flow, heart rate, or other health data.

[0048] Continuing with reference to FIG. 4 , process 410 may further include predicting 412 human body features based on the features extracted from operations 404 and 406 using a machine learning model 408. Operation 412 may, in some examples, be performed by feature extraction system 306 ( FIG. 3 ). As discussed with respect to the embodiment of FIG. 3 , the predicted human body features may include 3D body shape and body shape indicators (e.g., body volume, body fat, bone mineral density, or other indicators). Additionally, the human body features may include heart disease-related features, such as those similar to the second biomarker described in the embodiment of FIG. 1 . Thus, machine learning model 408 may include relationships between training user images / user data and various human body features.

[0049] 4, process 410 can further include assessing health risks, including diabetes risk, at operation 414. Operation 414 can, in some embodiments, be performed in disease risk assessment system 314 (FIG. 3) to generate a disease risk assessment. In some examples, the disease risk assessment can be obtained from disease risk assessment system 314 (FIG. 3). Thus, the disease risk assessment can include risk assessments for diabetes and cardiovascular disease.

[0050] Process 400 may further include a training process 420 for training machine learning model 408. In some examples, process 420 may include acquiring user images in operation 402′, extracting body shape features in operation 404′, and acquiring user data in operation 406′. Process 420 may use the images / features / data from operations 402′, 404′, and / or 406′ to train a machine learning model in operation 422. Processes 402′, 404′, and 406′ may be performed in a manner similar to processes 402, 404, and 406, respectively, except that the user images acquired from process 402′ are different from the user images acquired from process 402, and the user data acquired from 406′ are different from the user data acquired from 406. In a non-limiting example, operation 402′ may retrieve user images from a training dataset in a manner similar to operation 202′ (FIG. 2), the description of which will not be repeated.

[0051] In some examples, the training data can include multiple sets each collected from subjects in a group of subjects, with each set including corresponding ground truth data. In some examples, operation 422 can train a machine learning network based on the collected training data to generate a machine learning model 408. In some examples, the training process 422 can generate a single machine learning model 408 based on the training data collected from the group of subjects. The training process 422 can be similar to 222 (FIG. 2), except that the features used for training differ in that training is performed on a unified machine learning network that includes both diabetes and cardiac features.

[0052] In some other examples, the training process 422 can generate multiple machine learning models 408, each based on training data from a subgroup of subjects or a single subject. For example, the training process can generate machine learning models for subgroups of grained subjects divided by ethnic group, by gender, by age, by height, or by other demographic measures such as occupation, education, etc. Thus, the machine learning models 408 can include one or more disease risk models (e.g., 310 in FIG. 3). Returning to process 410, an operation of assessing disease risk can be performed using the machine learning models 408 learned from process 420.

[0053] Referring to systems 100 and 300 of FIGS. 1 and 3, respectively, the systems may include a training system for training and optimizing one or more machine learning models based on training data. The training data may be obtained from a user image database, a body scan database, and / or a medical image database. In some examples, the system may be configured to train a 3D shape model of a human body. In a non-limiting example, the 3D shape model may include multiple 3D shape parameters. Examples of 3D shape parameters may include height, weight, chest circumference measurements, or full-body anthropometric measurements, or additional parameters related to human body shape. In a non-limiting example, the 3D shape parameters may include 15 parameters. Other suitable numbers of body shape parameters are also possible. For example, three parameters may be used, including height, weight, and gender. In one or more other examples, four or more parameters or sixteen or more parameters may also be possible.

[0054] In some examples, the system can be configured to train a 2D joint model of the human body from user images, e.g., images captured from the image capture system 104. The 2D joint model can include multiple joints of the human body in a 2D domain and can be used to train a machine learning model. For example, the system can use information from the 2D joint model to obtain a 3D body shape model of the human body. The system can also use other information, such as the user's age, weight, gender, ethnicity, etc., which can be input by the user via a user interaction and display system (e.g., 102, 302 in FIGS. 1 and 3 ). In some examples, the joint model can include multiple parameters that represent skeletal joint positions. Thus, training the 2D joint model includes training the parameters of the 2D joint model.

[0055] In some examples, the system can receive a captured user image (e.g., obtained from the image capture system 104 of FIG. 1) and use the received image to estimate body joints (in a 2D region) via a machine learning network. The system can obtain a human body contour from a trained 2D joint model by connecting the joints within the 2D joint model and subsequently performing image augmentation. The contour defines the outer boundary of the 2D representation of the user.

[0056] Returning to FIGS. 1 and 3, the system can be configured to train a user body heatmap threshold. The user body heatmap can include a visual representation of body scan parameters. For example, the user body heatmap can include a representation of the body fat and / or bone mineral density of the human body.

[0057] In some examples, the body heatmap can be generated based on body scan parameters within a body scan database. The system can generate a heatmap and display body scan parameters (e.g., body fat, bone mineral density) in one or more colors according to the threshold. In some examples, the system trains a machine learning model to learn the heatmap threshold and uses the trained machine learning model to predict a future human body heatmap from a captured user image. In some examples, the training of the heatmap threshold can be performed on a per-person basis, whereby the system can monitor / estimate the individual's body parameters over time.

[0058] Examples of machine learning models used in the systems and processes described in FIGS. 1-4 can include adapted networks such as U-net, V-net, MobileNet, or other machine learning models. Additionally and / or alternatively, the machine learning model can also include a suitable convolutional neural network (CNN), such as VGG or other CNNs. In some examples, the machine learning model can learn together from user images and medical images via co-registration of these two types of images.

[0059] FIG. 5 illustrates a simplified block structure for a computing device that may be used with system 100 (of FIG. 1) or integrated into one or more components of the system. For example, the image capture systems 104, 304, the user interaction and display systems 102, 302, the biomarker extraction systems 106, 108, the feature extraction system 308, and / or other components in systems 100, 300 (FIGS. 1 and 3) may include one or more of the components shown in FIG. 5 and may be used to implement one or more blocks or perform one or more of the components or operations disclosed in FIGS. 1-4. In FIG. 5, computing device 1100 may include one or more processing elements 1102, an input / output interface 1104, a display 1106, one or more memory components 1108, a network interface 1110, and one or more external devices 1112. Each of the various components may communicate with one another via one or more buses, wireless means, etc.

[0060] The processing element 1102 may be any type of electronic device capable of processing, receiving, and / or transmitting instructions. For example, the processing element 1102 may be a central processing unit, a microprocessor, a processor, or a microcontroller. Furthermore, it should be noted that some components of the computer 1100 may be controlled by a first processor and other components may be controlled by a second processor, and that the first and second processors may or may not be in communication with each other.

[0061] Memory component 1108 is used by computer 1100 to store instructions for processing element 1102 and to store data, such as machine learning models and / or training images or data. Memory component 1108 can be, for example, a magneto-optical storage device, a read-only memory, a random access memory, an erasable programmable memory, a flash memory, or a combination of one or more types of memory components.

[0062] The display 1106 provides audio and / or visual guidance to the user, such as displaying a skeleton or other visual representation to guide the user in capturing one or more user images, or other visual representations such as may be implemented in the user interaction and display systems 102, 302 (FIGS. 1 and 3). Optionally, the display 1106 may serve as an input element that allows the user to control, operate, and calibrate various components of the computing device 1100. The display 1106 may be a liquid crystal display, a plasma display, an organic light-emitting diode display, and / or other suitable display. In embodiments in which the display 1106 is used as an input, the display may include one or more touch or input sensors, such as capacitive touch sensors, resistive grids, etc.

[0063] I / O interface 1104 allows a user to input data into computer 1100 and provides input / output for computer 1100 to communicate with other devices or services. I / O interface 1104 may include one or more input buttons, a touchpad, etc.

[0064] The network interface 1110 provides communication between the computer 1100 and other devices. For example, the network interface 1110 allows the system 100 (FIG. 1) to communicate with various components within the system over a communications network. The network interface 1110 includes one or more communication protocols, such as, but not limited to, Wi-Fi, Ethernet, Bluetooth, etc. The network interface 1110 may also include one or more hardwired components, such as a Universal Serial Bus (USB) cable. The configuration of the network interface 1110 depends on the type of communication desired and can also be modified to communicate via Wi-Fi, Bluetooth, etc.

[0065] External device 1112 is one or more devices, such as a mouse, microphone, keyboard, trackpad, etc., that can be used to provide various inputs to computing device 1100. External device 1112 can be local or remote and can be changed as desired. In some examples, external device 1112 can also include one or more additional sensors that can be used in obtaining a disease risk assessment.

[0066] The above description has broad applicability. For example, while the examples disclosed herein may focus on a centralized communication system, it should be understood that the concepts disclosed herein may equally be applied to other systems, such as distributed, centralized, or decentralized systems, or cloud systems. For example, a machine learning model (e.g., 110 in FIG. 1 or 310 in FIG. 3) or other components may reside on a server in a client / server system. A machine learning model may also reside on any device on a network, such as a mobile phone, and operate in a decentralized manner. A machine learning model, or portions thereof, may also reside in a controller VM or hypervisor in a virtual machine (VM) computing environment. Thus, one or more components in systems 100, 300 (FIGS. 1 and 3) may be implemented in various configurations to achieve optimal performance in terms of accuracy and processing speed. As such, this disclosure is intended only to provide examples of various systems and methods and is not intended to suggest that the scope of the disclosure, including the claims, is limited to these examples.

[0067] 1-5 provide advantages in assessing a user's disease risk based on user images captured from a mobile phone or other image capture device without requiring any expensive equipment on-site. The training and use of various machine learning models in the assessment system is advantageous in achieving high accuracy.

[0068] From the foregoing, it will be appreciated that, although specific embodiments of the present disclosure have been described herein for purposes of illustration, various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, the scope of the present disclosure should not be limited to any of the specific embodiments described herein.

Claims

1. an input data module configured to retrieve first input data and second input data; a processor; a computer-readable medium containing programming instructions; wherein the programming instructions, when executed, cause the processor to: creating a 3D body shape model including biometrics from the first input data and the second input data; generating health risk indicators and assessments using the biometrics, including a first disease risk assessment; generating a second risk assessment value based on the second input data; fusing the biometrics with the first input data and the second input data, and optionally the first disease risk assessment and the second disease risk assessment, to generate a multi-category disease risk assessment; A device that performs the following.

2. The apparatus of claim 1 , wherein the input data module is integrated with an image capture device configured to capture one or more images of an object.

3. The apparatus of claim 2 , wherein the second input data comprises the one or more images of the object.

4. The apparatus of claim 3 , wherein the first disease risk assessment is based on the one or more images.

5. The device of claim 1 , wherein the first disease comprises diabetes.

6. The device of claim 5 , wherein the second disease comprises a cardiovascular disease.

7. an image capture device configured to capture one or more images of the object; a processor; a computer-readable medium containing programming instructions; wherein the programming instructions, when executed, cause the processor to: generating a diabetes risk assessment based on the one or more images using a 3D body shape model; generating a cardiovascular risk assessment based on the one or more images; generating a disease risk assessment value by fusing the diabetes risk assessment value and the cardiovascular risk assessment value; A device that performs the following.

8. The apparatus of claim 7 , wherein the programming instructions are further configured to update the 3D body shape model based on an output of a fusion of the diabetes risk assessment and the cardiovascular risk assessment.

9. The programming instructions include: generating first biomarkers representing body characteristics from the one or more images using the 3D body shape model; generating the diabetes risk assessment based on the physical characteristics; The apparatus of claim 7 , further configured to:

10. The device of claim 9 , wherein the first biomarker comprises one or more of a 3D body shape or a body shape indicator.

11. 11. The device of claim 10, wherein the diabetes risk assessment is indicative of risk associated with one or more of type 2 diabetes, obesity, central obesity, or metabolic syndrome.

12. The programming instructions include: generating a second biomarker from the one or more images, the second biomarker being indicative of a cardiac disease-related biomarker; generating a cardiovascular assessment based on the second biomarker; The apparatus of claim 7 , further configured to:

13. The device of claim 12 , wherein the second biomarker comprises one or more of blood flow, blood pressure, heart rate, respiratory rate, heart rate variability, cardiac work, irregular heartbeat, or stress index.

14. The apparatus of claim 13 , wherein the cardiovascular risk assessment indicates risk associated with one or more of cardiovascular disease, heart attack, or stroke.

15. The apparatus of claim 12 , wherein the programming instructions are further configured to generate the second biomarkers from a facial image in the one or more images, the second biomarkers including at least blood flow.

16. an input data module configured to retrieve input data of interest; a processor; a computer-readable medium containing programming instructions; wherein the programming instructions, when executed, cause the processor to: generating a body signature from the input data using a disease risk model; generating a disease risk assessment value based on the human body characteristics; Let them do this, The human body characteristics include at least one or more of a 3D body shape or a body shape indicator; one or more of blood flow, blood pressure, heart rate, respiratory rate, heart rate variability, cardiac workload, irregular heart rate, or stress index; 1. An apparatus comprising:

17. The disease risk assessment value is at least one or more of type 2 diabetes, obesity, central obesity, or metabolic syndrome; one or more of cardiovascular disease, heart attack, or stroke; 17. The device of claim 16, which indicates a risk associated with

18. 20. The apparatus of claim 17, wherein the programming instructions are further configured to generate the disease risk model from one or more training images using a machine learning network.

19. The apparatus of claim 16 , wherein the input data module comprises an image capture device configured to capture one or more images of the object.

20. The apparatus of claim 19 , wherein the input data includes the one or more images of the object.

Images