Predicting user body volume for medical and medication management - Patents.com

Abstract

In some examples, a device, such as a mobile phone, may include a camera, a processor, and a computer-readable medium. The camera captures one or more images of a person. The processor may use a machine learning model to predict a body volume of the person based on the captured images. The model may be trained based on a training dataset that includes at least a plurality of user images. The device may send the predicted body volume to a medication and medical management system and receive an adjusted medication or medical plan from the system. The device may further execute the adjusted medication or medical plan.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 170910, entitled “PREDICTING USER BODY VOLUME TO MANAGE MEDICAL TREATMENT AND MEDICATION,” filed April 5, 2021, the entire disclosure of which is incorporated herein by reference. [Background technology]

[0002] Medical treatments and medications are often recommended based on a patient's weight. For example, dosages for adults are different from dosages for children. In certain medical treatments, such as cancer treatment, treatment plans may be different for adults with different weights. However, in many existing medical treatments, the patient's weight does not directly establish a relationship between the patient's physical characteristics and treatment plans, and body volume information, such as the patient's body mass index, body mass index, muscle mass, body fat mass, or body fat volume, is not taken into account.

[0003] Acquisition of a patient's body volume is often not feasible. For example, body scanning techniques such as dual energy X-ray absorptiometry (DXA or DEXA) facilitate body composition measurements but have the drawback of being expensive and time consuming. DEXA may also have associated health effects. In this regard, although the radiation doses used in this technique are generally very small, for repeated clinical and commercial use it is recommended that individuals be scanned no more than twice per year.

[0004] Medical treatments, such as cancer treatments, can affect a patient's weight or body fat (e.g., muscle loss). Very often, this effect is not taken into account in medical treatments and medication control. Therefore, it is desirable to estimate the body volume information of the human body cheaply, safely, and accurately. Summary of the Invention [Means for solving the problem]

[0005] Aspects of the present disclosure provide for a simple, fast, safe, inexpensive and completely non-invasive determination of a patient's body volume information. [Brief description of the drawings]

[0006] To illustrate the manner in which the advantages and features of the present disclosure can be obtained, reference will be made to specific examples, configurations and embodiments of the apparatus, methods and systems which are illustrated in the accompanying drawings, which will be used to describe and explain the present disclosure with additional specificity and detail, with the understanding that these drawings illustrate only exemplary embodiments of the disclosure and are not intended to limit the scope of the disclosure.

[0007] [Figure 1] FIG. 1 is a block diagram of a system for managing medical care and medication, according to some embodiments described herein. [Diagram 2] FIG. 1 is a block diagram of a system for predicting a body volume of a human body, according to some embodiments described herein. [Diagram 3] 1 is an example process for predicting a person's body volume using a machine learning network, according to some embodiments described in the present disclosure. [Figure 4] 1 illustrates an example process for managing medical care and medication regimens in a remote medical system, according to some embodiments described in the present disclosure. [Diagram 5] 1 illustrates an example of background segmentation and joint estimation according to some embodiments described in this disclosure. [Figure 6] 1A-1C illustrate examples of front and side views of a person, according to some embodiments described in the present disclosure. [Figure 7] 1A-1C show various views of a constructed human body with UV depth perception features extracted from a photographic image, according to some embodiments described in the present disclosure. [Figure 8] 1 illustrates an example of a computer-generated 3D representation of a person with a particular shape and body composition, according to some embodiments described in the present disclosure. [Figure 9]FIG. 2 is a block diagram of a computing device that may be used to implement with, or integrated into, one or more components of, the system in FIG. 1, in accordance with some embodiments described in this disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] In the following, specific details are described to allow a thorough understanding of examples of various embodiments of the present disclosure. However, it is understood that the examples described herein can be practiced without these specific details. Furthermore, the specific examples of the present disclosure described herein should not be construed to limit the scope of the present disclosure to these specific examples. In other cases, well-known circuits, control signals, timing protocols and software operations have not been shown in detail to avoid unnecessarily obscuring the embodiments of the present disclosure. Furthermore, terms such as "coupled" and "coupled" mean that two components can be electrically coupled directly or indirectly. Indirectly coupled may mean that two components are coupled via one or more intermediate components.

[0009] The devices, methods and systems disclosed herein solve many of the problems in the prior art. In short, aspects of the present disclosure provide for a simple, fast, safe, inexpensive and completely non-invasive determination of a patient's body volume information. This information can be collected and communicated frequently to a medical provider, such as a doctor or nurse, to develop an optimal medical treatment plan. As a patient's body volume characteristics change before or during treatment, such treatment plans can change and be adjusted over time to meet the patient's needs. Treatment plans may include one or more of any number of medical interventions, including medication, diet, physical therapy, and the like. In particular, when used herein as a non-limiting example, aspects of the present disclosure include cancer treatment plans, such as chemotherapy, radiation doses and radiation schedules. Details of these devices, methods and systems, as well as their advantages, are disclosed in more detail below.

[0010] FIG. 1 is a block diagram of a system 100 for managing medical care and medication, according to some embodiments described in the present disclosure. In at least one embodiment, the system 100 may include a body volume assessment system or device 104 and a medical and / or medication management system or device 106. In some embodiments, the system 100 may also include an image capture device (not shown in FIG. 1). The image capture device may be any device capable of capturing one or more images of a patient. Such images may be captured in the visible spectrum of light, for example, using electromagnetic waves from about 400 nanometers to about 700 nanometers. The image capture device may be, for example, a digital camera in a user's mobile phone, tablet, or other computing device. The image capture device may also include a stand-alone camera, such as a digital camera. A "user" as referred to herein may be any number of persons, including the patient, the patient's medical provider, or another person assisting the patient in acquiring images or using the system described herein.

[0011] The body volume assessment system or device 104 shown in FIG. 1 may be configured to extract body volume information about a user from at least one image captured by said image capture device. Once acquired, a medical and / or medication administration system or device 106 may receive the body volume information and generate and recommend a treatment plan for the patient. Details of each of the devices / systems 104, 106 shown in FIG. 1 are provided in greater detail below. Such details describe the automatic features of each of the systems 104, 106 that reduce the steps, time and equipment required to perform an advantageous assessment of the patient's body volume and the generation of a corresponding treatment plan. The system 100 may be configured such that a patient (or other user) need only take a picture using a widely available personal image capture device, such as their mobile phone camera, to inform the patient's healthcare provider of an optimal medical course or changes to existing treatment that may be required.

[0012] Accordingly, the present disclosure details a method for determining a treatment for a patient. Such a method is illustrated in at least Figures 1-4 of the present disclosure. A first step of the method may include obtaining an image of the patient. This step is illustrated in at least Figure 4. The patient may themselves capture the image. The image may be a photograph, e.g., a digital photograph, of the appearance of the patient's body, including images generated in the visible spectrum of light, as described above. For example, the patient or other user may capture an image of the patient's body at home using their everyday camera.

[0013] Another step of the method may include providing recommendations for treating the patient based on information extracted from the images. The types of information and methods for generating recommendations based on the information are described in more detail below. This step of the method is illustrated at least in FIGS. 1-4. From the patient or other user's perspective and from the healthcare provider's perspective, the above two steps may be the only steps that need to be performed. In at least one embodiment, all other intermediate steps, analysis and data generation that enable the above advantages may be performed automatically by the systems 104, 106 and devices described herein.

[0014] The body volume assessment system 104 may be configured to estimate, predict or determine information about the patient's body volume, such as body mass, muscle mass, body fat, bone mineral density, or other body volume characteristics and information. In some embodiments, the body volume assessment system 104 may be coupled to an image capture device, such as a camera, to capture user images from different angles and estimate the user's body volume information based on the user images. In some embodiments, the body volume assessment system 104 may use a machine learning network to predict the user body volume information based on the user images. Details of the body volume assessment system 104 are further described with reference to FIG. 2.

[0015] The body volume assessment system 104 may use a multivariate-based machine learning approach to build a 3D model of the patient based on one or more captured images of the patient. Correlating certain visual features of the patient based on the images, such as the patient's silhouette and joint positions (among other features), the system 104 can build a 3D model from one or more machine learning training databases that include subjects with similar age, sex, ethnicity, weight, height, etc. Once a 3D model of the patient is built, the system 104 can perform an analysis using the same or additional databases to determine various body volume characteristics found in the patient by learning (via a machine learning algorithm) from known data of the subjects in the training databases.

[0016] 1, the system 100 may further include a medical and medication management system 106 coupled to the body volume assessment system 104 via any suitable communication link. For example, the communication link may be wired, such as through Ethernet or cable. The communication link may also be wireless, such as through Wi-Fi, a 5G network, a mesh network, a femtocell network, or other communication network, or a combination thereof.

[0017] The medical and medication management system 106 may be configured to receive the body volume assessment data from the body volume assessment system 104 and use the body volume information in medical and medication management. For example, the body volume may indicate the patient's body fat, muscle mass, total volume, body mass, bone mineral density, or other characteristics of the human body. The medical system 106 may recommend medication adjustments to the physician based on the amount (mass) or volume of the patient's body fat. Such recommendations may be performed periodically, e.g., daily, weekly, monthly, or any other suitable period. For example, the body volume assessment system 104 may capture a user image of the patient every week (e.g., via a mobile phone) and use the captured image to estimate, determine, or predict various information regarding the patient's body volume. The medical and medication management system 106 may provide recommendations for adjusting medication dosages or medical regimens. The physician may then evaluate the medical and medication management system 106's recommendations and decide accordingly to appropriately adjust (or not adjust) the patient's treatment and medication regimen.

[0018] For example, if a patient's body fat percentage or body fat volume increases, as indicated by the body volume assessment system 104, the medical and medication management system 106 may provide a recommendation to increase the medication dosage or medication type. Similarly, if a patient's body fat percentage or body fat volume decreases, as indicated by the body volume assessment system 104, the medical and medication management system 106 may provide a recommendation to decrease the medication dosage or medication type. The recommendations provided by the medication dosage or medical plan 106 may depend on any of the body volume information collected by the body volume assessment system 104.

[0019] Advantageously, in this manner, as the patient's body changes due to aging, health, or other events that affect the patient's body composition, system 100 can assist the physician (and / or patient) in providing the most effective dosages and treatments.

[0020] Additionally or alternatively, in at least one embodiment of the system 100, the body volume assessment system 104 may determine the location and / or distribution of body composition components such as fat and muscle, and the dosage or medical plan 106 may make recommendations accordingly. For example, in at least one embodiment, the body volume assessment system 104 is configured to identify excess abdominal fat. Once communicated, the dosage or medical plan 106 may recommend changes in radiation therapy to abdominal organs. The same is true for breast cancer treatments, where the fat composition of a patient's breasts changes before or during treatment.

[0021] 2 is a block diagram of a system 200 for predicting body volume information of a human body according to some embodiments described in this disclosure. In some embodiments, the system 200 may be implemented in the body volume assessment system 104 (FIG. 1). In some embodiments, the system 200 may include an image capture system 204 that includes an image capture device such as a camera (e.g., a mobile phone with a built-in camera or a fixed camera). The image capture device may be configured to capture one or more photographic images of a user from multiple viewpoints.

[0022] The system 200 may include a user interaction display system 202 coupled to an image capture system 204. The user interaction display system 202 may include a computer display configured to provide visual and audio aids to guide the user to capture an optimal image depending on whether the user is capturing the image themselves or is being captured by another person. For example, during user image capture, the user interaction display system 202 may display a visual representation of a human body, such as a skeleton or silhouette, to guide the user to move a body part to a desired position so that the captured body image matches the visual representation.

[0023] In some embodiments, the representation may include a body outline, bounding box, or other symbol indicating a recommended position of one or more body parts or the entire body of the user. For example, the system 202 may display a representation of an arm that guides the user to move the arm or extend the arm in a desired posture. Similarly, the system 202 may display a representation of the entire body, which may include the head, arms, legs, chest, and / or other parts of the body. The representation may be generated based on an initially captured user image from an image capture system, such that it is displayed on the display of the user interaction display system 202 in proportion to the captured image.

[0024] In some embodiments, the system 200 may include a 3D representation system 208 configured to receive a captured user image from the image capture system 204 and generate a 3D representation of the human body using the user image. In some embodiments, the 3D representation system 208 may predict a 3D representation of the human body based on the user image using a machine learning network 214. The machine learning network 214 may be configured to train a machine learning model based on one or more databases of various types. For example, the machine learning model may be trained based on previously captured user images stored in a user image database 222. Additionally and / or alternatively, the machine learning model may be trained based on body scan parameters in a body scan database 224. In some embodiments, the body scan parameters may be collected from DEXA scans of various parts of the human body. For example, the body scan parameters may include body fat and / or bone mineral density (measured in Z-scores and T-scores) of different parts of the body, such as the torso, thighs, or hips. Additionally and / or alternatively, the machine learning model may be trained based on medical images stored in the medical image database 226. In some examples, the medical images may include medical images captured from a medical imaging device, such as a CT or MRT. In some examples, the medical images may include anatomical landmarks. In non-limiting examples, the anatomical landmarks may include specific parts of the body, such as the mid-tummy point (navel) or one or more joints of the body. The use of various types of databases, such as the user image database 222, the body scan database 224, and the medical image database 226, may more accurately predict the 3D representation of the human body by incorporating various features of the human body. Details of the machine learning network 214 are described further in this disclosure.

[0025] 2, the system 200 may further include a body measurement adjustment system 210. In some embodiments, the system 210 may receive the predicted 3D representation of the human body provided by the 3D representation system 208 and fine-tune the 3D representation. Additionally, the system 210 may generate or determine body volume information of the individual based on the 3D representation. As shown in FIG. 1, the body volume may indicate the patient's body fat, body mass, muscle mass, water content, bone mineral density, or other characteristics of the human body. Similar to the 3D representation system 208, the body measurement adjustment system 210 may also use a machine learning network 214, the details of which are described further in this disclosure.

[0026] In some embodiments, one or more of the 3D representation system 208 and the body measurement adjustment system 210 may receive data from various sensors and / or databases. For example, the GPS sensor(s) 216 may be configured to provide a user geographic location. The health sensor 218 may be configured to provide user health data such as the user's heart rate, pulse, etc. In some embodiments, the health sensor 218 may be a smart watch that provides the health data. The health sensor 218 may also include a MEMs sensor configured to provide acceleration associated with the user's movements. This data may be indicative of the user's activity. Additionally and / or alternatively, user fitness data indicative of the user's activity may be used to construct the 3D representation of the human body.

[0027] Additionally and / or alternatively, the body measurement adjustment system 210 may receive medical data of the user and use the medical data to adjust the body measurements. For example, if the user is undergoing cancer treatment, the user's body fat may change. If the user is taking medication, the user's body fat may also change depending on the dosage, the duration the user has been taking the medication, the user's diet, the user's age and gender, etc. In some examples, the medical data may include the type and duration of treatment, the name of the medication, the dosage and duration of the medication, the user's diet, age, and gender. These various medical data may be used in various combinations to adjust the body measurements. Additionally, the medical data may be used in combination with the user's activity data during treatment. For example, a user undergoing cancer treatment who is not exercising (or moving) may tend to lose muscle or gain body fat.

[0028] 2, the system 200 may include an image feature extraction system 212 coupled to the machine learning network 214. The image feature extraction system 212 may be configured to extract various features from the captured image for use by the machine learning network 214.

[0029] Additionally and / or alternatively, system 200 may include a classification system 206 coupled to 3D representation system 208 or body measurement adjustment system 210. In some embodiments, classification system 206 may be configured to score images captured from image capture system 204 to determine the acceptability of the images to machine learning network 214. In some embodiments, classification system 206 may include a machine learning system configured to analyze captured images (e.g., from image capture system 204) and score the images, with the score indicating the acceptability. For example, the score may indicate whether the background of the image is good or bad, or how good or bad the user's position, posture, or orientation is relative to what is expected.

[0030] Further details of the system 200 are provided below.

[0031] Classification System In some implementations, the classification system 206 may be configured to cluster the captured user images from 204 and segment the foreground and background of the images. In a non-limiting implementation, the system 206 may perform facial recognition on the user images to identify the user's face and then use the face as a seed region for the foreground. In some implementations, the system may generate a skeleton based on the segmentation results and display the skeleton on the display of the user interaction display system 202 to guide the user to capture additional images. The skeleton may be based on the segmented foreground regions. In other implementations, the system may prompt the user (e.g., via the user interaction display system 202) to input the user's height and weight and use the user's height and weight to build the skeleton based on a pre-stored database. Additional data such as gender, age, etc. may be used to build the skeleton.

[0032] In some examples, the classification system 206 may use a 2D joint model and generate a contour based on the 2D joint model.

[0033] In some embodiments, the classifier 206 may analyze and score the captured user image to determine whether the image is acceptable for a subsequent machine learning process, such as the machine learning network 214 used in the 3D representation system 108 or the body measurement adjustment system 210.

[0034] Machine learning networks and 3D representation systems 2, the machine learning network 214 may include a training system that trains and optimizes one or more machine learning models based on training data. The training data may be obtained from a user image database 222, a body scan database 224, and / or a medical image database 226. The trained machine learning models may be provided to a 3D representation system 208 to predict a 3D representation of the human body based on the user images. Details of the machine learning network 214 and the 3D representation system 208 are further described with reference to FIG. 3.

[0035] 3 is an example process for predicting a person's body volume using a machine learning network, according to some embodiments described in this disclosure. In some embodiments, the process 300 may be implemented (performed) in the body volume assessment system 104 (FIG. 1) or the 3D representation system 208 (FIG. 2). With reference to FIG. 3, the process 300 may include a prediction process 310. The prediction process 310 may include a step of capturing a user image in operation 302. For example, the operation 302 may be performed in the image capture system 204 (FIG. 2) to obtain one or more user images, such as a front image, a side image, a back image, and / or a user image from different angles. The user image may be a face image, an upper body image, and / or a full body image.

[0036] In some embodiments, the process 310 may further include extracting body features at operation 304. In some embodiments, the operation 304 may be implemented in the image feature extraction system 212 (FIG. 2). Examples of body features may include a 2D silhouette representing a foreground of a human body, 2D joints, or other body features. In some embodiments, the body features may be obtained based on a captured user image. Additionally and / or alternatively, the process 310 may further include receiving user data at operation 306. For example, at operation 306, data entered by a user may be received, such as the user's weight, height, age, gender, ethnic group, etc. In some embodiments, this operation may be implemented in the user interaction display system 202 (FIG. 2). At operation 306, one or more databases may be evaluated to retrieve other user data, such as user health fitness data.

[0037] Continuing to refer to FIG. 3, the process 310 may further include, at operation 312, determining, estimating or predicting body volume information using a machine learning network based on the extracted body shape features (304) and / or the received user data (306). In some examples, operation 312 may be implemented in a 3D representation system (208 in FIG. 2) or a body volume assessment (104 in FIG. 1). As discussed with respect to the embodiments of FIGS. 1 and 2, the predicted body volume information may include body fat volume or percentage, body mass index, bone mineral density or other compositional characteristics of a human. The body volume information may be predicted using a machine learning model 308. The machine learning model may include a body volume model. In some examples, the machine learning may include a 3D body shape model. The body volume and 3D body shape model may be trained. Details of training the model are described further below.

[0038] 3, process 300 may include a training process 320 to train machine learning model 308. In some embodiments, process 320 may include acquiring a user image at operation 302', extracting body shape features at operation 304', and acquiring user data at operation 306'. Process 320 may train a machine learning model at operation 322 using images / features / data from operations 302', 304', and / or 306'. Processes 302', 304', and 306' may be performed in the same manner as processes 302, 304, and 306, respectively, except that the user image acquired at operation 302' differs from the user image captured at operation 302, and the user data acquired at 306' differs from the user data acquired at 306.

[0039] In a non-limiting example, operation 302' may retrieve user images from a training dataset. For example, the training dataset may include a collection of previously captured or collected training user images and / or training user data along with ground truth data associated with the training dataset. The ground truth data may include ground truth 3D body type and / or other body characteristics such as body volume information.

[0040] In some examples, the training data may include multiple and / or multivariate sets, each collected from subjects in a group of subjects, with each set containing a corresponding ground truth data set. In some examples, a machine learning network may be trained to generate a machine learning model 308 based on the collected training data in operation 322. In some examples, the training process 322 may generate a single machine learning model 308 based on the training data collected from the group of subjects.

[0041] In some other embodiments, the training process 322 may generate multiple machine learning models 308, each based on training data from a subgroup of subjects or a single subject. For example, the training process may generate machine learning models for subgroups of training subjects divided by ethnic group, gender, age, height, or other demographic measures such as occupation, education, etc. Thus, the machine learning models 308 may include one or more models (e.g., 308 of FIG. 3).

[0042] Returning to process 310, user images and user data (e.g., weight, height, age, etc.) may be acquired in real time from the user via the image capture system 204 and / or the user interaction display system 202. The user data may be acquired from one or more sensors or a database (e.g., user fitness data), as previously described. The operation of predicting body volume may be performed using the machine learning model 308 learned from process 320.

[0043] In some examples, the 3D shape model may include multiple 3D shape parameters. Examples of 3D shape parameters may include height, weight, chest measurement, etc., or additional parameters associated with a human body type. In a non-limiting example, the 3D shape parameters may include 15 parameters. Other suitable numbers of body type parameters may be possible.

[0044] In some examples, the machine learning training process 320 may be configured to train a 2D joint model of the human body based on a user image, e.g., a user image captured from the image capture system 204. The 2D joint model may include a plurality of joints of the human body in a 2D domain and may be used in the training operation 322. For example, in the operation 322, information from the 2D joint model may be used to obtain a 3D body type model of the human body. The machine learning network may use other information, such as the user's age, weight, gender, ethnic group, etc., that the user may input via the user interaction display system 202. In some examples, the 2D joint model may include a plurality of parameters that represent the positions of the skeletal joints. Thus, training the 2D joint model includes training the parameters of the 2D joint model. Examples of the positions of the 2D joints are further illustrated in FIG. 5A-C.

[0045] 5A-5C show examples of background segmentation and joint estimation according to some embodiments described in the present disclosure. For example, FIG. 5A shows an example user image from an image capture system (e.g., 204 in FIG. 2). FIG. 5B shows a segmentation result of the captured user image in FIG. 5A, where the foreground is separated from the background. FIG. 5B includes a number of skeletal joint points (shown in red) of the user's body shown in the user image, such as the chin, shoulder point, elbow, wrist, hip, knee, and ankle, visually identified from the image. FIG. 5C shows the corresponding estimated skeletal joint points (shown in green) overlaid on the same image, where the estimated skeletal joint points are obtained from the machine learning network 214 (FIG. 2) or machine learning model (308 in FIG. 3).

[0046] The skeletal joint points shown in Figure 5B and the skeletal joint points shown in Figure 5C are overlaid on the original user image in Figure 5A. As shown, the skeletal joint points from the machine learning network (green) match closely with the skeletal joint points. In some embodiments, the estimated joint points identified by the machine learning network 214 (Figure 2) or the machine learning model (308 in Figure 3) may be more accurate in determining the actual location of the skeletal joint points than the estimated joint points identified by a human.

[0047] In some embodiments, a system (e.g., 100 in FIG. 1 or 200 in FIG. 2) may receive a captured user image (e.g., from image capture system 204 in FIG. 2) and use the received image to estimate the body joints (in the 2D domain) via machine learning network 214. The system may obtain a body contour from the trained 2D joint model by connecting the joints in the 2D joint model and subsequently augmenting the image. The contour defines the outer boundary of the user's 2D representation. FIGS. 6A and 6B show examples of a 2D representation of a person in front and side views, respectively.

[0048] Returning to FIG. 2, the machine learning network 214, which may implement the training process 320 (FIG. 3), may be configured to train a user body volume. In some embodiments, the training data may be retrieved from the body scan database 224. The machine learning network 214 may train a machine learning model to learn weight and parameters indicative of body volume, and use the trained machine learning model to predict future volumes of the human body based on captured user images. In some embodiments, the 3D body shape and body volume training may be performed on an individual basis, allowing the system to monitor / estimate the individual's body parameters over time.

[0049] Examples of machine learning models used in the machine learning network 214 may include U-net, V-net, or other machine learning models. Additionally and / or alternatively, the machine learning models may include a suitable convolutional neural network (CNN), such as VGG-16 or other CNNs. In some embodiments, the machine learning network 214 may perform training in a training process (e.g., 320 of FIG. 3) using both user images (e.g., 222) and medical images (e.g., 226), as further described below.

[0050] Returning to FIG. 1 , in some embodiments, once the machine learning model is trained (learned), the system may use the learned model to estimate (or predict) a 3D body, a 2D joint model, a body volume, or a combination thereof based on the captured user images. In some embodiments, once the system (e.g., 100 in FIG. 1 ) completes predicting the body volume based on one or more captured user images, the system may compare the distribution of the patient's body volume over time to evaluate the effectiveness of the medical treatment or medication. The system may further adjust the treatment or medication based on the evaluation results. For example, the medical and medication management system 106 may determine that the patient's body fat has increased over the course of the treatment period and adjust the treatment plan or medication accordingly.

[0051] Returning to FIG. 2, the 3D representation system 208 may incorporate various features when using the machine learning network 214. In some implementations, the 3D representation system 208 may incorporate the user's location (e.g., from a GPS sensor) to estimate the user's ethnic group and use a 3D or 2D body model specific to that particular ethnic group. In other words, the machine learning network 214 may train different models for different ethnic groups, each associated with a geographic location. When training a model for a given ethnic group, images of the user of that ethnic group may be used. When predicting a user's 3D body shape and body volume, the system may determine the user's ethnic group based on the user's location when the image is captured and then use the corresponding model or training data associated with that ethnic group.

[0052] Image Feature Extraction System 2, in the image feature extraction system 212, image features extracted from the captured image may include 3D shape parameters, depth information, micro-motion of the user, or curvature information. In some embodiments, the feature extraction system 212 may be configured to determine health information of the user without using health sensors. For example, the system may use pixel amplification to detect micro-motion of the body (e.g., detecting micro-motion of breathing based on face images) to obtain health data such as heart rate without using health sensors.

[0053] In some embodiments, the image feature extraction system 212 may be further configured to extract UV depth perception features from the captured image. The UV depth features may include the normals of the person's surface that give the perception of depth information. In some embodiments, the system may use a UV depth sensor to obtain depth information, such as a vector perpendicular to the surface. In some embodiments, the depth sensor may be attached to a mobile phone or other portable electronic device. The system may use the depth information to determine the curvature of the surface of the person's body. This information may be used by the machine learning network 214 to improve the accuracy of the estimation. For example, the machine learning network 214 may determine the fat / muscle distribution based on the depth of the human body. FIG. 7 illustrates an example of a 3D body shape estimated using depth information at various angles. Any number of depth perception sensors may provide data that is incorporated into the present system and method to further improve the accuracy of the machine learning estimation. In one embodiment, the depth perception sensor may include, but is in no way limited to, a structured light sensor, a time-of-flight sensor, a camera array sensor, or any combination thereof.

[0054] Body Measurement Adjustment System 2, the body measurement adjustment system 210 may be configured to adjust body composition data or measurements. For example, if the user is athletically active (e.g., runs or jogs regularly, this information may be obtained from health sensors or based on pixel-based image analysis of amplified body movements), the system may adjust the body composition to the "lean" side. For example, the system may obtain an estimated body fat from the machine learning network 214 and increase the estimated body fat by percentage. Conversely, if the user is inactive, the system may adjust the body volume to decrease the estimated body fat by percentage.

[0055] In some embodiments, the body measurement adjustment system 210 may be further configured to determine whether the posture of the human body is acceptable to the machine learning network for accurate body measurements. The system may use the trained 3D body shape to make the determination. If the posture of the human body is not acceptable, the system may instruct the user (e.g., via the user interaction display system 202) to correct the posture and, for example, stand still in order to obtain a correct measurement. For example, a message may be displayed to the user on the image capture screen to instruct the user to stand still. Alternatively and / or additionally, the system may display the skeleton to guide the user to ensure that the user's body in the image capture screen matches the skeleton. Furthermore, the system may detect a user position or posture that does not match the machine learning network's preferred skeleton orientation for accurate body measurements and may reorient the image to closely approximate the desired preferred skeleton orientation. In one embodiment, detection of a joint or a user's body part outside the expected area, or an asymmetric orientation between the sides of the captured image, may be interpreted by the system as the user orientation in an unpreferred direction, whereby the system reorients the image to more closely approximate the desired preferred skeleton orientation.

[0056] 8A-B show examples of computer-generated 3D representations of a person having a particular shape and body volume, according to some embodiments described in this disclosure. For example, one or more components of system 200 (FIG. 2) can be implemented to estimate the person's 3D body shape and body volume based on user images captured from different angles. In some embodiments, the estimated 3D body shape and body volume may be displayed in an avatar that mimics the person's exact shape and profile in different clothing.

[0057] Various embodiments of Figures 1-8, such as 100 in Figure 1, can be implemented in a single computing system. For example, in Figure 1, the body volume assessment system 104 and the medical and medication management system 106 can be integrated into a single system, such as a server or cloud system in a hospital where a doctor can capture a user image of a patient and perform / adjust the patient's medical care or medication. Alternatively, the system 100 in Figure 1 can be implemented in a telemedicine environment including a user device, such as a mobile phone, and a medical / medication management system, which will be further described with reference to Figure 4.

[0058] FIG. 4 illustrates an exemplary process for managing medical care and drug regimens in a telemedicine system according to some embodiments described in the present disclosure. In a non-limiting example, the process 400 may be implemented in a user device, such as a mobile phone, to predict a user's body volume in real time. For example, the process 400 may include capturing a user image in operation 402. Optionally, the process 400 may include a user interaction process that prompts the user to take multiple user images from different perspectives. For example, the operation 402 may be implemented by a user interaction display system (202 in FIG. 2). The process 400 may also include predicting a user body volume in operation 404. For example, the operation 404 may include one or more operations in process 310 (FIG. 3). Alternatively and / or additionally, the operation 404 may implement one or more operations in process 320 (FIG. 3).

[0059] Returning to FIG. 4 , process 400 may further include monitoring a change in body volume over time at operation 406. For example, the change in body volume may include a difference (change) in predicted body volume obtained from operation 404 at different times, e.g., days apart, one month apart, or months apart. Process 400 may transmit the instantaneous predicted body volume and / or the change in body volume to a healthcare and medication management system at operation 408. In some examples, the healthcare and medication management system may be implemented in system 106 ( FIG. 1 ). Process 400 may further receive a medication and treatment plan from the healthcare and medication management system at operation 410. For example, the received medication plan may include an adjusted dosage of a drug prescribed to the patient. In response, process 400 may further perform an adjusted medication, e.g., at operation 412. Performing an adjusted medication, e.g., a change in dosage or a switch in drug, may include transmitting the drug information to a drug dispensing server of a pharmacy dispensing the drug to the patient.

[0060] 4, process 420 may be implemented in a medical and medication management system, such as 106 of FIG. 1. Process 420 may include receiving user body volume data in operation 422. For example, the user body volume data may be received from a user device. The user body volume data may include a predicted user body volume or a change in body volume over time, which may be obtained from process 400. Additionally, process 420 may also receive other user data in operation 424. The other user data may include a medical record of the user, which may be retrieved from a medical record database. Process 420 may generate or adjust a medication and / or treatment plan in operation 426 based on the user body volume data and / or the other user data. For example, operation 426 may be implemented in medical and medication management system 106 (FIG. 1). In some examples, a physician may intervene to confirm, verify, or adjust the medical care or medication in operation 426. Process 420 may transmit the medication and medical plan to the user's device in operation 428. Alternatively and / or additionally, operation 428 may include transmitting the medication to a dispensing server of the pharmacy. As described with respect to process 400, operation 410 may receive the transmitted medication and medical plan.

[0061] FIG. 9 illustrates a simplified block structure of a computing device that may be used with the system 100 (FIG. 1) or integrated into one or more components of the system. For example, the body volume assessment system 104, the medical and medication administration system 106, or one or more components of the systems 104, 106, such as the image capture system 204, the user interaction display system 202, the classification system 206, the 3D representation system 208, the body measurement adjustment system 210, the image feature extraction system 212, or the machine learning network 214, may include one or more of the components illustrated in FIG. 9 and may be used to implement one or more blocks or perform one or more of the components or operations disclosed in FIGS. 1-8. In FIG. 9, the computing device 1100 may include one or more processing elements 1102, an input / output interface 1104, a display 1106, one or more memory components 1108, a network interface 1110, and one or more external devices 1112. Each of the various components may communicate with each other via one or more buses, wireless means, or the like.

[0062] The processing element 1102 may be any type of electronic device capable of processing, receiving, and / or transmitting instructions. For example, the processing element 1102 may be a central processing unit, a microprocessor, a processor, or a microcontroller. Additionally, it should be noted that some components of the computer 1100 may be controlled by a first processor and other components may be controlled by a second processor, and that the first and second processors may or may not communicate with each other.

[0063] The memory component 1108 is used by the computer 1100 to store instructions for the processing elements 1102 and to store data such as a knowledge base (e.g., 222, 224, 226 in FIG. 2). The memory component 1108 may be, for example, a magneto-optical storage device, a read-only memory, a random access memory, an erasable programmable memory, a flash memory, or a combination of one or more types of memory components.

[0064] Display 1106 provides audio and / or visual guidance to the user, for example, displaying a skeleton or other visual representation to guide the user to capture one or more user images, or other visual representations that may be implemented in user interaction display system 202 (FIG. 2). Optionally, display 1106 may serve as an input element that allows a user to control, manipulate, and calibrate various components of computing device 1100. Display 1106 may be a liquid crystal display, a plasma display, an organic light emitting diode display, and / or other suitable display. In embodiments in which display 1106 is used as an input, the display may include one or more touch sensors or input sensors, such as capacitive touch sensors, resistive grids, etc.

[0065] I / O interface 1104 allows a user to input data into computer 1100 and provides input and output for computer 1100 to communicate with other devices or services (e.g., user interaction display system 202 of FIG. 2). I / O interface 1104 may include one or more input buttons, a touch pad, or the like.

[0066] The network interface 1110 provides for communication between the computer 1100 and other devices. For example, the network interface 1110 may implement the communications link 102 (FIG. 1) that allows various systems to communicate with each other. The network interface 1110 may include one or more communications protocols, such as, but not limited to, WiFi, Ethernet, Bluetooth, etc. The network interface 1110 may also include one or more hardwired components, such as, a Universal Serial Bus (USB) cable, etc. The configuration of the network interface 1110 depends on the type of communication desired and may be modified to communicate via WiFi, Bluetooth, etc.

[0067] External device 1112 is one or more devices that can be used to provide various inputs to computing device 1100, such as a mouse, microphone, keyboard, trackpad, etc. External device 1112 can be local or remote and can vary as desired. In some examples, external device 1112 can include one or more additional sensors, such as sensors 216, 218, 220 (FIG. 2), that can be used to obtain physical measurements of the user.

[0068] The foregoing description has broad application. For example, although the embodiments disclosed herein focus on a central communication system, it should be understood that the concepts disclosed herein can be applied to other systems, such as distributed systems, centralized or distributed systems, or cloud systems, as well. For example, the machine learning network 214 or other components (FIG. 1) can reside on a server in a client / server system. The machine learning network 114 can also reside on any device on the network, such as a mobile phone, and operate in a distributed manner. The machine learning network 114 or a portion thereof can also reside on a controller virtual machine (VM) or a hypervisor in a VM computing environment. Thus, one or more components in the system 100 (FIG. 1) can be implemented in various configurations to achieve optimal performance in terms of accuracy and processing speed. Thus, the present disclosure is merely intended to provide examples of various systems and methods, and is not intended to suggest that the scope of the present disclosure, including the claims, is limited to these examples.

[0069] The various embodiments described in Figures 1-9 provide the advantage of predicting accurate body composition and therefore body measurements based on user images captured from a mobile phone or other image capture device without the need for expensive equipment at a given location.

[0070] Each of the embodiments, examples or configurations described in the detailed description above may include any of the features, options and possibilities described in the present disclosure, including those based on other independent embodiments, and may include any combination of any of the features, options and possibilities described in the present disclosure and drawings. Further examples consistent with the present teachings described herein are described in the following numbered paragraphs.

[0071] (Item 1) An apparatus comprising: a processor; and a computer-readable medium containing programming instructions that, when executed, cause the processor to perform the steps of predicting body volume information using a machine learning model and one or more images of the subject, receiving a medical plan based on the body volume information, and executing the received medical plan.

[0072] (Item 2) The device described in Item 1, wherein the programming instructions, when executed, further cause the processor to perform the steps of sending predicted body volume information to a drug therapy and medication management system via a communications link and receiving a medication or medical plan from the drug therapy and medication management system.

[0073] (Item 3) The device described in item 1 or 2, wherein the machine learning model includes a body volume model.

[0074] (Item 4) The device described in any one of items 1 to 3, wherein, when executed, the programming instructions further cause the processor to perform a step of training the machine learning model using a machine learning network based at least on a training dataset including a plurality of user images.

[0075] (Item 5) The device of any one of Items 1 to 4, wherein the programming instructions, when executed, further cause the processor to execute the received medical plan by transmitting the medical plan to a drug dispensing server.

[0076] (Item 6) An apparatus including a processor and a computer-readable medium including programming instructions that, when executed, cause the processor to perform the steps of receiving user body volume data from a user device via a communications link, adjusting a medical plan based on the received user body volume data, and transmitting the adjusted medical plan to the user device.

[0077] (Item 7) The device described in Item 6, wherein the programming instructions, when executed, further cause the processor to perform the steps of receiving user data from a user device and adjusting the medical plan further based on the received user body volume data.

[0078] (Item 8) The device described in item 6 or 7, wherein the user body volume data includes a change in body volume over a period of time.

[0079] (Item 9) The device according to any one of items 6 to 8, wherein the user body volume data includes one or more of body fat, body mass, or bone mineral density.

[0080] (Item 10) The device according to any one of items 6 to 9, wherein the user body volume data includes a distribution of body fat.

[0081] (Item 11) A method for determining a treatment for a patient, the method comprising the steps of acquiring an image of the patient and providing a recommendation for treating the patient based on information extracted from the image.

[0082] (Item 12) The method described in Item 11, wherein the image includes an external appearance of the patient's body.

[0083] (Item 13) The method according to item 11 or 12, wherein the treatment includes a change in the dosage or dosage of a medication administered to the patient.

[0084] (Item 14) The method according to Item 13, wherein the dosage of the drug comprises a cancer drug.

[0085] (Item 15) The method according to any one of items 11 to 14, wherein the step of acquiring the image includes a step of taking a photograph using an image capture device.

[0086] (Item 16) The method according to item 15, wherein the patient takes the photograph.

[0087] (Item 17) The method according to item 15 or 15, wherein the patient is photographed in the visible light spectrum having electromagnetic wavelengths of about 400 nanometers to about 700 nanometers.

[0088] (Item 18) The method described in items 11 to 17, wherein the information extracted from the image includes body volume information of the patient.

[0089] (Item 19) The method according to any one of items 11 to 18, wherein the information extracted from the image includes the patient's body mass index.

[0090] (Item 20) The method according to any one of items 11 to 19, wherein the information extracted from the image includes the patient's body volume index.

[0091] (Item 21) A system for managing a patient's treatment, comprising: an image capture device that captures images in the visible spectrum; a body volume assessment device that extracts body volume information from at least one image captured by the image capture device; and a medical recommendation device that provides treatment recommendations based on the body volume information extracted by the body volume assessment device.

[0092] (Item 22) The body volume assessment device includes a processor and a computer-readable medium containing programming instructions, which, when executed, cause the processor to perform a step of predicting body volume information using a machine learning model and one or more images captured by an image capture device.The system described in Item 21.

[0093] (Item 23) The system described in Item 21 or 22, wherein the medical recommendation device includes a processor and a computer-readable medium containing programming instructions, which, when executed, cause the processor to perform the steps of receiving body volume information from a body volume assessment device, determining a treatment recommendation based on the body volume information extracted by the body volume assessment device, and providing a treatment recommendation to a treatment provider.

[0094] The articles "a," "an," and "the" are intended to mean that there is one or more elements in the preceding description. The terms "comprising," "including," and "having" are intended to be inclusive and mean that there may be additional elements other than the listed elements. Additionally, it should be understood that references to "one embodiment" or "embodiments" of the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments incorporating the recited features. Any numerical values, percentages, ratios, or other values ​​described herein are intended to include that value, and other values ​​that are "about" or "approximately" the recited value, as would be understood by one of ordinary skill in the art encompassed by the embodiments of the present disclosure. Thus, the recited values ​​should be interpreted broadly enough to encompass values ​​that are at least sufficiently close to the recited value to perform the desired function or achieve the desired result. The recited values ​​include at least the expected variation (variation) in a suitable manufacturing or production process, and may include values ​​within 5%, 1%, 0.1%, or 0.01% of the recited value.

[0095] Those skilled in the art should understand in light of this disclosure that equivalent structures do not depart from the spirit and scope of the disclosure, and that various changes, substitutions, and modifications can be made to the embodiments disclosed herein without departing from the spirit and scope of the disclosure. Equivalent structures, including functional "means-plus-function" clauses, are intended to cover structures described herein as performing the described functions, including both structural equivalents that operate in the same manner, and equivalent structures that provide the same function. It is the express intention of the applicants not to make means-plus-function or other functional claim statements in any of the claims, except for claims using the language "means for" with the relevant function. All additions, deletions, and modifications to the embodiments that come within the meaning and scope of the claims are intended to be embraced by the claims.

[0096] As used herein, the terms "approximately," "about," and "substantially" refer to an amount close to the stated amount that still performs a desired function or achieves a desired result. For example, the terms "approximately," "about," and "substantially" can refer to an amount in the range of 5% or less, 1% or less, 0.1% or less, 0.01% or less of the stated amount. Furthermore, it should be understood that any directions or reference frames in the foregoing description are relative directions or movements only. For example, references to "up" and "down," or "above" or "below" are merely intended to describe the relative positions or movements of associated elements.

[0097] From the foregoing, it will be appreciated that, although specific embodiments of the present disclosure have been described herein for purposes of illustration, various modifications can be made without departing from the spirit and scope of the present disclosure, and therefore, the scope of the present disclosure should not be limited to any of the specific embodiments described herein.

Claims

1. A processor; and a computer-readable medium containing programming instructions that, when executed, cause the processor to: predicting body volume information using the machine learning model and one or more images of the subject; adjusting the predicted body volume information based on at least one of user health data, medical data, or an analysis of one or more images of the subject; receiving a medical plan based on the adjusted body volume information; and executing the received medical plan.

2. The programming instructions, when executed, cause the processor to: transmitting the predicted body volume information to a medical therapy management system via a communication link; and receiving the medical plan from the medical treatment and medication management system.

3. The apparatus of claim 1 , wherein the machine learning model comprises a body volume model.

4. 4. The apparatus of claim 3, wherein the programming instructions, when executed, further cause the processor to perform the step of training the machine learning model using a machine learning network based at least on a training dataset comprising a plurality of user images.

5. 10. The apparatus of claim 1, wherein the programming instructions, when executed, further cause the processor to perform the step of implementing the received medical plan by transmitting the medical plan to a drug dispensing server.

6. A processor; and a computer-readable medium containing programming instructions that, when executed, cause the processor to: receiving user body volume data from a user device via a communications link; adjusting the user body volume data based on at least one of user health data, medical data, or analysis of one or more images of a subject on the user device; adjusting a medical plan based on the adjusted user body volume data; transmitting the coordinated medical care to the user device.

7. The programming instructions, when executed, cause the processor to: receiving user data from the user device; and adjusting the medical plan further based on the received user body volume data.

8. The apparatus of claim 6 , wherein the user body volume data comprises a change in body volume over a period of time.

9. The apparatus of claim 6 , wherein the user body volume data includes one or more of body fat, body mass, or bone mineral density.

10. The apparatus of claim 6 , wherein the user body volume data includes body fat distribution.

11. 1. A method for determining a treatment for a patient, comprising: acquiring an image of the patient; predicting body volume information using a machine learning model and the images; adjusting the predicted body volume information based on at least one of user health data, medical data, or analysis of the image; and providing a recommendation for treating the patient based on the adjusted body volume information.

12. The method of claim 11 , wherein the image comprises an external appearance of the patient's body.

13. The method of claim 11 , wherein the treatment comprises a dosage or change in dosage administered to the patient.

14. 14. The method of claim 13, wherein the dosage of a drug comprises a cancer drug.

15. The method of claim 11 , wherein obtaining the image comprises taking a photograph with an image capture device.

16. The method of claim 15 , wherein the patient takes the photograph.

17. 16. The method of claim 15, wherein the patient is photographed in the visible light spectrum having electromagnetic wavelengths between about 400 nanometers and about 700 nanometers.

18. The method of claim 11 , wherein the information extracted from the images includes body volume information of the patient.

19. The method of claim 11 , wherein the information extracted from the images includes the body mass index of the patient.

20. The method of claim 11 , wherein the information extracted from the images includes a body volume index of the patient.

21. an image capture device for capturing images in the visible spectrum; a body volume assessment device for extracting body volume information from at least one image captured by the image capture device; an adjustment device for adjusting the extracted body volume information based on at least one of user health data, medical data, or an analysis of the at least one image; and a medical recommendation device that provides treatment recommendations based on the adjusted body volume information.

22. The body volume assessment device comprises: A processor; and a computer-readable medium containing programming instructions that, when executed, cause the processor to:

22. The system of claim 21, further comprising: predicting the body volume information using a machine learning model and one or more images captured by the image capture device.

23. The medical recommendation device comprises: A processor; and a computer-readable medium containing programming instructions that, when executed, cause the processor to: receiving the body volume information from the body volume assessment device; determining the treatment recommendation based on body volume information extracted by the body volume assessment device; and providing the treatment recommendation to a treatment provider.

Images