Predicting user body volume for managing medical care and medication.

Abstract

In some examples, a device, such as a mobile phone, may include a camera, a processor, and a computer-readable medium. The camera captures one or more images of a person. The processor may use a machine learning model to predict a body volume of the person based on the captured images. The model may be trained based on a training dataset that includes at least a plurality of user images. The device may send the predicted body volume to a medication and medical management system and receive an adjusted medication or medical plan from the system. The device may further execute the adjusted medication or medical plan.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the priority of U.S. Provisional Patent Application No. 63 / 170910, titled "PREDICTING USER BODY VOLUME TO MANAGE MEDICAL TREATMENT AND MEDICATION", filed on April 5, 2021, the entire disclosure of which is incorporated herein by reference.

Background Art

[0002] Medical treatment and medication are often recommended based on a patient's weight. For example, the dosage for an adult is different from that for a child. In certain medical treatments such as cancer treatment, the treatment plan may vary for adults with different weights. However, in many existing medical treatments, the patient's weight does not directly establish a relationship between the patient's physical characteristics and the treatment plan, and body volume information such as the patient's body mass index, body mass index, muscle mass, body fat mass, or body fat volume is not considered.

[0003] It is often not possible to obtain a patient's body volume. For example, body scan technologies such as dual - energy X - ray absorptiometry (DXA or DEXA) facilitate body composition measurement but have the disadvantages of being expensive and time - consuming. Also, DEXA may have related health effects. In this regard, although the radiation dose used in this technology is generally very small, it is recommended that an individual be scanned only twice a year for repeated clinical and commercial use.

[0004] Medical treatments such as cancer treatment can affect a patient's weight or body fat (e.g., muscle loss). In very many cases, this effect is not considered in medical treatment and medication control. Therefore, it is desirable to estimate the body volume information of the human body cheaply, safely, and accurately.

Summary of the Invention

Means for Solving the Problems

[0005] Aspects of this disclosure provide a simple, rapid, safe, inexpensive, and completely non-invasive method for determining patient body volume information. [Brief explanation of the drawing]

[0006] To illustrate how the advantages and features of this disclosure can be obtained, specific examples, configurations and embodiments of the apparatus, methods and systems shown in the accompanying drawings will be used for further specific and detailed description of this disclosure, with the understanding that these drawings only illustrate typical embodiments of this disclosure and do not limit the scope of this disclosure.

[0007] [Figure 1] This is a block diagram of a system for managing medical care and medication, relating to some of the embodiments described in this disclosure. [Figure 2] This is a block diagram of a system for predicting the body volume of a human being, according to some embodiments described in this disclosure. [Figure 3] This is an exemplary process for predicting human body volume using a machine learning network, relating to some embodiments described in this disclosure. [Figure 4] This disclosure illustrates exemplary processes for managing medical and medication plans in a telemedicine system, relating to several embodiments described herein. [Figure 5] Examples of background segmentation and joint estimation relating to some embodiments described in this disclosure are shown. [Figure 6] Examples of front and side views of a person relating to some embodiments described in this disclosure are shown. [Figure 7] The following are various figures of a constructed human body having UV depth perception features extracted from photographic images, relating to some of the embodiments described in this disclosure. [Figure 8] Examples of computer-generated 3D representations of people having specific shapes and body compositions, relating to some embodiments described in this disclosure, are shown. [Figure 9]This is a block diagram of a computing device that may be used to implement with the system in Figure 1 or integrated into one or more components of the system, relating to some embodiments described in this disclosure. [Modes for carrying out the invention]

[0008] The following describes specific details to allow for a full understanding of the various embodiments of this disclosure. However, it should be understood that the embodiments described herein can be implemented without these specific details. Furthermore, the specific embodiments of this disclosure described herein should not be construed as limiting the scope of this disclosure to those specific embodiments. In other cases, well-known circuits, control signals, timing protocols, and software operations are not described in detail to avoid unnecessarily obscuring the embodiments of this disclosure. Furthermore, terms such as “coupled” and “coupled” mean that two components may be electrically coupled directly or indirectly. Indirect coupling may mean that two components are coupled through one or more intermediate components.

[0009] The apparatus, methods, and systems disclosed herein solve many of the problems of the prior art described above. In other words, aspects of this disclosure provide a simple, rapid, safe, inexpensive, and completely non-invasive method for determining a patient's body volume information. This information can be frequently collected and communicated to healthcare providers, such as physicians or nurses, in order to develop an optimal medical treatment plan. Because a patient's body volume characteristics change before or during treatment, such a treatment plan can be modified and adjusted over time to meet the patient's needs. The treatment plan may include one or more of any number of medical interventions, including drug therapy, diet therapy, and physiotherapy. In particular, aspects of this disclosure, when used herein as a non-limiting embodiment, include cancer treatment plans such as chemotherapy, radiation dose, and radiation schedule. The details of these apparatus, methods, and systems, as well as their advantages, are disclosed in more detail below.

[0010] Figure 1 is a block diagram of a medical and medication management system 100 relating to some embodiments described herein. In at least one embodiment, system 100 may include a body volume assessment system or device 104 and a medical and / or medication management system or device 106. In some embodiments, system 100 may also include an image acquisition device (not shown in Figure 1). The image acquisition device may be any device capable of acquiring one or more images of a patient. Such images may be acquired in the visible spectrum of light using, for example, electromagnetic waves in the range of about 400 nanometers to about 700 nanometers. The image acquisition device may be, for example, a digital camera in a user's mobile phone, tablet, or other computing device. The image acquisition device may also include a standalone camera, such as a digital camera. The term “user” as used herein may be any number of persons, including a patient, a patient’s healthcare provider, or other persons assisting the patient in acquiring images or using the system described herein.

[0011] The body volume assessment system or device 104 shown in Figure 1 may be configured to extract body volume information about a user from at least one image captured by the image acquisition device. Once acquired, the medical and / or medication management system or device 106 can receive the body volume information and create and recommend a treatment plan for the patient. The details of each of the devices / systems 104 and 106 shown in Figure 1 are provided below in more detail. Such details describe the automated features of each of the systems 104 and 106 that reduce the steps, time, and equipment required to perform a favorable assessment of a patient's body volume and generate a corresponding treatment plan. The system 100 may be configured such that it is sufficient for the patient (or another user) to take a photograph using a widely available personal image acquisition device, such as the camera on their mobile phone, to notify the patient's healthcare provider of the optimal medical course or any necessary changes to existing treatment.

[0012] Accordingly, this disclosure details a method for determining a patient's treatment. Such a method is shown in at least Figures 1–4 of this disclosure. The first step of the method may include a step of acquiring an image of the patient. This step is shown in at least Figure 4. The patient himself may acquire the image. The image may be a photograph of the patient's physical appearance, such as a digital photograph, including an image generated in the visible spectrum of light, as described above. For example, the patient or another user may acquire an image of the patient's body at home using their everyday camera.

[0013] Another step of the method may include providing recommendations for treating the patient based on information extracted from the images. The types of information and the methods for generating recommendations based on that information are described in more detail below. This step of the method is shown in at least Figures 1-4. From the perspective of the patient or other users, and from the perspective of the healthcare provider, only the above two steps may need to be performed. In at least one embodiment, all other intermediate steps, analyses, and data generation that enable the above benefits may be performed automatically by systems 104, 106 and apparatus described herein.

[0014] The body volume evaluation system 104 may be configured to estimate, predict, or determine information about a patient's body volume, such as body mass, muscle mass, body fat, and bone mineral concentration, or other body volume characteristics and information. In some embodiments, the body volume evaluation system 104 may be coupled to an image acquisition device such as a camera to capture user images from different angles and estimate the user's body volume information based on the user images. In some embodiments, the body volume evaluation system 104 may use a machine learning network to predict user body volume information based on user images. Further details of the body volume evaluation system 104 will be described with reference to Figure 2.

[0015] The body volume assessment system 104 may use a multivariate-based machine learning approach to construct a 3D model of the patient based on one or more captured images of the patient. By correlating specific visual features of the patient based on the images, such as the patient's silhouette and joint positions (among other features), the system 104 can construct a 3D model from one or more machine learning training databases containing subjects with similar age, sex, ethnicity, weight, height, etc. Once the patient's 3D model is constructed, the system 104 can determine various body volume features observed in the patient by performing analysis using the same or additional databases and learning (via machine learning algorithms) from known data of subjects in the training databases.

[0016] Continuing to refer to Figure 1, the system 100 may further include a medical and medication management system 106 coupled to the body volume assessment system 104 via any suitable communication link. For example, the communication link may be wired, such as via Ethernet or cable. The communication link may also be wireless, such as Wi-Fi, a 5G network, a mesh network, a femtocell network, or other communication networks, or a combination thereof.

[0017] The medical and medication management system 106 may be configured to receive body volume evaluation data from the body volume evaluation system 104 and use body volume information in the management of medical treatment and medication. For example, the body volume may indicate the patient's body fat, muscle mass, total volume, body mass, bone mineral density, or other characteristics of the human body. The medical system 106 may recommend an adjustment of the dosage to the physician based on the amount (mass) or volume of the patient's body fat. Such recommendations may be made periodically, for example, daily, weekly, monthly, or any other appropriate period. For example, the body volume evaluation system 104 may capture the patient's user image (e.g., via a mobile phone) weekly and use the captured image to estimate, determine, or predict various information regarding the patient's body volume. The medical and medication management system 106 may provide recommendations for adjusting the dosage or medical plan. The physician may then evaluate the recommendations of the medical and medication management system 106 and decide whether to appropriately adjust (or not adjust) the patient's treatment and medication plan accordingly.

[0018] For example, as shown by the body volume evaluation system 104, if the patient's body fat percentage or body fat volume increases, the medical and medication management system 106 may provide a recommendation to increase the dosage or type of medication. Similarly, for example, as shown by the body volume evaluation system 104, if the patient's body fat percentage or body fat volume decreases, the medical and medication management system 106 may provide a recommendation to decrease the dosage or type of medication. The recommendations provided by the dosage or medical plan 106 may depend on any of the body volume information collected by the body volume evaluation system 104.

[0019] Advantageously, in this way, when the patient's body changes due to aging, health, or other events affecting the patient's body composition, the system 100 can assist the physician (and / or patient) in providing the most effective dosage and treatment.

[0020] Additionally or alternatively, in at least one embodiment of the system 100, the body volume assessment system 104 may determine the location and / or distribution of body composition components such as fat and muscle, and the dosage or treatment plan 106 may make recommendations accordingly. For example, in at least one embodiment, the body volume assessment system 104 is configured to identify excessive abdominal fat. When communicated, the dosage or treatment plan 106 may recommend a change in radiation therapy for abdominal organs. The same is true for breast cancer treatment where the fat composition of the patient's breast changes before or during treatment.

[0021] FIG. 2 is a block diagram of a system 200 for predicting body volume information of a human body according to some embodiments described in the present disclosure. In some embodiments, the system 200 may be implemented in a body volume assessment system 104 (FIG. 1). In some embodiments, the system 200 may include an image capture system 204 that includes an image capture device such as a camera (e.g., a mobile phone with a built-in camera or a fixed camera). The image capture device may be configured to capture one or more photographic images of the user from multiple viewpoints.

[0022] The system 200 may include a user interaction display system 202 coupled to the image capture system 204. The user interaction display system 202 may include a computer display, which is configured to provide visual and audio assistance for guiding the user to capture an optimal image depending on whether the user is capturing the image himself or the image is being captured by another person. For example, during the capture of the user image, the user interaction display system 202 may display a visual representation of the human body such as a skeleton or silhouette so that the captured body image matches the visual representation by guiding the user to move a part of the body to a desired position.

[0023] In some embodiments, the representation may include the outline of a human body, a bounding box, or other symbols indicating the recommended position of one or more body parts or the whole body of the user. For example, system 202 may display a representation of an arm that guides the user to move the arm or extend the arm in a desired posture. Similarly, system 202 may display a representation of the whole body, which may include the head, arms, legs, chest and / or other parts of the body. The representation can be generated based on a user image initially captured from the image capture system so that it is displayed on the display of the user interaction display system 202 in proportion to the captured image.

[0024] In some embodiments, system 200 may include a 3D representation system 208 configured to receive user images captured from an image acquisition system 204 and generate a 3D representation of the human body using the user images. In some embodiments, the 3D representation system 208 may use a machine learning network 214 to predict a 3D representation of the human body based on the user images. The machine learning network 214 may be configured to train a machine learning model on one or more databases of various types. For example, the machine learning model may be trained on previously captured user images stored in a user image database 222. Additionally and / or alternatively, the machine learning model may be trained on body scan parameters in a body scan database 224. In some embodiments, the body scan parameters may be collected from DEXA scans of various parts of the human body. For example, the body scan parameters may include body fat and / or bone mineral concentrations (measured by Z-scores and T-scores) of different parts of the body, such as the torso, thighs, or hips. Additionally and / or alternatively, the machine learning model may be trained on medical images stored in the medical image database 226. In some embodiments, the medical images may include medical images acquired from a medical imaging device, such as a CT or MRT. In some embodiments, the medical images may include anatomical landmarks. In non-limiting embodiments, anatomical landmarks may include specific parts of the body, such as the navel or one or more joints of the body. The use of various types of databases, such as the user image database 222, the body scan database 224, and the medical image database 226, can more accurately predict the 3D representation of the human body by incorporating various features of the human body. Further details of the machine learning network 214 are described in this disclosure.

[0025] Referring further to Figure 2, the system 200 may further include a anthropometric adjustment system 210. In some embodiments, the system 210 may receive a predicted 3D representation of the human body provided by the 3D representation system 208 and fine-tune the 3D representation. Additionally, the system 210 may generate or determine an individual's body volume information based on the 3D representation. As shown in Figure 1, the body volume may represent the patient's body fat, body mass, muscle mass, water content, bone mineral concentration, or other characteristics of the human body. Similar to the 3D representation system 208, the anthropometric adjustment system 210 may also use a machine learning network 214, the details of which are further described in this disclosure.

[0026] In some embodiments, one or more of the 3D representation system 208 and the body measurement adjustment system 210 may receive data from various sensors and / or databases. For example, a GPS sensor 216(or more) may be configured to provide the user's geographical location. A health sensor 218 may be configured to provide user health data such as the user's heart rate and pulse. In some embodiments, the health sensor 218 may be a smartwatch providing the above health data. The health sensor 218 may also include a MEM sensor configured to provide acceleration associated with the user's movement. This data can indicate the user's activity level. Additionally and / or alternatively, user fitness data indicating the user's activity level may be used to construct a 3D representation of the human body.

[0027] Additionally and / or alternatively, the anthropometric adjustment system 210 can receive and use the user's medical data to adjust the anthropometric measurements. For example, if the user is undergoing cancer treatment, the user's body fat may change. If the user is taking medication, the user's body fat may also change depending on the dosage, the duration of medication use, the user's diet, the user's age, and gender. In some embodiments, the medical data may include the type and duration of treatment, the name of the medication, the dosage and duration of medication use, the user's diet, age, and gender. These various medical data may be used in various combinations to adjust the anthropometric measurements. Additionally, the medical data may be used in combination with the user's activity data during medical treatment. For example, a user undergoing cancer treatment who is not exercising at all (or is inactive) may tend to have decreased muscle mass or increased body fat.

[0028] Referring further to Figure 2, the system 200 may also include an image feature extraction system 212 coupled to the machine learning network 214. The image feature extraction system 212 may be configured to extract various features used by the machine learning network 214 from the captured image.

[0029] Additionally and / or alternatively, system 200 may include a classification system 206 coupled to a 3D representation system 208 or a physiometric adjustment system 210. In some embodiments, the classification system 206 may be configured to score images acquired from an image acquisition system 204 to determine the acceptability of the images to a machine learning network 214. In some embodiments, the classification system 206 may include a machine learning system configured to analyze images acquired (e.g., from the image acquisition system 204) and score the images, with the score indicating acceptability. For example, the score may indicate whether the background of the image is good or bad, or how good or bad the user's position, posture or orientation is compared to what is expected.

[0030] Further details of System 200 are provided below.

[0031] Classification system In some embodiments, the classification system 206 may be configured to cluster user images captured from 204 and divide the images into foreground and background. In non-limiting embodiments, the system 206 may perform face recognition on the user images to identify the user's face and then use the face as a seed region for the foreground. In some embodiments, the system may generate a skeleton based on the division results and display the skeleton on the display of the user interaction display system 202 to guide the user to capture additional images. The skeleton may be based on the divided foreground regions. In other embodiments, the system may instruct the user to enter their height and weight (e.g., via the user interaction display system 202) and use the user's height and weight to construct a skeleton based on a pre-stored database. Additional data such as gender and age may be used to construct the skeleton.

[0032] In some embodiments, the classification system 206 may use a 2D joint model and generate contours based on the 2D joint model.

[0033] In some embodiments, classification 206 may analyze and score the captured user images to determine whether the images are acceptable for subsequent machine learning processes, such as a machine learning network 214 used in a 3D representation system 108 or a physiometric adjustment system 210.

[0034] Machine learning networks and 3D representation systems Referring further to Figure 2, the machine learning network 214 may include a training system that trains and optimizes one or more machine learning models based on training data. The training data may be obtained from a user image database 222, a body scan database 224, and / or a medical image database 226. The trained machine learning models may be provided to a 3D representation system 208 to predict a 3D representation of the human body based on user images. Further details of the machine learning network 214 and the 3D representation system 208 are described with reference to Figure 3.

[0035] Figure 3 shows an exemplary process for predicting a person's body volume using a machine learning network, according to some embodiments described in this disclosure. In some embodiments, process 300 may be implemented (executed) in a body volume evaluation system 104 (Figure 1) or a 3D representation system 208 (Figure 2). Referring to Figure 3, process 300 may include a prediction process 310. The prediction process 310 may include a step of acquiring user images in operation 302. For example, operation 302 may be performed in an image acquisition system 204 (Figure 2) to acquire one or more user images, such as a front view, a side view, a back view, and / or user images from different angles. The user images may be a face image, an upper body image, and / or a full body image.

[0036] In some embodiments, process 310 may further include a step of extracting body shape features in operation 304. In some embodiments, operation 304 may be implemented in an image feature extraction system 212 (Figure 2). Examples of body shape features may include a 2D silhouette representing the foreground of a human body, 2D joints, or other body shape features. In some embodiments, body shape features may be obtained based on an acquired user image. Additionally and / or alternatively, process 310 may further include a step of receiving user data in operation 306. For example, in operation 306, data entered by the user, such as the user's weight, height, age, gender, ethnic group, etc., may be received. In some embodiments, this operation may be implemented in a user interaction display system 202 (Figure 2). In operation 306, one or more databases may be evaluated to retrieve other user data, such as user health and fitness data.

[0037] Continuing with reference to Figure 3, process 310 may further include the step in operation 312 of determining, estimating, or predicting body volume information using a machine learning network based on extracted body shape features (304) and / or received user data (306). In some embodiments, operation 312 may be implemented in a 3D representation system (208 in Figure 2) or a body volume assessment (104 in Figure 1). As discussed with respect to the embodiments in Figures 1 and 2, the predicted body volume information may include body fat volume or body fat percentage, body mass index, bone mineral concentration, or other human compositional characteristics. The body volume information may be predicted using a machine learning model 308. The machine learning model may include a body volume model. In some embodiments, the machine learning may include a 3D body shape model. The body volume and 3D body shape models may be trained. Details of model training will be described further.

[0038] Referring further to Figure 3, process 300 may include a training process 320 for training a machine learning model 308. In some embodiments, process 320 may include the steps of acquiring a user image in operation 302', extracting body shape features in operation 304', and acquiring user data in operation 306'. Process 320 may train the machine learning model in operation 322 using the images / features / data from operations 302', 304', and / or 306'. Processes 302', 304', and 306' may be performed in the same manner as processes 302, 304, and 306, respectively, except that the user image acquired in operation 302' is different from the user image taken in operation 302, and the user data acquired in 306' is different from the user data acquired in 306.

[0039] In a non-limiting embodiment, in operation 302', user images may be retrieved from the training dataset. For example, the training dataset may include a collection of previously captured or collected training user images and / or training user data, along with ground truth data associated with the training dataset. The ground truth data may include other physical features such as ground truth 3D body shape and / or body volume information.

[0040] In some embodiments, the training data may include multiple and / or multivariate sets, each collected from subjects within a group of subjects, with each set containing a corresponding ground truth dataset. In some embodiments, in operation 322, a machine learning network may be trained to generate a machine learning model 308 based on the collected training data. In some embodiments, the training process 322 may generate a single machine learning model 308 based on training data collected from a group of subjects.

[0041] In some other embodiments, the training process 322 may generate multiple machine learning models 308, each machine learning model 308 being based on training data from a subgroup of subjects or a single subject. For example, the training process may generate machine learning models for subgroups of training subjects divided by ethnic group, sex, age, height, or other demographic measures such as occupation or education. Thus, the machine learning models 308 may include one or more models (e.g., 308 in Figure 3).

[0042] Returning to process 310, user images and user data (e.g., weight, height, age, etc.) may be acquired in real time from the user via the image acquisition system 204 and / or the user interaction display system 202. User data may be acquired from one or more sensors or databases (e.g., user fitness data) as described above. The operation of predicting body volume may be performed using the machine learning model 308 learned from process 320.

[0043] In some embodiments, the 3D shape model may include multiple 3D shape parameters. Examples of 3D shape parameters may include height, weight, chest circumference measurements, or additional parameters associated with human body shape. In non-limiting embodiments, the 3D shape parameters may include 15 parameters. Other appropriate numbers of body shape parameters are also possible.

[0044] In some embodiments, the machine learning training process 320 may be configured to train a 2D joint model of the human body based on user images, for example, user images acquired from an image acquisition system 204. The 2D joint model may include multiple joints of the human body in a 2D region and may be used in the training action 322. For example, in action 322, information from the 2D joint model may be used to obtain a 3D body shape model of the human body. The machine learning network may also use other information of the user, such as the user's age, weight, gender, and ethnic group, which the user may input via the user interaction display system 202. In some embodiments, the 2D joint model may include multiple parameters representing the positions of skeletal joints. Therefore, training the 2D joint model includes training the parameters of the 2D joint model. Examples of 2D joint positions are further shown in Figures 5A-C.

[0045] Figures 5A to 5C show examples of background segmentation and joint estimation according to some embodiments described in this disclosure. For example, Figure 5A shows an exemplary user image from an image acquisition system (e.g., 204 in Figure 2). Figure 5B shows the segmentation result of the acquired user image in Figure 5A, where the foreground is separated from the background. Figure 5B includes multiple skeletal joint points (shown in red) of the user's body shown in the user image, visually identified from the image, such as the jaw, shoulders, elbows, wrists, hips, knees, and ankles. Figure 5C shows the corresponding estimated skeletal joint points (shown in green) superimposed on the same image, where the multiple estimated skeletal joint points are obtained from a machine learning network 214 (Figure 2) or a machine learning model (308 in Figure 3).

[0046] The skeletal joint points shown in Figure 5B and Figure 5C are overlaid on the original user image in Figure 5A. As shown, the skeletal joint points from the machine learning network (green) precisely match the actual skeletal joint points. In some embodiments, the estimated joint points identified by the machine learning network 214 (Figure 2) or the machine learning model (308 in Figure 3) may be more accurate in determining the actual location of the skeletal joint points than the estimated joint points identified by humans.

[0047] In some embodiments, the system (e.g., 100 in Figure 1 or 200 in Figure 2) may receive an acquired user image (e.g., from an image acquisition system 204 in Figure 2) and use the received image to estimate the joints of the body (in a 2D region) via a machine learning network 214. The system may obtain the contour of the human body from a trained 2D joint model by connecting the joints in the 2D joint model and then augmenting the image. The contour defines the outer boundary of the 2D representation of the user. Figures 6A and 6B show examples of a front view and a side view of a person in a 2D representation, respectively.

[0048] Returning to Figure 2, the machine learning network 214, which can implement the training process 320 (Figure 3), may be configured to train user body volume. In some embodiments, the training data may be retrieved from a body scan database 224. The machine learning network 214 may train a machine learning model to learn weight and parameters that indicate body volume, and use the trained machine learning model to predict future human body volume based on captured user images. In some embodiments, the training of 3D body shape and body volume may be performed on an individual basis, so that the system can monitor / estimate an individual's body parameters over time.

[0049] Examples of machine learning models used in the machine learning network 214 may include U-net, V-net, or other machine learning models. Additionally and / or alternatively, the machine learning model may include a suitable convolutional neural network (CNN), e.g., VGG-16 or other CNNs. In some embodiments, the machine learning network 214 may be trained in the training process (e.g., 320 in Figure 3) by using both user images (e.g., 222) and medical images (e.g., 226). This will be further explained below.

[0050] Returning to Figure 1, in some embodiments, once a machine learning model is trained (learned), the system may use the trained model to estimate (or predict) a 3D body, a 2D joint model, body volume, or a combination thereof based on captured user images. In some embodiments, once the system (e.g., 100 in Figure 1) has completed predicting body volume based on one or more captured user images, the system may compare the distribution of the patient's body volume over time to evaluate the effectiveness of medical treatment or medication. The system may further adjust the treatment or medication based on the evaluation results. For example, the medical and medication management system 106 may determine that the patient's body fat has increased during the treatment period and adjust the treatment plan or dosage accordingly.

[0051] Returning to Figure 2, the 3D representation system 208 can incorporate various features when using the machine learning network 214. In some embodiments, the 3D representation system 208 may incorporate user location (e.g., from a GPS sensor) to estimate the user's ethnic group and use a 3D or 2D body model specific to that particular ethnic group. In other words, the machine learning network 214 may train different models for different ethnic groups, each associated with a geographical location. When training a model for a given ethnic group, images of users from that ethnic group may be used. When predicting a user's 3D body shape and body volume, the system may determine the user's ethnic group based on the user's location when the image was acquired, and then use the corresponding model or training data associated with that ethnic group.

[0052] Image Feature Extraction System Referring further to Figure 2, the image features extracted from the captured image in the image feature extraction system 212 may include 3D shape parameters, depth information, and information on the user's minute movements or curvature. In some embodiments, the feature extraction system 212 may be configured to determine the user's health information without using health sensors. For example, the system may use pixel amplification to detect minute body movements (e.g., detecting minute movements during breathing based on a facial image) and acquire health data such as heart rate without using health sensors.

[0053] In some embodiments, the image feature extraction system 212 may be further configured to extract UV depth perception features from the captured image. The UV depth features may include surface normals of a person that give a sense of depth information. In some embodiments, the system may use a UV depth sensor to acquire depth information such as vectors perpendicular to the surface. In some embodiments, the depth sensor may be attached to a mobile phone or other portable electronic device. The system may use the depth information to determine the curvature of the surface of a person's body. This information can be used by a machine learning network 214 to improve the accuracy of the estimation. For example, the machine learning network 214 can determine the fat / muscle distribution based on the depth of the human body. Figure 7 shows an example of a 3D body shape estimated using depth information at various angles. Any number of depth perception sensors can provide data to be incorporated into the system and method to further improve the accuracy of the machine learning estimation. In one embodiment, the depth perception sensor may include, but is not limited to, a structural light sensor, a time-of-flight sensor, a camera array sensor, or any combination thereof.

[0054] Body Measurement Adjustment System Referring further to Figure 2, the body measurement adjustment system 210 may be configured to adjust body composition data or measurements. For example, if the user is athletically active (e.g., regularly runs or jogs, this information may be obtained from health sensors or from pixel-based image analysis of amplified body movements), the system may adjust the body composition to the "leaner" side. For example, the system may obtain estimated body fat from the machine learning network 214 and increase the estimated body fat by a percentage. Conversely, if the user is inactive, the system may adjust the body volume to decrease the estimated body fat by a percentage.

[0055] In some embodiments, the anthropometric adjustment system 210 may further be configured to determine whether the posture of the human body is acceptable to the machine learning network for accurate anthropometric measurements. The system can make this determination using a trained 3D body shape. If the posture of the human body is unacceptable, the system can instruct the user to correct their posture, for example, to stand still (e.g., via the user interaction display system 202) in order to obtain correct measurements. For example, a message may be displayed to the user on the imaging screen to instruct the user to stand still. Alternatively and / or additionally, the system may display the skeleton to guide the user to ensure that the user's body in the imaging screen matches the skeleton. Furthermore, the system can detect the user's position or posture that does not match a preferred skeletal orientation for accurate anthropometric measurements of the machine learning network and can reorient the image to more closely approximate a desired preferred skeletal orientation. In one embodiment, detection of joints or parts of the user's body outside the expected area, or asymmetric orientation between the sides of the captured image, can be interpreted by the system as a user orientation in an undesirable direction, thereby causing the system to reorient the image to more closely approximate a desired preferred skeletal orientation.

[0056] Figures 8A and 8B show examples of computer-generated 3D representations of a person having a specific shape and body volume, according to some embodiments of the present disclosure. For example, one or more components of system 200 (Figure 2) may be implemented to estimate a person's 3D body shape and body volume based on user images captured from different angles. In some embodiments, the estimated 3D body shape and body volume may be displayed on an avatar that mimics the exact shape and profile of the person wearing different clothing.

[0057] Various embodiments shown in Figures 1 to 8, such as system 100 in Figure 1, can be implemented in a single computing system. For example, in Figure 1, the body volume assessment system 104 and the medical and medication management system 106 can be integrated into a single system, such as a hospital server or cloud system where a physician can capture user images of a patient and perform / adjust the patient's medical or medication dosage. Alternatively, system 100 in Figure 1 may be implemented in a telemedicine environment including a user device such as a mobile phone and a medical / medication management system, which will be further described with reference to Figure 4.

[0058] Figure 4 shows an exemplary process for managing medical and medication planning in a telemedicine system according to some embodiments described herein. In non-limiting embodiments, process 400 may be implemented in a user device such as a mobile phone to predict the user's body volume in real time. For example, process 400 may include a step of capturing a user image in operation 402. Optionally, process 400 may include a user interaction process that instructs the user to take multiple user images from different viewpoints. For example, operation 402 can be implemented by a user interaction display system (202 in Figure 2). Process 400 may also include a step of predicting the user's body volume in operation 404. For example, operation 404 may include one or more operations in process 310 (Figure 3). Alternatively and / or additionally, operation 404 may implement one or more operations in process 320 (Figure 3).

[0059] Returning to Figure 4, process 400 may further include the step of monitoring changes in body volume over time in operation 406. For example, changes in body volume may include differences (changes) in predicted body volume obtained from operation 404 at different points in time, e.g., at intervals of several days, one month, or several months. Process 400 may transmit instantaneous predicted body volume and / or changes in body volume to a medical and medication management system in operation 408. In some embodiments, the medical and medication management system may be implemented in system 106 (Figure 1). Process 400 may further receive medication and treatment plans from the medical and medication management system in operation 410. For example, the received medication plan may include adjusted dosages of drugs prescribed to the patient. Accordingly, process 400 may further perform adjusted medication, for example in operation 412. The step of performing adjusted medication, e.g., changes in dosage or switching drugs, may include the step of transmitting drug information to a drug dispensing server at a pharmacy that dispenses drugs to the patient.

[0060] Referring further to Figure 4, process 420 may be implemented in a medical and medication management system, for example, 106 in Figure 1. Process 420 may include a step in operation 422 to receive user body volume data. For example, user body volume data may be received from a user device. User body volume data may include predicted user body volume or changes in body volume over time, which can be obtained from process 400. Additionally, process 420 may also receive other user data in operation 424. Other user data may include the user's medical record, which can be retrieved from a medical record database. Process 420 may generate or adjust a medication and / or treatment plan in operation 426 based on the user body volume data and / or other user data. For example, operation 426 can be implemented in a medical and medication management system 106 (Figure 1). In some embodiments, a physician may intervene in operation 426 to confirm, verify, or adjust the medical or medication. Process 420 may transmit the medication and medical plan to the user's device in operation 428. Alternatively and / or additionally, operation 428 may include the step of sending a medication to the pharmacy's dispensing server. As described with respect to process 400, operation 410 may receive the sent medication and medical plan.

[0061] Figure 9 shows a simplified block structure of a computing device that may be used with system 100 (Figure 1) or integrated into one or more components of the system. For example, the body volume assessment system 104, the medical and medication management system 106, or one or more components of systems 104, 106, such as the image acquisition system 204, the user interaction display system 202, the classification system 206, the 3D representation system 208, the anthropometric adjustment system 210, the image feature extraction system 212, or the machine learning network 214 may include one or more of the components shown in Figure 9 and may be used to implement one or more blocks or to perform one or more of the components or operations disclosed in Figures 1-8. In Figure 9, the computing device 1100 may include one or more processing elements 1102, an input / output interface 1104, a display 1106, one or more memory components 1108, a network interface 1110, and one or more external devices 1112. Each of the various components can communicate with one or more buses, wireless means, etc.

[0062] The processing element 1102 may be any type of electronic device capable of processing, receiving, and / or transmitting instructions. For example, the processing element 1102 may be a central processing unit, a microprocessor, a processor, or a microcontroller. Additionally, some components of the computer 1100 may be controlled by the first processor, and other components may be controlled by the second processor, and the first and second processors may or may not communicate with each other.

[0063] The memory component 1108 is used by the computer 1100 to store instructions for the processing element 1102 and to store data such as a knowledge base (e.g., 222, 224, 226 in Figure 2). The memory component 1108 may be, for example, a magneto-optical memory, read-only memory, random-access memory, erasable programmable memory, flash memory, or a combination of one or more types of memory components.

[0064] The display 1106 provides the user with audio and / or visual guidance, for example, by displaying a skeleton or other visual representation to guide the user to capture one or more user images, or by displaying other visual representations that can be implemented in the user interaction display system 202 (Figure 2). Optionally, the display 1106 may also function as an input element, allowing the user to control, operate, and calibrate various components of the computing device 1100. The display 1106 may be a liquid crystal display, a plasma display, an organic light-emitting diode display, and / or other suitable display. In embodiments where the display 1106 is used as an input, the display may include one or more touch sensors or input sensors, such as capacitive touch sensors, resistive grids, etc.

[0065] The I / O interface 1104 allows the user to input data into the computer 1100 and provides input / output for the computer 1100 to communicate with other devices or services (e.g., the user interaction display system 202 in Figure 2). The I / O interface 1104 may include one or more input buttons, a touchpad, and the like.

[0066] The network interface 1110 provides communication between the computer 1100 and other devices. For example, the network interface 1110 can implement a communication link 102 (Figure 1) that enables various systems to communicate with each other. The network interface 1110 includes, but is not limited to, one or more communication protocols, such as WiFi, Ethernet, Bluetooth®, etc. The network interface 1110 may also include one or more wiring components, such as a Universal Serial Bus (USB) cable. The configuration of the network interface 1110 depends on the desired type of communication and may be modified to communicate via WiFi, Bluetooth®, etc.

[0067] The external device 1112 is one or more devices that can be used to provide various inputs to the computing device 1100, such as a mouse, microphone, keyboard, trackpad, etc. The external device 1112 may be local or remote and can vary as needed. In some embodiments, the external device 1112 may include one or more additional sensors, such as sensors 216, 218, 220 (Figure 2), which can be used to obtain user body measurements.

[0068] The foregoing description has broad applicability. For example, while the embodiments disclosed herein focus on a central communication system, it should be understood that the concepts disclosed herein are equally applicable to other systems such as distributed systems, centralized or decentralized systems, or cloud systems. For example, the machine learning network 214 or other components (Figure 1) may reside on a server in a client / server system. The machine learning network 114 may also reside on any device on the network, such as a mobile phone, and operate in a distributed manner. The machine learning network 114 or a part thereof may also reside on a controller virtual machine (VM) or a hypervisor in a VM computing environment. Thus, one or more components in system 100 (Figure 1) may be implemented in various configurations to achieve optimal performance in terms of accuracy and processing speed. Therefore, this disclosure is intended to provide embodiments of various systems and methods and does not imply that the scope of this disclosure, including the claims, is limited to these embodiments.

[0069] The various embodiments shown in Figures 1-9 offer advantages in accurately predicting body composition and, consequently, performing body measurements based on user images captured from a mobile phone or other image acquisition device, without requiring expensive equipment at a predetermined location.

[0070] Each of the embodiments, examples, or configurations described in the above detailed description may include any of the features, options, and possibilities described in this disclosure, including those based on other independent embodiments, and may include any combination of any of the features, options, and possibilities described in this disclosure and the drawings. Further embodiments consistent with this teaching described herein are described in the following numbered sections.

[0071] (Item 1) Apparatus comprising a processor and a computer-readable medium containing programming instructions, wherein, when executed, the programming instructions cause the processor to perform the steps of: predicting body volume information using a machine learning model and one or more images of a subject; receiving a medical plan based on the body volume information; and executing the received medical plan.

[0072] (Item 2) The apparatus according to Item 1, wherein, when the above programming instruction is executed, the processor further performs the steps of: transmitting predicted body volume information via a communication link to a drug therapy and medication management system; and receiving medication or medical plans from a drug therapy and medication management system.

[0073] (Item 3) The apparatus described in Item 1 or 2, including the machine learning model described above, which includes a body volume model.

[0074] (Item 4) The apparatus described in any one of Items 1 to 3, wherein, when the above programming instruction is executed, the processor further performs the step of training the above machine learning model using a machine learning network on a training dataset containing multiple user images.

[0075] (Item 5) The apparatus described in any one of Items 1 to 4, wherein, when the above programming instruction is executed, the processor further performs the step of executing the received medical plan by sending the medical plan to the drug dispensing server.

[0076] (Item 6) A device comprising a processor and a computer-readable medium containing programming instructions, wherein, when executed, the programming instructions cause the processor to perform the steps of: receiving user body volume data from a user device via a communication link; adjusting a medical plan based on the received user body volume data; and transmitting the adjusted medical plan to the user device.

[0077] (Item 7) The apparatus described in Item 6, which, when the above programming instruction is executed, causes the processor to further perform the steps of receiving user data from the user device and adjusting the medical plan based on the received user body volume data.

[0078] (Item 8) The above user body volume data includes changes in body volume over a certain period of time, as described in item 6 or 7 of the apparatus.

[0079] (Item 9) The above user body volume data includes one or more of body fat, body mass, or bone mineral concentration, as described in any one of items 6 to 8.

[0080] (Item 10) The above user body volume data includes the distribution of body fat, as described in any one of items 6 to 9.

[0081] (Item 11) A method for determining the treatment of a patient, comprising the steps of: obtaining images of the patient; and providing recommendations for treating the patient based on information extracted from the images.

[0082] (Item 12) The above image shows the patient's physical appearance, as described in Item 11.

[0083] (Item 13) The treatment described above is the method described in Item 11 or 12, including the amount of medication administered to the patient or a change in the amount of medication administered.

[0084] (Item 14) The above dosage of drugs is the method described in Item 13, including cancer drugs.

[0085] (Item 15) The method described in any one of Items 11 to 14, wherein the step of acquiring the above image includes the step of taking a photograph using an image acquisition device.

[0086] (Item 16) The above patient takes a photograph, as described in Item 15.

[0087] (Item 17) Taking a photograph of the above patient in a visible light spectrum having electromagnetic wavelengths of approximately 400 nanometers to approximately 700 nanometers, as described in Item 15 or the method thereof.

[0088] (Item 18) The information extracted from the above image, including the patient's body volume information, is as described in Items 11-17.

[0089] (Item 19) The information extracted from the above image is the patient's body max index, as described in any one of Items 11-18.

[0090] (Item 20) The information extracted from the above image is the method described in any one of Items 11-19, including the patient's body volume index.

[0091] (Item 21) A system for managing patient treatment, comprising: an image acquisition device that acquires images in the visible spectrum; a body volume evaluation device that extracts body volume information from at least one image acquired by the image acquisition device; and a medical recommendation device that provides treatment recommendations based on the body volume information extracted by the body volume evaluation device.

[0092] (Item 22) The body volume evaluation device described in Item 21, comprising a processor and a computer-readable medium including programming instructions, wherein, when executed, the programming instructions cause the processor to perform the step of predicting body volume information using a machine learning model and one or more images acquired by an image acquisition device.

[0093] (Item 23) The medical recommendation device described above includes a processor and a computer-readable medium containing programming instructions, wherein, when executed, the programming instructions cause the processor to perform the steps of: receiving body volume information from a body volume evaluation device; determining a treatment recommendation based on the body volume information extracted by the body volume evaluation device; and providing a treatment recommendation to a healthcare provider, as described in Item 21 or 22.

[0094] The articles “a,” “an,” and “the” are intended to indicate that the preceding description contains one or more elements. The terms “comprising,” “including,” and “having” are intended to be comprehensive and mean that additional elements other than those listed may exist. Additionally, it should be understood that any reference in this disclosure to “one embodiment” or “embodiment” is not intended to be construed as excluding the existence of additional embodiments incorporating the described features. Any numerical values, percentages, ratios, or other values ​​described herein are intended to include their values, and any other values ​​that are described as “about” or “approximately,” as understood by those skilled in the art in the field encompassed by the embodiments of this disclosure. Accordingly, the described values ​​should be interpreted broadly enough to include values ​​that are at least sufficiently close to the described value in order to perform the desired function or achieve the desired result. The described values ​​may include at least the variation (variation) expected in a suitable manufacturing or production process, and may include values ​​within 5%, 1%, 0.1%, or 0.01% of the described value.

[0095] Those skilled in the art should understand, in light of this disclosure, that equivalent structures will not deviate from the spirit and scope of this disclosure, and that various changes, substitutions, and modifications can be made to the embodiments disclosed herein without deviating from the spirit and scope of this disclosure. Equivalent structures including functional “means-plus-function” items are intended to encompass all structures described herein as performing the described function, including both structural equivalents that operate in the same manner and equivalent structures that provide the same function. Except for claims using the phrase “means for” in conjunction with the relevant function, it is the express intention of the applicant to avoid making any means-plus-function or other functional claims. All additions, deletions, and modifications to the meaning and scope of the claims and the embodiments within them shall be incorporated into the claims.

[0096] As used herein, the terms “approximately,” “about,” and “substantially” refer to quantities close to the stated quantity that still perform the desired function or achieve the desired result. For example, the terms “approximately,” “about,” and “substantially” may mean quantities in the range of 5% or less, 1% or less, 0.1% or less, and 0.01% or less of the stated quantity. Furthermore, it should be understood that the directions or reference systems in the foregoing descriptions are merely relative directions or movements. For example, references to “up” and “down,” or “above” and “below” are merely to describe the relative position or movement of the elements in question.

[0097] As can be understood from the foregoing, while specific embodiments of this disclosure are described herein for illustrative purposes, various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, the scope of this disclosure should not be limited to any of the specific embodiments described herein.

Claims

1. Processor and A computer-readable medium containing programming instructions, wherein, when the programming instructions are executed, the processor, A step of predicting body volume information using a machine learning model and one or more images of a subject, The steps include adjusting the predicted body volume information based on at least one of user health data, medical data, or analysis of one or more images of the subject, The steps include receiving a medical plan based on the adjusted body volume information, A device for performing the steps of carrying out the medical plan received.

2. When the aforementioned programming instruction is executed, the processor will: The steps include transmitting the predicted body volume information to a drug treatment management system via a communication link, The apparatus according to claim 1, further comprising the step of receiving the medical plan from the drug treatment and medication management system.

3. The apparatus according to claim 1, wherein the machine learning model includes a body volume model.

4. The apparatus according to claim 3, wherein, when the programming instruction is executed, it causes the processor to further perform the step of training the machine learning model using a machine learning network on at least a training dataset comprising a plurality of user images.

5. The apparatus according to claim 1, wherein, when the programming instruction is executed, the processor further performs the step of executing the received medical plan by transmitting the medical plan to the drug dispensing server.

6. Processor and A computer-readable medium containing programming instructions, wherein, when the programming instructions are executed, the processor, The steps include receiving user body volume data from a user device via a communication link, The steps include adjusting the user body volume data based on at least one of user health data, medical data, or analysis of one or more images of the subject of the user device, The steps include adjusting the medical plan based on the adjusted user body volume data, A device that performs the steps of transmitting adjusted medical information to the user device.

7. When the aforementioned programming instruction is executed, the processor will: The steps include receiving user data from the user device, The apparatus according to claim 6, further comprising the step of adjusting the medical plan based on the received user body volume data.

8. The apparatus according to claim 6, wherein the user body volume data includes changes in body volume over a certain period of time.

9. The apparatus according to claim 6, wherein the user body volume data includes one or more of body fat, body mass, or bone mineral concentration.

10. The apparatus according to claim 6, wherein the user body volume data includes the distribution of body fat.

11. A method for determining a patient's treatment, The steps include: obtaining an image of the patient, A step of predicting body volume information using a machine learning model and the aforementioned image, The steps include adjusting the predicted body volume information based on at least one of user health data, medical data, or analysis of the image, A method comprising the step of providing recommendations for treating the patient based on the adjusted body volume information.

12. The method according to claim 11, wherein the image includes the physical appearance of the patient.

13. The method according to claim 11, wherein the treatment includes the amount of medication administered to the patient or a change in the amount of medication administered.

14. The method according to claim 13, wherein the drug in the prescribed dosage includes an anticancer drug.

15. The method according to claim 11, wherein the step of acquiring the image includes the step of taking a photograph using an image acquisition device.

16. The method according to claim 15, wherein the patient takes the photograph.

17. The method according to claim 15, wherein a photograph of the patient is taken in a visible light spectrum having an electromagnetic wavelength of approximately 400 nanometers to approximately 700 nanometers.

18. The method according to claim 11, wherein the information extracted from the image includes the patient's body volume information.

19. The method according to claim 11, wherein the information extracted from the image includes the patient's body mass index.

20. The method according to claim 11, wherein the information extracted from the image includes the patient's body volume index.

21. An image acquisition device that captures images in the visible spectrum, A body volume evaluation device that extracts body volume information from at least one image captured by the image acquisition device, An adjustment device that adjusts the extracted body volume information based on at least one of user health data, medical data, or analysis of the at least one of the images, A system for managing patient treatment, including a medical recommendation device that provides treatment recommendations based on the adjusted body volume information.

22. The body volume evaluation device, Processor and A computer-readable medium containing programming instructions, wherein, when the programming instructions are executed, the processor, The system according to claim 21, comprising the step of predicting the body volume information using a machine learning model and one or more images acquired by the image acquisition device.

23. The aforementioned medically recommended device is Processor and A computer-readable medium containing programming instructions, wherein, when the programming instructions are executed, the processor, The steps include receiving body volume information from the body volume evaluation device, The steps include determining the treatment recommendation based on the body volume information extracted by the body volume evaluation device, The system according to claim 21, comprising the step of providing the treatment recommendation to a treatment provider and causing the provider to perform the latter step.

Images