Body fluid volume estimation device, body fluid volume estimation method, and program

Abstract

To estimate the body fluid amount of a patient from the face image of the patient.SOLUTION: A body fluid amount device 100 has a prior learning unit 1, a transfer learning unit 2, and an estimation unit 3. The prior learning unit 1 is pre-trained using information as teaching information that indicates a body fluid amount at the time a plurality of patient face images were captured. The transfer learning unit 2 is further transfer trained using the plurality of face images of one specific patient after pre-training so as to construct a trained model. The estimation unit 3 inputs the face images of one specific patient to the trained model and estimates a body fluid amount at the time the face images of one specific patient were captured.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a body fluid volume estimation device, a body fluid volume estimation method, and a program.

Background Art

[0002] In recent years, the development of methods for determining a person's health condition and the presence or absence of diseases from a person's appearance, such as a face image, has progressed (Patent Documents 1 to 3). Generally, it is known that morphological changes occur in the face, lower limbs, etc. according to a person's health condition. An increase in body volume is detected as edema, and a decrease in volume is detected as a decrease in skin tension. Edema mainly refers to a state in which excessive water is stored in the tissue spaces and the body fluid volume increases, and it occurs due to various causes such as central nervous system diseases, respiratory and circulatory diseases, kidney diseases, orthopedic diseases, metabolic diseases, and malignant tumors. In addition, a decrease in skin tension occurs due to a decrease in body water and occurs in various conditions such as dehydration and heat stroke.

[0003] For example, when waste products and water in the body cannot be removed due to reduced kidney function, edema occurs, and currently, excess water in the body is removed by dialysis treatment. Therefore, it is important for dialysis patients to keep the body water (i.e., body fluid volume) within a desirable range, and for this purpose, it is necessary to restrict the intake of water and salt in daily life.

[0004]

[0005]

[0006]

[0007]

[0008]

[0009]

[0010]

[0011]

[0012]

[0013]

[0014]

[0015] To understand the condition of patients with diseases that cause edema, it is necessary to measure the degree of edema, i.e., the volume of body fluid. As a common edema estimation technique, for example, a method has been proposed in which the degree of edema is measured by a pitting test performed by medical staff (Non-Patent Literature 1). In this method, images are taken during a pitting test of the lower limbs, and the degree of peripheral edema, such as in the lower limbs, is estimated from these images using a Support Vector Machine (SVM) or a Convolutional Neural Network (CNN).

[0006] Furthermore, a method has been proposed (Non-Patent Document 2) for measuring the degree of edema based on images taken of peripheral areas such as the hands and feet of a patient using a Short-Wave Infrared (SWIR) camera. This method utilizes the property that the absorption coefficients of water, collagen, and lipids become larger in specific spectral regions of the SWIR camera when edema is present, allowing the edema level to be estimated from the spectral components. [Prior art documents] [Patent Documents]

[0007] [Patent Document 1] Special Publication No. 2022-512044 [Patent Document 2] Japanese Patent Publication No. 2020-199072 [Patent Document 3] Japanese Patent Publication No. 2005-65812 [Non-patent literature]

[0008] [Non-Patent Document 1] J. Chen et.al, “Camera-Based Peripheral Edema Measurement Using Machine Learning,” in Proc. IEEE Int. Conf. Healthcare Informatics (ICHI), 2018, pp. 115-122. [Non-Patent Document 2] A. G. Smith et.al, “Objective determination of peripheral edema in heart failure patients using short-wave infrared molecular chemical imaging,” Journal of Biomedical Optics, vol. 26, no. 10, pp. 105002, 2021. [Non-Patent Document 3] Kaiming He et al., “Momentum Contrast for Unsupervised Visual Representation Learning”, in Proc. IEEE / CVF conf. computer vision and pattern recognition (CVPR), 2020, pp. 9729-9738. [Non-Patent Document 4] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton,”A simple framework for contrastive learning of visual representations,” Proc. Int. conf. machine learning (ICML), 2020, pp. 1597-1607. [Non-Patent Document 5] P. Khosla et al., “Supervised Contrastive Learning”, in Proc. Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 18661-18673, 2020. [Non-Patent Document 6] B. Dufumier et al., “Contrastive Learning with Continuous Proxy Meta-Data for 3D MRI Classification”, in Int. Conf. Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2021, pp. 58-68. [Non-Patent Document 7] DP Kingma and J. Ba, “ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION,” arXiv preprint arXiv:1412.6980, 2014. [Non-Patent Document 8] J. Deng, W. Dong, R. Socher, L. Li, et al., “Imagenet: A Large-Scale Hierarchical Image Database,” Proc. IEEE / CVF conf. computer vision and pattern recognition (CVPR), 2009, pp. 248-255. [Overview of the Initiative] [Problems that the invention aims to solve]

[0009] As mentioned above, in order to routinely manage the degree of edema in patients, i.e., their body fluid volume, it is desirable for patients themselves to measure their body fluid volume and manage their fluid intake according to the measurement results. However, the methods for measuring body fluid volume described above must be performed by specialized medical staff and have limitations such as requiring specific equipment such as special cameras.

[0010] On the other hand, for patients to autonomously restrict their fluid and salt intake in their daily lives, it is necessary to establish a method that allows patients to measure their body fluid volume on a daily basis.

[0011] This disclosure is made in view of the above circumstances and aims to estimate the volume of a patient's body fluids from a patient's facial image. [Means for solving the problem]

[0012] A fluid volume estimation device according to one aspect of this disclosure includes: a pre-learning unit that pre-learns information indicating the fluid volume when facial images of multiple patients are captured as training information; a transfer learning unit that, after the pre-learning, further transfer-learns multiple facial images of a specific patient to construct a trained model; and an estimation unit that inputs the facial image of the specific patient into the trained model to estimate the fluid volume of the specific patient at the time the facial image of the specific patient was captured.

[0013] One aspect of the present disclosure is a method for estimating body fluid volume, which involves pre-training information indicating the body fluid volume when facial images of multiple patients are captured as training information, constructing a trained model by further transfer learning using multiple facial images of a specific patient after the pre-training, and inputting the facial images of the specific patient into the trained model to estimate the body fluid volume of that specific patient at the time the facial images of that patient were captured.

[0014] A program according to an aspect of the present disclosure causes a computer to execute a process of pre-training, using information indicating the amount of body fluid when capturing face images of a plurality of patients as teacher information, a process of further performing transfer learning on a plurality of face images of a specific patient after the pre-training to construct a learned model, and a process of inputting the face image of the specific patient into the learned model to estimate the amount of body fluid of the specific patient at the time when the face image of the specific patient is captured.

Advantages of the Invention

[0015] According to the present disclosure, the amount of body fluid of a patient can be estimated from a face image of the patient.

Brief Description of the Drawings

[0016] [Figure 1] It is a diagram schematically showing the configuration of a body fluid amount estimation device according to Embodiment 1. [Figure 2] It is a diagram schematically showing a modification example of the body fluid amount estimation device according to Embodiment 1. [Figure 3] It is a diagram schematically showing a modification example of the body fluid amount estimation device according to Embodiment 1. [Figure 4] It is a diagram showing the sequence of processing of the body fluid amount estimation device according to Embodiment 1. [Figure 5] It is a diagram showing an overview of pre-training by WeightSupMoCo (Weight-Aware Supervised Momentum Contrast) according to Embodiment 1. [Figure 6] It is a diagram showing an overview of the feature amount distribution of face images before and after dialysis in the feature amount space after pre-training. [Figure 7] It is a diagram showing an overview of transfer learning according to Embodiment 1. [Figure 8] It is a diagram showing the classification before and after dialysis and the estimation accuracy of body weight in each method. [Figure 9] It is a diagram showing the transition between the estimated body weight and the correct answer of the patient with the highest estimation accuracy. [Figure 10]This figure shows the trend between the estimated weight of the patient with the lowest estimation accuracy and the correct weight. [Figure 11] This figure shows the change in Accuracy, which is the estimation accuracy of pre- and post-dialysis classification, as a function of the number of days of transfer learning data. [Figure 12] This figure shows the change in MAE (Mean absolute error), which is the estimation accuracy of the classification of estimated body weight before and after dialysis, as a function of the number of days of transfer learning data. [Figure 13] This is a diagram showing an example of a computer configuration. [Modes for carrying out the invention]

[0017] Embodiments of the present invention will be described below with reference to the drawings. In each drawing, the same elements are denoted by the same reference numerals, and redundant explanations are omitted where necessary.

[0018] Embodiment 1 The body fluid volume estimation device 100 according to Embodiment 1 will now be described. The body fluid volume estimation device 100 is configured to perform comparative learning of a patient's facial image using a new method called Weight-Aware Supervised Momentum Contrast (WeightSupMoCo) based on information representing the patient's body fluid volume. The body fluid volume estimation device 100 can then input the facial image of the patient to be estimated into a trained model to estimate the patient's body fluid volume, for example, the change in body fluid volume.

[0019] Figure 1 schematically shows the configuration of the body fluid volume estimation device 100 according to Embodiment 1. The body fluid volume estimation device 100 includes a pre-learning unit 1, a transfer learning unit 2, and an estimation unit 3.

[0020] The pre-training unit 1 receives the pre-training data Dp (described later) and performs pre-training to machine-learn facial images using body fluid volume labels, which indicate the volume of a patient's body fluids, as training labels.

[0021] After pre-training is complete, the transfer learning unit 2 reads the transfer learning data Dt, which will be described later, consisting of information on a specific patient whose body fluid volume is to be estimated, and performs transfer learning using the facial image. As a result, the transfer learning unit 2 can construct a trained model M that estimates body fluid volume based on the facial image of a specific patient that is input separately.

[0022] The pre-training data Dp and transfer learning data Dt may be provided to the body fluid volume estimation device 100 from outside the device via various communication means such as a network.

[0023] Furthermore, the pre-training data Dp and transfer learning data Dt may be stored in advance in a memory unit provided in the body fluid volume estimation device 100. Figure 2 schematically shows a modified example of the body fluid volume estimation device 100 according to Embodiment 1. As shown in Figure 2, the body fluid volume estimation device 100 may further have a memory unit 4. The memory unit 4 has the pre-training data Dp and transfer learning data Dt stored in advance. The pre-training unit 1 can read the pre-training data Dp from the memory unit 4 as appropriate. The transfer learning unit 2 can read the transfer learning data Dt from the memory unit 4 as appropriate. The information stored in the memory unit 4 may be provided to the memory unit 4 from outside the memory unit 4 via various communication means such as a network.

[0024] The estimation unit 3 inputs a facial image IN of a specific patient, captured at any given time, into a constructed, trained model M to estimate the patient's body fluid volume at the time the facial image IN was captured.

[0025] The facial image input may be provided to the fluid volume estimation device 100 from outside the device via various communication means, such as a network. In this case, by installing the fluid volume estimation device 100, for example, in a medical institution, the facial image captured by the patient can be transmitted to the medical institution, and the fluid volume of the target patient can be estimated at the medical institution.

[0026] Furthermore, the face image IN may be an image captured by an imaging unit provided in the body fluid volume estimation device 100. Figure 3 schematically shows a modified example of the body fluid volume estimation device 100 according to Embodiment 1. As shown in Figure 3, the body fluid volume estimation device 100 may further have an imaging unit 5. The imaging unit 5 may be various imaging devices capable of acquiring images, and can appropriately capture a face image of a specific patient at any time. The captured face image IN is input to the estimation unit 3 as appropriate. The captured face image IN may be input to the estimation unit 3 via other processing units (not shown), or it may be stored in the storage unit 4 and then read by the estimation unit 3.

[0027] As shown in Figure 3, by integrating the pre-learning unit 1, transfer learning unit 2, estimation unit 3, memory unit 4, and imaging unit 5 into a single body fluid volume estimation device 100, the body fluid volume estimation device 100 can be mounted on various mobile terminals equipped with imaging devices such as cameras, such as smartphones.

[0028] In this embodiment, the patient's body fluid volume is estimated by estimating whether or not edema is present in the patient's face at the time of facial image acquisition, and the patient's weight. Furthermore, in this embodiment, targeting dialysis patients, in order to clearly determine whether or not edema is present in the patient's face, it is estimated whether the facial image was taken before dialysis when edema is present, or after dialysis when edema is not present.

[0029] This allows the system to detect whether a patient's body fluid volume has significantly changed from a normal state by estimating whether edema is present in the input facial image, assuming that the state without edema, i.e., the post-dialysis state, is the normal state. If the estimation unit 3 estimates that edema is present in the input facial image of the patient being estimated, it may output, for example, that the body fluid volume has increased compared to the normal state as a detection result. The estimation unit 3 may also estimate the change in body fluid volume from the estimated weight and output the estimation result. Such estimation and detection results may be displayed on a display device (not shown, for example, the output unit 1007 in Figure 13), or, if the body fluid volume estimation device 100 is installed on a smartphone, they may be displayed on the screen as appropriate.

[0030] Next, we will explain the pre-training data Dp. The pre-training data Dp consists of multiple records, each containing a patient's face image, label information indicating whether the patient was undergoing or not (also simply called a dialysis label) when the face image was taken, and label information indicating the patient's weight at the time the face image was taken (also simply called a weight label). In other words, one record contains the face image, dialysis label, and weight label of one patient.

[0031] For example, in the case of dialysis patients, the amount of body fluid removed from the patient's body by dialysis may be divided by the corresponding weight label value of the patient's face image before dialysis, and this value may be used as the weight label corresponding to the face image after dialysis. In this case, the weight label corresponding to the face image of the patient after dialysis can more accurately reflect the change in body fluid volume compared to measuring the patient's weight after dialysis.

[0032] Alternatively, for example, the amount of fluid removed from the patient's body by dialysis may be added to the weight label corresponding to the patient's face image after dialysis, and this value may be used as the weight label corresponding to the face image before dialysis. In this case, the change in fluid volume can be reflected more accurately in the weight label corresponding to the patient's face image before dialysis compared to measuring the patient's weight after dialysis. Alternatively, a standard weight may be predetermined for each patient after dialysis, i.e., for patients without edema, and the value obtained by adding the amount of fluid removed from the patient's body by dialysis to this standard weight may be used as the weight label corresponding to the patient's face image before dialysis.

[0033] Next, the transfer learning data Dt will be described. The transfer learning data Dt consists of multiple records containing a facial image, dialysis label information, and weight label of a specific patient who is the target of pre- and post-dialysis classification and weight estimation by the body fluid volume estimation device 100. The structure of each record is the same as that of the pre-training data Dp.

[0034] In this embodiment, the number of records included in the pre-training data Dp is greater than the number of records included in the transfer training data Dt.

[0035] The dialysis label is a discrete label that indicates whether the patient was pre-dialysis or post-dialysis when the facial image was captured. For example, it may be labeled "1" if the corresponding facial image was taken before dialysis and shows edema, and "0" if the corresponding facial image was taken after dialysis and does not show edema.

[0036] In contrast, the weight label for dialysis patients is given as a numerical value indicating the patient's weight at the time the corresponding facial image was captured, i.e., as a continuous label.

[0037] The face images included in each record may be obtained by, for example, the pre-training unit 1 reading original images containing the faces of dialysis patients that have been captured in advance and performing appropriate image processing. For example, the pre-training unit 1 performs face detection on the original image and then extracts the central part of the face. Then, if necessary, the pre-training unit 1 may perform data augmentation on the extracted face images, such as resizing, flipping horizontally, color conversion, or grayscale conversion, to obtain the face images for each record. Such data augmentation may be performed by the pre-training unit 1 or by an image processing unit provided separately from the pre-training unit 1.

[0038] Next, the flow and processing details of the controlled learning process of the body fluid volume estimation device 100 will be explained. As described above, the body fluid volume estimation device 100 performs pre-training using WeightSupMoCo, which takes pre-training data Dp from an unspecified number of patients as input data, and transfer learning, which takes transfer learning data Dt from a specific patient to be estimated as input data. Figure 4 shows the processing sequence of the body fluid volume estimation device 100 according to Embodiment 1.

[0039] Step S1: Pre-learning In pre-training, the body fluid volume estimation device 100 performs pre-training using WeightSupMoCo with pre-training data Dp from an unspecified number of patients as input data.

[0040] In the pre-training phase, comparative learning based on Momentum Contrast (MoCo, Non-Patent Literature 3) is performed. However, while general comparative learning using MoCo involves self-supervised learning without using teacher labels, in this embodiment, as described above, comparative learning using WeightSupMoCo is performed using dialysis labels and weight labels as teacher labels.

[0041] Figure 5 shows an overview of pre-training using WeightSupMoCo according to Embodiment 1. WeightSupMoCo uses an encoder (feature extractor) and a momentum encoder, and a fully connected layer called a projection head is provided after each. The features output from the projection head after the encoder are called queries, and the features output from the projection head after the momentum encoder are called keys. The keys are stored in a dictionary as a queue, and comparative learning is performed using the queries and the queue.

[0042] On the other hand, contrast learning methods using conventional encoders instead of momentum encoders, such as Simple Framework for Contrastive Learning of Visual Representations (SimCLR, Non-Patent Literature 4), are also known. These methods have the disadvantage that the number of samples used for contrast learning becomes small because the dictionary size is equivalent to the mini-batch size.

[0043] Furthermore, as a control learning method using discrete-type training labels such as dialysis labels, for example, a method using Supervised Contrastive (SupCon) Loss (Non-Patent Literature 5) as the loss function is known. As a control learning method using continuous-type training labels such as weight labels, for example, a method using y-Aware InfoNCE Loss (Non-Patent Literature 6) as the loss function is known. However, methods using SupCon Loss and y-Aware InfoNCE Loss are SimCLR-type methods, and as mentioned above, they have the disadvantage of requiring a small number of samples for control learning.

[0044] In contrast, in this embodiment, when performing comparative learning using discrete dialysis labels and continuous weight labels as training labels, the MoCo-type WeightSupMoCo Loss is used as the loss function. Therefore, unlike SimCLR-type methods, it is possible to perform comparative learning with a larger dictionary size, and it is possible to obtain better feature representations.

[0045] Next, WeightSupMoCo according to this embodiment will be described in more detail. In WeightSupMoCo, when the labels before and after dialysis are the same, the system learns to make the feature representation of the face image closer to the similarity of the weight labels. This will be explained in detail below. This is because, between face images before dialysis, or between face images after dialysis, the presence or absence of edema is the same, and it can be assumed that the degree of edema is similar the closer the weight is.

[0046] The following provides a detailed explanation. When the weight label is y, the dialysis label is a, and the feature output from the projection head is z, the WeightSupMoCo Loss is expressed by the following formula.

number

number

[0047] Figure 6 shows an overview of the feature distribution of pre- and post-dialysis facial images in the feature space after pre-training. As shown in Figure 6, as a result of pre-training, facial images with the same dialysis label, "pre-dialysis," are distributed at close distances in the feature space. In contrast, facial images with the same dialysis label, "post-dialysis," are distributed at relatively distant locations from facial images with the same dialysis label, "pre-dialysis."

[0048] Furthermore, for facial images with the same dialysis label, in the feature space, they will be distributed according to the similarity of their weight labels; that is, if the weight labels are similar, they will be distributed at close distances, and if the weight labels are not similar, they will be distributed at far distances.

[0049] Step S2: Transfer Learning Next, the transfer learning unit 2 takes the transfer learning data Dt of a specific patient, which is the target of estimation, as input data and performs transfer learning using the pre-training result PL. In this embodiment, in transfer learning, only the encoder is used, without using the projection head used in pre-training. Figure 7 shows an overview of the transfer learning according to Embodiment 1. As a result, a single linear layer is added after the encoder, consisting of a classification layer with a dimensionality of 2 for the feature quantities output in the estimation of pre- and post-dialysis classification using the transfer learning data Dt, which is the data of a specific patient, and a regression layer with a dimensionality of 1 for the feature quantities output in the estimation of body weight.

[0050] Then, using data from a specific patient, fine-tuning is performed by transfer learning of the encoder and all the added linear layers. Here, as the loss function for fine-tuning, cross-entropy loss is used for pre- and post-dialysis classification, and mean squared error loss is used for weight estimation.

[0051] Step S3: Estimate By performing the above steps, a trained model M is constructed. By inputting a facial image IN of a specific patient to be estimated into this trained model M, the estimation unit 3 can estimate whether the patient extracted from the facial image is before or after dialysis (presence or absence of edema) and the patient's weight, and output the estimation result OUT.

[0052] By following the above procedure, a trained model can be constructed through pre-training using data from an unspecified number of patients and transfer learning using data from a specific patient. Then, by inputting the facial image of that specific patient into the trained model, it is possible to estimate whether the patient is pre- or post-dialysis (presence or absence of edema) and their weight.

[0053] Next, in order to verify the significance of the method in this embodiment, a comparative experiment with a general method was conducted. The experimental conditions and results are described below.

[0054] For pre-training, 80% of the data obtained from multiple patients was randomly selected and used as pre-training data (Dp), while the remaining 20% ​​was used as validation data.

[0055] In transfer learning and estimation, multiple sets of facial images of a specific patient before and after dialysis were obtained for several different dialysis occasions. One set was used as test data, 80% of the remaining sets were randomly selected as transfer learning data (Dt), and the remaining 20% ​​were used as validation data. Although the occasions in which a patient receives dialysis are referred to as dialysis occasions, since dialysis is usually performed within one day (generally about 4 hours), dialysis occasions will henceforth be simply referred to as dialysis days. Furthermore, in this experiment, the specific patient used for transfer learning and estimation is not included among the multiple patients used for pre-training.

[0056] The facial images used in this experiment were resized to 224x224 pixels, and data augmentation was performed by horizontal flipping, color conversion, and grayscale conversion. In this experiment, training was performed for 100 epochs in pre-training and 20 epochs in transfer training. However, the model with the smallest validation data error in the pre-training epoch was used for the subsequent transfer training. The optimization algorithm used was Adam (Non-Patent Literature 7), and the learning rate for pre-training was set to 10 -4 , the learning rate of transfer learning is set to 10 -3 In the transfer learning process, weight estimation was performed by normalizing the weight to a mean of 0 and a variance of 1.

[0057] In this experiment, the following methods were used as comparative examples for comparison with the method according to this embodiment.

[0058] Comparative Example 1 To verify the effectiveness of pre-training, a method that trained using only the data of a specific patient without pre-training (no pre-training) was designated as Comparative Example 1.

[0059] Comparative Example 2 To verify the effectiveness of using teacher labels, we used SimCLR, a self-supervised learning method, as Comparative Example 2.

[0060] Comparative Example 3 To verify the effectiveness of using teacher labels, MoCo, a self-supervised learning method, was used as Comparative Example 3.

[0061] Comparative Example 4 Similarly, as a pre-training method using general, conventional supervised learning, a classification method using dialysis labels based on cross-entropy loss was presented as Comparative Example 4.

[0062] Comparative Example 5 To verify the effectiveness of pre-training based on the method (WeightSupMoCo) according to this embodiment, Comparative Example 5 was set to a case where a pre-training method based on SupCon using dialysis labels as discrete training information was used.

[0063] Comparative Example 6 Similarly, to verify the effectiveness of pre-training based on the method according to this embodiment (WeightSupMoCo), Comparative Example 6 was presented, which uses a pre-training method based on y-Aware InfoNCE that uses weight labels as continuous training information.

[0064] For the classification before and after dialysis, the accuracy of the estimated labels, ROC-AUC (Area under the receiver operating characteristic curve), and PR-AUC (Area under the precision-recall curve) were used as evaluation indicators. In addition, for weight estimation, the MAE (Mean absolute error), RMSE (Root mean squared error), and correlation coefficient (CorrCoef) between the estimated weight and the ground truth data were used as evaluation indicators.

[0065] Experimental results Figure 8 shows the accuracy of pre- and post-dialysis classification and weight estimation for each method. From Figure 8, it can be confirmed that WeightSupMoCo according to this embodiment can estimate pre- and post-dialysis classification and weight with higher accuracy than any of the comparison methods.

[0066] First, a comparison between the method without pre-training ("No Pre-training" in Figure 8) and other methods clearly demonstrates the significant effectiveness of pre-training. Furthermore, a comparison of the self-supervised learning methods SimCLR and MoCo with WeightSupMoCo in this embodiment confirms the effectiveness of using dialysis labels and dialysis patient weight as training data for body fluid volume. Finally, a comparison of WeightSupMoCo with SupCon and y-Aware InfoNCE confirms the effectiveness of collaboratively using dialysis labels and weight labels, as well as the effectiveness of increasing dictionary size using the MoCo type.

[0067] Next, we examine the changes between estimated and actual body weight in relation to dialysis sessions. Figure 9 shows the changes between estimated and actual body weight for the patient with the highest estimation accuracy. From Figure 9, it was confirmed that, for some patients, it is possible to estimate the changes in body weight with high accuracy. Figure 10 shows the changes between estimated and actual body weight for the patient with the lowest estimation accuracy. For some patients, there are dialysis sessions in which the estimation accuracy is low, but it was confirmed that the increase or decrease in body weight before and after dialysis can be estimated favorably. Therefore, it can be understood that it is possible to estimate body weight, which represents the degree of edema, from facial images of dialysis patients.

[0068] Next, we examine the estimation accuracy of pre- and post-dialysis classification and weight estimation in relation to the number of days of training data used for transfer learning. Figure 11 shows the change in Accuracy, which is the estimation accuracy of pre- and post-dialysis classification, in relation to the number of days of transfer learning data. Figure 12 shows the change in MAE, which is the estimation accuracy of pre- and post-dialysis weight classification, in relation to the number of days of transfer learning data. Figures 11 and 12 compare a method without pre-training (no pre-training), pre-training based on WeightSupMoCo, and a method in which an ImageNet pre-trained model was pre-trained based on WeightSupMoCo (ImageNet → WeightSupMoCo).

[0069] In all methods, estimation accuracy improved as the number of days of transfer learning data increased. Furthermore, comparing methods without pre-training with other methods, the effectiveness of pre-training was clearly demonstrated.

[0070] Furthermore, ImageNet → WeightSupMoCo yielded the highest accuracy, confirming that pre-training an ImageNet pre-trained model based on WeightSupMoCo is effective.

[0071] Since it is desirable to minimize the number of days of training data for a specific patient that constitutes the transfer learning data, pre-training (ImageNet → WeightSupMoCo) allows us to use transfer learning data from a relatively short period of 3 days, enabling relatively high-accuracy estimations such as an accuracy of 76.8% for pre- and post-dialysis classification and a MAE of 0.65 kg for weight estimation.

[0072] As explained above, the fluid volume estimation device 100 not only determines the presence or absence of edema from the patient's facial image, but also estimates the weight of the dialysis patient, thereby estimating the amount of fluid in the dialysis patient's body, or in other words, the degree of edema. Furthermore, since the device is trained using dialysis labels and weight labels, and the estimation is performed using the training results, it is possible to reduce the variability in edema levels and other factors that depend on the physician's subjective judgment, as in previous diagnoses, and obtain more objective diagnostic results.

[0073] Furthermore, while the amount of data obtainable from a single specific patient is limited, such as facial images, the presence or absence of edema, and weight, this configuration allows for pre-training using data obtained from a large number of unspecified patients, enabling the use of a sufficient amount of training data to achieve high estimation accuracy.

[0074] As shown in Figure 3, by incorporating facial image acquisition means and other components into the fluid volume estimation device 100, patients can acquire a facial image of themselves at any location (e.g., at home) and at any time, and perform estimation using the fluid volume estimation device 100 to determine whether or not they have edema and to what extent, i.e., their fluid volume. This allows dialysis patients to easily and appropriately check their own fluid volume in a timely manner. Furthermore, this configuration is advantageous compared to general examination methods in that it does not require the involvement of medical professionals or special equipment.

[0075] This configuration allows for the determination of whether a patient's condition is closer to a state with or without edema by utilizing a predictive score representing the presence or absence of edema, derived from the estimation results. Furthermore, by utilizing the estimated weight, patients can assess the degree of edema themselves and estimate their body water content. In addition, based on the estimated edema score and weight, patients can adjust their food and fluid intake, choose their meal menus, and adjust medication dosages (such as diuretics).

[0076] Even in cases where patients cannot go to hospitals or other medical facilities (for example, in remote areas), the body fluid volume estimation device 100 can transmit information indicating the estimated body fluid volume to doctors at medical facilities. This allows doctors to accurately understand the patient's body fluid volume and provide accurate diagnoses and lifestyle guidance.

[0077] Other embodiments It should be noted that the present invention is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention. For example, in the embodiments described above, dialysis labels and weight labels were used as training labels, but other discrete and continuous labels may be used in combination as appropriate. Also, although the invention has been described using one type of discrete label and one type of continuous label as training labels, the number of discrete labels and continuous labels may be any number.

[0078] While the above embodiment focused on dialysis patients, it goes without saying that the fluid volume estimation device 100 can also be applied to estimate the occurrence of edema in patients with other diseases that cause edema.

[0079] In the above-described embodiment, the presence or absence of edema was explained as an estimation method, but this is merely an example. It is also possible to estimate changes in facial images other than edema due to disease, such as changes in facial size and shape or changes in complexion due to heatstroke or other conditions.

[0080] In the embodiments described above, the configuration of the body fluid volume estimation device 100 was described as a hardware configuration, but the present invention is not limited thereto. Processing in the body fluid volume estimation device 100 can also be realized by having a CPU (Central Processing Unit) execute a computer program. Furthermore, the above-described program can be stored using various types of non-transitory computer-readable media and supplied to a computer. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, RAMs (Random Access Memory)). Furthermore, the program may be supplied to the computer using various types of transient computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. Temporary computer-readable media can supply programs to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0081] Let's describe an example of a computer. A computer can be implemented using various types of computers, such as dedicated computers and personal computers (PCs). However, a computer does not need to be physically a single unit; multiple computers are acceptable when performing distributed processing.

[0082] Figure 13 shows an example of a computer configuration. Computer 1000 in Figure 13 has a CPU (Central Processing Unit) 1001, ROM (Read Only Memory) 1002, and RAM (Random Access Memory) 1003, which are interconnected via a bus 1004. While the explanation of the operating system software necessary to run the computer is omitted, it is assumed that this computer also possesses such software.

[0083] An input / output interface 1005 is also connected to the bus 1004. The input / output interface 1005 is connected to, for example, an input unit 1006 consisting of a keyboard, mouse, sensor, etc., a display consisting of a CRT, LCD, etc., an output unit 1007 consisting of headphones, speakers, etc., a storage unit 1008 consisting of a hard disk, etc., and a communication unit 1009 consisting of a modem, terminal adapter, etc.

[0084] The CPU 1001 performs various processes according to the various programs stored in the ROM 1002 or the various programs loaded from the storage unit 1008 into the RAM 1003. In the above embodiment, for example, it performs the processing of each part of the body fluid volume estimation device 100. Alternatively, a GPU (Graphics Processing Unit) may be provided to perform various processes according to the various programs stored in the ROM 1002 or the various programs loaded from the storage unit 1008 into the RAM 1003, similar to the CPU 1001. In this embodiment, for example, it performs the processing of each part of the body fluid volume estimation device 100. The GPU is suitable for performing routine processing in parallel, and by applying it to processing in neural networks, which will be described later, it is possible to improve the processing speed compared to the CPU 1001. The RAM 1003 also appropriately stores data necessary for the CPU 1001 and GPU to perform various processes.

[0085] The communication unit 1009 performs communication processing, for example, via the Internet (not shown), transmits data provided by the CPU 1001, and outputs data received from the communication partner to the CPU 1001, RAM 1003, and storage unit 1008. The storage unit 1008 exchanges information with the CPU 1001 and saves and erases information. The communication unit 1009 also performs communication processing of analog or digital signals with other devices.

[0086] The input / output interface 1005 can also be connected to a drive 1010 as needed, and a magnetic disk 1011, optical disk 1012, flexible disk 1013, or semiconductor memory 1014 may be installed as appropriate, and computer programs read from them may be installed in the storage unit 1008 as needed.

[0087] The present invention has been described above, but it can also be described as follows.

[0088] (Note 1) A body fluid volume estimation device comprising: a pre-learning unit that pre-trains multiple facial images of patients using information indicating the body fluid volume at the time the facial images of the multiple patients were captured as training information; a transfer learning unit that, after the pre-learning, further transfer-learns multiple facial images of a specific patient to construct a trained model; and an estimation unit that inputs the facial image of the specific patient into the trained model to estimate the body fluid volume of the specific patient at the time the facial image of the specific patient was captured.

[0089] (Note 2) The fluid volume estimation device according to Note 1, wherein the information indicating the volume of body fluid when facial images of multiple patients are captured includes information indicating the presence or absence of edema in the facial images of the multiple patients and information indicating the weight of the multiple patients, and the estimation unit inputs the facial image of a specific patient into the trained model to estimate the presence or absence of edema and the weight of the specific patient at the time the facial image of the specific patient was captured.

[0090] (Note 3) The fluid volume estimation device according to Appendix 2, wherein the estimation unit detects whether the fluid volume of a specific patient has changed based on the estimation result of whether or not the patient has edema, relative to a preset standard fluid volume for a specific patient, and obtains the difference in the fluid volume of the specific patient relative to the standard fluid volume from the weight estimation result.

[0091] (Note 4) The information indicating the presence or absence of edema in the facial images of the multiple patients is label information indicating the presence or absence of edema, and the pre-training unit pre-trains the feature quantities of facial images with the same label information indicating the presence or absence of edema to be as similar as the information indicating weight. This is the body fluid volume estimation device according to Note 2 or 3.

[0092] (Note 5) The aforementioned pre-training unit performs pre-training using WeightSupMoCo (Weight-Aware Supervised Momentum Contrast), and is a body fluid volume estimation device as described in any one of Appendix 2 to 4.

[0093] (Note 6) The fluid volume estimation device according to any one of the appendices 2 to 5, wherein the plurality of patients and the specific one patient are patients undergoing dialysis, and the post-dialysis weight of each of the plurality of patients and the specific one patient corresponding to the absence of edema is the value obtained by dividing the pre-dialysis weight of each of the plurality of patients and the specific one patient corresponding to the presence of edema by the amount of body fluid removed by dialysis.

[0094] (Note 7) The fluid volume estimation device according to any one of Notes 2 to 5, wherein the plurality of patients and the specific one patient are patients undergoing dialysis, and the pre-dialysis weight of each of the plurality of patients and the specific one patient corresponding to the presence of edema is the value obtained by adding the volume of body fluid removed by dialysis to the post-dialysis weight of each of the plurality of patients and the specific one patient corresponding to the absence of edema.

[0095] (Note 8) The fluid volume estimation device according to any one of the appendices 2 to 5, wherein the plurality of patients and the specific one patient are patients undergoing dialysis, and the pre-dialysis weight corresponding to the presence of edema in each of the plurality of patients and the specific one patient is a value obtained by adding the volume of body fluid removed by dialysis to the predetermined standard weight of each of the plurality of patients and the specific one patient.

[0096] (Note 9) A body fluid volume estimation device according to any one of Notes 1 to 8, further comprising a storage unit that stores facial images of the multiple patients used for pre-training, information indicating the body fluid volume at the time the facial images of the multiple patients were captured, and multiple facial images of a specific one patient used for transfer learning, wherein the pre-training unit reads the facial images of the multiple patients and the information indicating the body fluid volume at the time the facial images of the multiple patients were captured from the storage unit to perform pre-training, and the transfer learning unit reads the multiple facial images of a specific one patient from the storage unit to perform transfer learning.

[0097] (Note 10) A body fluid volume estimation device according to any one of Notes 1 to 9, further comprising an imaging unit, wherein a facial image of a specific patient captured by the imaging unit is input to the estimation unit.

[0098] (Note 11) A method for estimating body fluid volume, comprising: pre-training multiple facial images of patients using information indicating the body fluid volume at the time the facial images of the multiple patients were captured as training information; constructing a trained model by further transfer learning of multiple facial images of a specific patient after the pre-training; and inputting the facial images of the specific patient into the trained model to estimate the body fluid volume of the specific patient at the time the facial images of the specific patient were captured.

[0099] (Note 12) A program that causes a computer to perform the following steps: pre-training of multiple facial images of patients using information indicating the volume of bodily fluids at the time the images were taken as training information; after the pre-training, further transfer learning of multiple facial images of a specific patient to construct a trained model; and inputting the facial images of the specific patient into the trained model to estimate the volume of bodily fluids of the specific patient at the time the facial images were taken. [Explanation of Symbols]

[0100] 100 Body fluid volume estimation device 1. Pre-learning section 2. Transfer Learning Section 3 Estimation part 4 Storage section 5. Imaging Department 1000 computers 1001 CPU 1002 ROM 1003 RAM 1004 Bus 1005 Input / Output Interface 1006 Input section 1007 Output section 1008 Storage section 1009 Communications Department 1010 Drive 1011 Magnetic disk 1012 Optical Disc 1013 Flexible disk 1014 Semiconductor memory Dp pre-training data Dt Transfer Learning Data

Claims

1. A pre-training unit that pre-trains multiple patient facial images using information indicating the volume of bodily fluids at the time the facial images of the multiple patients were captured as training information, After the aforementioned pre-training, the transfer learning unit constructs a trained model by further transfer learning of multiple facial images of a specific patient. The system includes an estimation unit that inputs the facial image of a specific patient into the trained model and estimates the body fluid volume of that specific patient at the time the facial image was captured. Body fluid volume estimation device.

2. The information indicating the body fluid volume when facial images of the multiple patients are captured includes information indicating the presence or absence of edema in the facial images of the multiple patients, and information indicating the weight of the multiple patients. The estimation unit inputs the facial image of the specific patient into the trained model to estimate the presence or absence of edema and the weight of the specific patient at the time the facial image was taken. The body fluid volume estimation device according to claim 1.

3. The estimation unit, Based on the estimated presence or absence of edema in the specified patient, the system detects whether the body fluid volume of the specified patient has changed relative to the standard body fluid volume of the specified patient, which is set in advance. From the estimated weight, the difference in the body fluid volume of the specific patient compared to the standard body fluid volume is obtained. The body fluid volume estimation device according to claim 2.

4. The information indicating the presence or absence of edema in the facial images of the aforementioned multiple patients is label information representing the presence or absence of edema. The pre-training unit pre-trains the feature quantities of face images with the same label information indicating the presence or absence of edema to be as similar as the information indicating weight. The body fluid volume estimation device according to claim 2 or 3.

5. The aforementioned pre-training unit performs pre-training using WeightSupMoCo (Weight-Aware Supervised Momentum Contrast). A body fluid volume estimation device according to any one of claims 2 or 3.

6. The aforementioned multiple patients and the aforementioned one specific patient are patients undergoing dialysis, The post-dialysis weight of each of the aforementioned multiple patients and the aforementioned specific patient, corresponding to the case without edema, is the value obtained by dividing the pre-dialysis weight of each of the aforementioned multiple patients and the aforementioned specific patient, corresponding to the case with edema, by the volume of body fluid removed by dialysis. The body fluid volume estimation device according to claim 2 or 3.

7. The aforementioned multiple patients and the aforementioned one specific patient are patients undergoing dialysis, The pre-dialysis weight of each of the aforementioned multiple patients and the aforementioned specific one patient, corresponding to the case where edema is present, is the value obtained by adding the volume of body fluid removed by dialysis to the post-dialysis weight of each of the aforementioned multiple patients and the aforementioned specific one patient, corresponding to the case where edema is absent. The body fluid volume estimation device according to claim 2 or 3.

8. The aforementioned multiple patients and the aforementioned one specific patient are patients undergoing dialysis, The pre-dialysis weight corresponding to the presence of edema in each of the aforementioned multiple patients and the aforementioned specific one patient is the value obtained by adding the volume of body fluid removed by dialysis to the predetermined standard weight of each of the aforementioned multiple patients and the aforementioned specific one patient. The body fluid volume estimation device according to claim 2 or 3.

9. The system further includes a storage unit that stores the facial images of multiple patients used in pre-training, information indicating the volume of bodily fluids at the time the facial images of the multiple patients were captured, and the facial images of a specific single patient used in transfer learning. The pre-learning unit reads from the storage unit the facial images of the multiple patients and information indicating the volume of bodily fluids at the time the facial images of the multiple patients were captured, and performs pre-learning. The transfer learning unit reads multiple facial images of a specific patient from the memory unit and performs transfer learning. The body fluid volume estimation device according to claim 1 or 2.

10. It is further equipped with an imaging unit, The facial image of the specific patient captured by the imaging unit is input to the estimation unit. The body fluid volume estimation device according to claim 1 or 2.

11. Multiple patient facial images are pre-trained using information indicating the volume of bodily fluids at the time the facial images of the multiple patients were captured as training data. After the aforementioned pre-training, a trained model is constructed by further transfer learning using multiple facial images of a specific patient. The facial image of the specific patient is input into the trained model to estimate the body fluid volume of the specific patient at the time the facial image was taken. Body fluid volume estimation method.

12. A process of pre-training multiple patient facial images using information indicating the volume of bodily fluids at the time of acquisition as training data, After the aforementioned pre-training, the process involves further transfer learning of multiple facial images of a specific patient to construct a trained model. The computer is instructed to input the facial image of a specific patient into the trained model and to perform the process of estimating the volume of bodily fluids of that specific patient at the time the facial image was taken. program.

Images