HEART FAILURE ESTIMATION DEVICE, HEART FAILURE ESTIMATION SYSTEM, HEART FAILURE ESTIMATION METHOD, AND HEART FAILURE ESTIMATION PROGRAM

Abstract

This heart failure estimation device acquires an image or video obtained by imaging the face of a patient. The heart failure estimation device extracts one or more feature parameters from the image or video. On the basis of the one or more feature parameters, the heart failure estimation device estimates the degree of heart failure of the patient.

Description

[Technical field]

[0001] The present disclosure relates to a heart failure estimation device, a heart failure estimation system, a heart failure estimation method, and a heart failure estimation program. [Background technology]

[0002] Conventionally, a system for automatically monitoring a patient's congestive heart failure (CHF) is known (for example, Japanese Patent No. 5281002). This system automatically monitors congestive heart failure (CHF) by receiving signals generated by non-invasively monitoring the patient's respiration and cardiac physiology.

[0003] Also, a heart failure aggravation assessment system that assesses the degree of aggravation of a patient's heart failure is known (e.g., Japanese Patent No. 6901042). Also, a heart failure estimation system that can reduce the workload of medical professionals in assessing the stage of heart failure is known (e.g., Japanese Patent No. 6893002). Summary of the Invention [Problem to be solved by the invention]

[0004] The present disclosure aims to provide a heart failure estimation device, a heart failure estimation system, a heart failure estimation method, and a heart failure estimation program that can estimate the degree of heart failure of a patient from an image or video of the patient's facial area. [Means for solving the problem]

[0005] In order to achieve the above-mentioned object, the heart failure estimation device of the present disclosure is a heart failure estimation device that includes an acquisition unit that acquires an image or video obtained by imaging a patient's face, an extraction unit that extracts one or more feature parameters from the image or video acquired by the acquisition unit, an estimation unit that estimates the degree of heart failure of the patient based on the one or more feature parameters extracted by the extraction unit, and an output unit that outputs the result estimated by the estimation unit. Effect of the Invention

[0006] According to the present disclosure, an effect is obtained in that the degree of heart failure of a patient can be estimated from an image or a moving image of the patient's face region. [Brief description of the drawings]

[0007] [Figure 1] 1 is a diagram illustrating an example of a schematic configuration of a heart failure estimation system according to a first embodiment. [Diagram 2] FIG. 13 is a diagram for explaining feature parameters. [Diagram 3] FIG. 13 is a diagram for explaining feature parameters. [Figure 4] FIG. 13 is a diagram for explaining feature parameters. [Diagram 5] FIG. 13 is a diagram for explaining feature parameters. [Figure 6-1] FIG. 13 is a diagram for explaining feature parameters. [Figure 6-2] FIG. 13 is a diagram for explaining feature parameters. [Figure 6-3] FIG. 13 is a diagram for explaining feature parameters. [Figure 6-4] FIG. 13 is a diagram for explaining feature parameters. [Figure 7] FIG. 13 is a diagram for explaining feature parameters. [Figure 8] FIG. 13 is a diagram for explaining feature parameters. [Figure 9] FIG. 13 is a diagram for explaining feature parameters. [Figure 10] FIG. 13 is a diagram for explaining feature parameters. [Figure 11-1] FIG. 13 is a diagram for explaining feature parameters. [Figure 11-2] FIG. 13 is a diagram for explaining feature parameters. [Figure 12] FIG. 13 is a diagram for explaining feature parameters. [Figure 13] FIG. 13 is a diagram for explaining feature parameters. [Figure 14]FIG. 2 is a diagram illustrating an example of a usage form of the heart failure estimation system according to the first embodiment. [Figure 15] FIG. 2 is a diagram illustrating an example of a computer constituting a heart failure estimation device. [Figure 16] 1 is a diagram for explaining processing executed by the heart failure estimation system of the present embodiment. FIG. [Figure 17] FIG. 11 is a diagram illustrating an example of a schematic configuration of a heart failure estimation system according to a second embodiment. [Figure 18] FIG. 11 is a diagram illustrating an example of a usage form of the heart failure estimation system according to the second embodiment. [Figure 19] FIG. 11 is a diagram illustrating an example of a usage form of the heart failure estimation system according to the second embodiment. [Figure 20] FIG. 1 is a diagram for explaining an embodiment. [Figure 21] FIG. 1 is a diagram for explaining an embodiment. [Figure 22-1] FIG. 1 is a diagram for explaining an embodiment. [Figure 22-2] FIG. 1 is a diagram for explaining an embodiment. [Figure 22-3] FIG. 1 is a diagram for explaining an embodiment. [Figure 22-4] FIG. 1 is a diagram for explaining an embodiment. [Figure 23] FIG. 1 is a diagram for explaining an embodiment. [Figure 24] FIG. 1 is a diagram for explaining an embodiment. [Diagram 25] FIG. 1 is a diagram for explaining an embodiment. [Figure 26] FIG. 1 is a diagram for explaining an embodiment. [Figure 27-1] FIG. 1 is a diagram for explaining an embodiment. [Figure 27-2] FIG. 1 is a diagram for explaining an embodiment. [Figure 27-3] FIG. 1 is a diagram for explaining an embodiment. [Figure 27-4] FIG. 1 is a diagram for explaining an embodiment. [Figure 27-5] FIG. 1 is a diagram for explaining an embodiment. [Figure 27-6]FIG. 1 is a diagram for explaining an embodiment. [Figure 27-7] FIG. 1 is a diagram for explaining an embodiment. [Figure 27-8] FIG. 1 is a diagram for explaining an embodiment. [Figure 27-9] FIG. 1 is a diagram for explaining an embodiment. [Figure 27-10] FIG. 1 is a diagram for explaining an embodiment. [Figure 27-11] FIG. 1 is a diagram for explaining an embodiment. [Figure 28] FIG. 13 is a diagram for explaining a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, the embodiments will be described in detail with reference to the drawings.

[0009] <Heart failure estimation system according to the first embodiment>

[0010] 1 shows a heart failure estimation system 10 according to the first embodiment. As shown in FIG 1, the heart failure estimation system 10 of the first embodiment includes a camera 12, a NIR (Near Infra Red) camera 13, a heart failure estimation device 14, and a display device 16.

[0011] The heart failure estimation system 10 is capable of estimating the state of the patient's heart based on an image of the patient's face region captured by the camera 12 or a video of the patient's face region. In this embodiment, an example will be described in which the heart failure estimation system 10 calculates the degree of heart failure of the patient as the state of the patient's heart, and estimates whether the patient has heart failure or not based on the degree. The state of the patient's heart is not limited to the degree of heart failure of the patient, and may be the state of the patient's cardiac load, the state of pulmonary congestion of the patient, the state of fluid retention of the patient, or the like. A specific description will be given below.

[0012] 1, the heart failure estimation device 14 functionally includes an acquisition unit 20, an image data storage unit 22, a reference data storage unit 24, an extraction unit 26, an estimation unit 28, and an output unit 29. The heart failure estimation device 14 is realized by a computer as described below.

[0013] The acquisition unit 20 acquires an image of the patient's face region or a video of the patient's face region. In this embodiment, an example will be described in which the acquisition unit 20 acquires a video of the patient's face region. From the video of the patient's face region, it is possible to acquire an image of the patient's face region at each time. Note that the patient is a subject for whom it is estimated whether or not he or she has heart failure. The acquisition unit 20 then stores the video of the patient's face region in the image data storage unit 22.

[0014] The image data storage unit 22 stores moving images of the patient acquired by the acquisition unit 20.

[0015] The reference data storage unit 24 stores one or more characteristic parameters extracted from an image or a video of a reference patient (hereinafter simply referred to as reference data) of a patient who is known to have heart failure or not. The reference data is data of a patient whose degree of heart failure has already been diagnosed. Alternatively, the reference data storage unit 24 may store derived data based on these data.

[0016] The reference data storage unit 24 stores a calculation model for calculating the degree of heart failure of a patient using one or more feature parameters extracted from an image or video. The calculation model is, for example, a statistical model or a machine learning model. For example, when a regression model is used as the statistical model, the calculation formula of the regression model and the coefficient value of the regression model are stored in the reference data storage unit 24 as the calculation model. Also, for example, when a machine learning model is used, a learned model that is a combination of the structural formula of the machine learning model and acquired learned parameters of the machine learning model is stored in the reference data storage unit 24 as the calculation model. The coefficients or parameters of the statistical model or machine learning model are acquired in advance based on training data collected in advance. These calculation models are used when calculating the degree of heart failure of a user.

[0017] The degree of heart failure of a patient may be calculated using the similarity between feature parameters extracted from an image or video obtained from the patient and reference data or feature parameters extracted from the reference data, without using a statistical model or a machine learning model, etc. In this embodiment, an example will be described in which the degree of heart failure of a user is calculated using a calculation model and feature parameters for calculating the degree of heart failure of a user.

[0018] The extraction unit 26 reads out the moving image data stored in the image data storage unit 22. Then, the extraction unit 26 executes various processes on the moving image data and extracts one or more feature parameters from the moving image data.

[0019] Specifically, the extraction unit 26 extracts the positions of each part of the face from the image by applying a known machine learning based method to the image included in the video data. In this case, the positions of each part of the face are extracted as feature points. Then, the extraction unit 26 detects the patient's facial outline, eyes, mouth, etc. based on the extraction result of the feature points.

[0020] For example, the extraction unit 26 uses a known method such as Facemesh (https: / / google.github.io / mediapipe / solutions / face_mesh.html) or DLIB (http: / / dlib.net / ) to extract feature points from an image of the patient's face region and detect the patient's facial outline, eyes, mouth, etc. The extraction of feature points can be applied to any of black and white, color, and near-infrared still images and videos. In the case of a video, feature points are extracted from one frame of images constituting the video.

[0021] Then, the extraction unit 26 generates one or more feature parameters based on the plurality of feature points extracted from the image of the patient's face region. The feature parameters will be described later.

[0022] The estimation unit 28 estimates the degree of heart failure of the patient based on the one or more characteristic parameters extracted by the extraction unit 26.

[0023] For example, the estimation unit 28 inputs one or more feature parameters to the trained model based on the one or more feature parameters extracted by the extraction unit 26 and the trained model stored in the reference data storage unit 24, thereby obtaining a score representing the degree of heart failure of the patient output from the trained model. Then, the estimation unit 28 estimates the degree of heart failure of the patient based on the score.

[0024] Alternatively, for example, the estimation unit 28 inputs the one or more characteristic parameters as explanatory variables of the statistical model based on the one or more characteristic parameters extracted by the extraction unit 26 and the statistical model stored in the reference data storage unit 24, and acquires the objective variable obtained from the statistical model as a score representing the degree of heart failure of the patient. Then, the estimation unit 28 estimates the degree of heart failure of the patient based on the score.

[0025] The feature parameters used in this embodiment will be described below.

[0026] [1. Parameters related to the patient's facial shape] The points shown in Figures 2 and 3 are examples of feature points extracted from an image of the patient's face region. As shown in Figures 2 and 3, feature points (468 feature points are extracted in the example shown) are extracted from various locations on the patient's face. Each feature point has three-dimensional coordinates. An index (identification number) for identifying the feature point is added to each feature point.

[0027] In this embodiment, one or more feature parameters are generated based on these multiple feature points. The feature parameters in this embodiment include at least one of a parameter related to the shape of the patient's face, a parameter related to the patient's complexion, a parameter related to the patient's eyes, a parameter related to the patient's mouth, a parameter representing a result of a near-infrared analysis of the amount of moisture in the patient's face, and a parameter representing three-dimensional shape data of the patient's face.

[0028] The parameters relating to the shape of the patient's face that are extracted in this embodiment are at least one of the following parameters.

[0029] (1) Parameter representing face area Fig. 2 is a diagram for explaining a parameter representing the area of ​​a patient's face. As shown in Fig. 2, in this embodiment, an area surrounded by feature points located on the outline of the patient's head is used as a parameter representing the area of ​​the face. Since detection of the outline of the patient's head may be affected by bangs, the area of ​​the patient's face below the eyebrows, Area_brow, is also extracted as a feature parameter. The sum of the area of ​​the patient's face below the eyebrows, Area_brow, and the area of ​​the patient's face above the eyebrows is used as the parameter representing the area of ​​the face, Area_all.

[0030] (2) Parameter representing the horizontal width of the face FIG. 3 is a diagram for explaining a parameter representing the horizontal width of a patient's face. As shown in FIG. 3, in this embodiment, a face width corresponding to a distance between feature points located on the outline of the patient's head is used as a parameter representing the horizontal width of the face (hereinafter, also simply referred to as "horizontal width of face"). In the example shown in FIG. 3, the horizontal width W05_29, horizontal width W06_28, horizontal width W07_27, horizontal width W08_26, horizontal width W09_25, horizontal width W10_24, horizontal width W11_23, horizontal width W12_22, horizontal width W13_21, horizontal width W14_20, horizontal width W15_19, and horizontal width W16_18 of the lower part of the eye are shown. The numbers are indexes for identifying the feature points. Note that the size of the patient's face within the shooting angle of view is different every time an image is shot. In this embodiment, in order to correct this, normalization is performed by dividing each of the above widths by the distance (reference(w)) between the feature point representing the inner corner of the left eye and the feature point representing the inner corner of the right eye. This normalizes the width of the face, which differs for each image, and extracts a parameter representing the width of the face. Note that in this embodiment, a case in which the width of the face is normalized by the distance (reference(w)) between the feature point representing the inner corner of the left eye and the feature point representing the inner corner of the right eye is described as an example, but this is not limited to this. For example, a marker may be attached to the patient's face as a reference. Since the size (width) of the marker is known, the width of the face can be normalized according to the size (width) of the marker.

[0031] (3) Parameter representing the up-down direction of the face In this embodiment, the inclination of the patient's head in the pitch direction is used as a parameter (Pitch) representing the vertical orientation of the face. Specifically, based on the three-dimensional coordinates of each of the above-mentioned feature points, a parameter representing the vertical orientation of the face is calculated from the depth coordinate information of each feature point. For example, the depth coordinate of a feature point located in the upper half of the patient's face is compared with the depth coordinate of a feature point located in the lower half of the patient's face, and if the feature point located in the upper half of the face is located at a depth position farther from the camera 12 than the feature point located in the lower half of the face, the patient's head is considered to be inclined in a direction to protrude the chin, and a parameter representing the vertical orientation of the face is calculated. On the other hand, if the feature point located in the lower half of the face is located at a depth position farther from the camera 12 than the feature point located in the upper half of the face, the patient's head is considered to be inclined in a direction to retract the chin, and a parameter representing the vertical orientation of the face is calculated.

[0032] (4) Parameters of the vector representing facial features In this embodiment, a vector representing the facial features is used as a parameter. For example, by inputting a face image to a known machine learning model, a vector representing the facial features (hereinafter also simply referred to as a "face vector") is output from the machine learning model. In this embodiment, the face vector obtained in this manner is used as a parameter. In this embodiment, a known technology called FaceaNet (https: / / github.com / davidsandberg / facenet) is used to use a 512-dimensional face vector as a parameter. The features and positions of each part of the face are reflected in the face vector.

[0033] (5) Parameter representing change in face vector (embdiff) In this embodiment, a parameter (embdiff) that represents a change in face vector is used. The parameter that represents a change in face vector is at least one of the following parameters.

[0034] (5-1. Feature parameter (embdiff_day) representing the distance between the face vector obtained on the target day and the face vector obtained on the day before the target day) For example, the distance between the face vector obtained from the image of the patient on the target day D2 and the face vector obtained from the image of the patient on the day before the target day D2, D1, is generated as a new feature parameter. FIG. 4 is a diagram for explaining this new feature parameter (embdiff_day). As shown in FIG. 4, for example, an image of the patient's face region captured on 6 / 27 is input to a convolutional neural network (CNN) used in the above-mentioned FaceaNet, to extract a 512-dimensional face vector on 6 / 27. Also, as shown in FIG. 4, an image of the patient's face region captured on 6 / 28 is input to the CNN to extract a 512-dimensional face vector on 6 / 28. In this way, the face vectors of each day as shown in FIG. 4 are obtained. Then, the distance between the face vector obtained from the image of the patient on 6 / 28, which is the target day, and the face vector obtained from the image of the patient on 6 / 27, the day before the target day, is calculated. Specifically, for example, the distance between the 512-dimensional face vector of 6 / 27 and the 512-dimensional face vector of 6 / 28 is calculated as the feature parameter of 6 / 28 (6 / 28 embdiff_day). Similarly, the feature parameters of each day are calculated.

[0035] (5-2. Feature parameters (embdiff_dod, embdiff_doa) representing the distance between the face vector obtained on the target day and the face vector in the remission or exacerbation period) For example, the distance between the face vector obtained on the target day and the face vector in the remission or exacerbation period is generated as new feature parameters (embdiff_dod, embdiff_doa).

[0036] Specifically, the distance between the face vector obtained from the image of the patient on the target date D1 and the face vector obtained from the image of the patient on the date Dx when the patient is discharged from the hospital after entering the remission phase is generated as a new feature parameter (embdiff_dod). In this case, the current state of the patient is estimated based on the face vector of the patient's face in the remission phase. Note that "dod" is an abbreviation for "date of discharge".

[0037] FIG. 5 is a diagram for explaining this new feature parameter (embdiff_dod). As shown in FIG. 5, for example, by inputting an image of a patient's face region captured on 6 / 27 to the CNN, a 512-dimensional face vector on 6 / 27 is extracted. Also, as shown in FIG. 5, by inputting an image of a patient's face region captured on 6 / 28 to the CNN, a 512-dimensional face vector on 6 / 28 is extracted. In this manner, face vectors for each day as shown in FIG. 5 are obtained. Then, the distance between the face vector obtained from the image of the patient on 6 / 27, which is the target day, and the face vector obtained from the image of the patient on 7 / 9, which is the day the patient was discharged from the hospital, is calculated. For example, the distance between the 512-dimensional face vector on 6 / 27 and the 512-dimensional face vector on 7 / 9 is calculated as the feature parameter for 6 / 27 (6 / 27 embdiff_dod). Similarly, the feature parameters for each day are calculated.

[0038] In addition, the distance between the face vector obtained from the image of the patient on the target date D1 and the face vector obtained from the image of the patient on the date Dx when the patient entered the exacerbation stage and was admitted is generated as a new feature parameter (embdiff_doa). In this case, the current state of the patient is estimated based on the face vector of the patient's face in the exacerbation stage. Note that "doa" is an abbreviation for "date of admission."

[0039] Figure 6-1 is a diagram showing an example of transitions of embdiff_dod and embdiff_doa. Figure 6-1 (A) and (B) show transitions of the values ​​of embdiff_dod and embdiff_doa when a patient improves due to therapeutic intervention for heart failure. The transitions of embdiff_dod and embdiff_doa show the actual transitions of the patient, but the facial image shown at the bottom is displayed as a schematic diagram from the viewpoint of privacy.

[0040] In FIG. 6-1(A), the facial image in the remission phase is used as a reference. The vertical axis (Y axis) of the graph in the upper part shown in FIG. 6-1(A) represents the value of embdiff_dod, and the horizontal axis (X axis) represents the days. The days represented by the X axis of the graph in FIG. 6-1(A) represent the patient's Day 1 (exacerbation phase) to Day 12 (remission phase). The lower part of FIG. 6-1(A) shows the facial images of the patient corresponding to each date. Day 1 in the lower part of FIG. 6-1(A) corresponds to the exacerbation phase of heart failure, and Day 12 corresponds to the remission phase of heart failure. As shown in the lower part of FIG. 6-1(A), it can be seen that the degree of facial swelling decreases with the progress from Day 1 to Day 12. Also, as shown in FIG. 6-1(A), the difference (distance) between the facial vector obtained from the facial image on Day 1, when heart failure is most aggravated, and the facial vector obtained from the facial image in the remission phase as a reference is the largest. Furthermore, as the patient moves from the worsening phase (Day 1) to the remission phase (Day 12) due to the therapeutic intervention for heart failure, the difference between the face vectors becomes smaller. For this reason, it can be seen that embdiff_dod changes according to the state of heart failure, and the patient's condition gradually improves. By taking embdiff_dod on the vertical axis (Y-axis) and days or times on the horizontal axis (X-axis) in this way, it is possible to evaluate the progression of the disease (for example, the speed at which heart failure worsens or improves) from the degree of change in embdiff_dod (for example, calculated by dividing the difference between embdiff_dod on one day and embdiff_dod on another day by the number of days).

[0041] In addition, in FIG. 6-1(B), the facial image in the worsening period is used as the reference. The vertical axis (Y axis) of the graph shown in FIG. 6-1(B) represents the value of embdiff_doa, and the horizontal axis (X axis) represents the days. The days represented by the X axis of the graph in FIG. 6-1(B) represent the patient's Day 1 (worsening period) to Day 12 (remission period). As shown in FIG. 6-1(B), the difference (distance) between the facial vector of the facial image on Day 1, when the heart failure is most worse, and the facial vector obtained from the facial image in the worsening period as the reference is, is the smallest. In addition, the difference between the facial vectors increases as the treatment intervention for heart failure progresses from the worsening period on Day 1 to the remission period on Day 12. This is because the facial vector of each day moves away from the facial vector in the worsening period as the days go by, and it can be seen that embdiff_doa changes according to the state of heart failure, as in FIG. 6-1(A). In addition, the degree of change in embdiff_doa also makes it possible to evaluate the progression of the disease (for example, the speed at which heart failure worsens or improves).

[0042] The reference patient image is acquired in advance by capturing an image of the patient's face during the worsening or remission / stable period. Specifically, the reference patient image is acquired in advance by capturing an image of the patient's face at the time of transportation to the hospital, when the patient visits the hospital for the first time, when the patient is admitted, when the patient is discharged, when there is a change, or when the patient is at home. Then, the difference between the face vector obtained from the image of the patient on each day in a state of heart failure and the face vector obtained from the reference image is calculated as a feature parameter. In this way, by using the difference between the face vector obtained from the reference image and the face vector obtained from the image captured on the target day, the time series of the patient's condition can be observed. In this way, by capturing the change over time between the face vector obtained from the reference image and the face vector obtained from the image captured on the target day, it is possible to correct the so-called "difference in facial features that differ from patient to patient," and it is also possible to improve the accuracy of the estimation of the state of heart failure. In addition, if the accuracy of the trained model is improved by accumulating a large amount of data, it is also possible to estimate the state of heart failure with a single image capture without comparing the state over time.

[0043] When a patient falls into a state of heart failure, the patient's body is unable to excrete bodily fluids (specifically, water in the body) effectively. This causes swelling in the patient's body. This swelling changes the shape, contour, area, width, etc. of the face. When swelling occurs in the patient's face, the patient's face may be recognized as the face of another person. For this reason, the heart failure estimation system 10 of this embodiment extracts parameters related to the shape of the patient's face from an image of the patient's face area, and estimates the patient's heart failure based on the parameters. Furthermore, since each parameter described above or below is calculated as a continuous variable, it is possible to estimate the speed of exacerbation or improvement of heart failure from the magnitude of change in the value.

[0044] [2. Parameters related to the patient's complexion] In this embodiment, a parameter related to the complexion of the patient is used. The parameter related to the complexion of the patient that is extracted in this embodiment is at least one of the following parameters.

[0045] (1) Parameters that represent the patient’s facial hue In this embodiment, the hue of the patient's face is extracted from the image of the patient's face region and used as a parameter representing the hue of the face.

[0046] (2) Parameter representing the green component (Green) of the patient's face In this embodiment, parameters representing the green components (Green) of the cheeks, glabella, and nose in the patient's face are used. The cheeks, glabella, and nose regions are identified based on the positions of the aforementioned feature points.

[0047] Facial color changes mainly depending on the amount of melanin and hemoglobin. When the skin tans, it becomes darker because the amount of melanin increases. Hemoglobin is a protein that carries oxygen, and is known to have an absorption peak near 550 nm in the visible light wavelength range. This wavelength range near 550 nm corresponds to the green component of the three primary colors of light. When the amount of hemoglobin in the body decreases due to bleeding or anemia, the reflected light near 550 nm increases, and the redness of the face decreases, and the human eye perceives the face as being pale. The top graph in Figure 6-2 shows the relationship between the blood hemoglobin concentration and the value of the green component (Green) of the cheek obtained from an image for a specific case (G-HF017). The horizontal axis of the graph shows the value of the patient's blood hemoglobin concentration (Hb, g / dl) obtained from a blood test, and the vertical axis shows the value of the green component obtained from an image of the patient's face. It has been shown that as the hemoglobin concentration in blood increases, the absorption of light of wavelength components corresponding to green by hemoglobin increases, and the value of the green component obtained from the face image decreases. The graph at the bottom of Figure 6-2 shows the relationship between blood hemoglobin concentration and the hue value of the lower eyelid obtained from an image for a specific case (G-HF017). The horizontal axis of the graph shows the patient's blood hemoglobin concentration (Hb, g / dl) value obtained from a blood test, and the vertical axis shows the hue value obtained from the patient's facial image. As the blood hemoglobin concentration increases, the hue value obtained from the facial image decreases, indicating a change in the facial color toward redder color. The hue value indicates the hue angle in the L*a*b space, and the direction approaching 0 indicates an increase in redness. The graph in Figure 6-3 shows the relationship between blood hemoglobin concentration and the hue value (Hue) of the lower eyelid obtained from the image for several cases. The horizontal axis of the graph in Figure 6-3 shows the blood hemoglobin concentration (Hb, g / dl) value of the patient obtained from the blood test, and the vertical axis shows the hue value (Hue) obtained from the face image of the patient. As mentioned above, facial color changes mainly depending on the amount of melanin and hemoglobin. Melanin increases when exposed to ultraviolet rays from sunlight, so the amount varies from person to person. In addition, hemoglobin values ​​also vary from person to person for each case. The graph in Figure 6-3 shows the difference in hue due to individual differences caused by such melanin and hemoglobin amounts. However, when looking at changes within each case, unless there is a large change in the amount of melanin, the change in facial color due to the change in hemoglobin amount will be the main change. In particular, in an environment where there is little opportunity for exposure to ultraviolet rays, such as during hospitalization, it is possible to capture changes in blood hemoglobin concentration by measuring changes in facial color.

[0048] When a patient is in a state of heart failure, the concentration of hemoglobin generally decreases due to an increase in water retention in the body, leading to anemia. In addition, when a patient is in a state of heart failure, blood is redistributed because blood cannot be sent sufficiently to the entire body. That is, in order to ensure blood supply to important organs (e.g., the brain or internal organs), blood vessels constrict and blood supply to organs that do not have a high priority from the viewpoint of maintaining vital activity, such as the skin and muscles, is restricted. If blood vessels constrict and blood sharing is restricted, the amount of blood distributed to the organ decreases, and the supply of hemoglobin contained in the blood also decreases, so that the patient's face becomes pale, similar to anemia. For this reason, the heart failure estimation system 10 of this embodiment extracts parameters related to the patient's facial color from an image of the patient's facial area, and estimates the patient's heart failure based on the parameters. In order to obtain parameters representing the facial color, a colored marker for normalizing the facial color may be attached to the patient's face, etc. Since the color of the marker is known, the facial color can be normalized according to the color of the marker. The top graph in Figure 6-4 shows the time course of blood hemoglobin concentration and lower eyelid hue (Heu) obtained from an image for a specific case (G-HF017) during hospitalization and treatment for heart failure. The horizontal axis of the graph indicates the number of days since the day of hospitalization, and the further to the right the progress of treatment and the more alleviation of heart failure. The left vertical axis indicates the patient's blood hemoglobin concentration (Hb, g / dl) obtained from a blood test, and the right vertical axis indicates the hue (Hue) value obtained from an image of the patient's face. Focusing on the six days immediately after hospitalization due to worsening heart failure, the amount of water in the blood vessels decreases due to the removal of water by diuretics during the acute phase of treatment, and the blood hemoglobin concentration shows a transient increase, and then from the seventh day onwards, it remains at a stable value. Corresponding to this change in blood hemoglobin concentration, the hue value also decreases immediately after hospitalization (the decrease in hue corresponds to an increase in redness of the face), and it can be seen that it then shows a stable trend. In particular, from day 7 onwards, the hemoglobin concentration, which had risen transiently during the acute phase, began to drop again and stabilize. Despite the drop in hemoglobin concentration, the hue value did not increase and remained stable. This is thought to be due to the improvement of the heart failure condition, the improvement (increase) of blood flow to the skin, which had been restricted by redistribution, and the increase in the absolute amount of hemoglobin in the facial area. The graph at the bottom of Figure 6-4 shows the correlation between the NYHA functional classification and the lower eyelid hue value obtained from the image for a specific case (G-HF017). The horizontal axis of the graph shows the symptom index (NYHA functional classification) value at the time of data acquisition for the case evaluated by a cardiologist, and the vertical axis shows the hue value obtained from the patient's facial image. Heart failure treatment alleviates heart failure symptoms and reduces the NYHA functional classification (alleviating symptoms), but similarly, with improvement in the heart failure condition, the hue value decreases (increasing redness) reflecting an increase in hemoglobin concentration and blood flow (absolute hemoglobin amount), and it has been confirmed that the degree of heart failure symptoms and facial hue show a strong correlation (R=0.93).

[0049] [3. Patient's eye parameters] In this embodiment, a parameter related to the patient's eye is utilized. The parameter related to the patient's eye extracted in this embodiment is at least one of the parameters in Table 1 below.

[0050] [Table 1]

[0051] 7 to 9 are diagrams for explaining the parameters listed in Table 1 above.

[0052] (1) A parameter that represents the aspect ratio of the patient's eye (EAR: Eye Aspect Ratio) As shown in Fig. 7, by the above-mentioned process, for example, feature points P1, P2, P3, P4, P5, and P6 near the eyes are extracted. Therefore, a parameter (EAR) representing the aspect ratio of the patient's eye is calculated according to the distance between the feature points near the eyes as shown in Fig. 7. Specifically, the parameter (EAR) representing the aspect ratio of the patient's eye is calculated according to the following formula.

[0053]

number

[0054] (2) A parameter (diffLR) that represents the difference in the aspect ratio (EAR) between the patient's left and right eyes. Depending on the feature points near the eyes as shown in FIG. 7, a parameter (diffLR) is calculated that represents the difference in the aspect ratio (EAR) between the patient's left and right eyes.

[0055] (3) A parameter (width) that represents the duration of eye closure in one blink According to the feature points near the eyes as shown in FIG. 7, a parameter representing the time width of the closed eyes when the patient blinks is calculated as shown in FIG. 8. FIG. 8 shows an example in which the EAR value of the open eye state by maximum likelihood estimation is set to 100% and the EAR value of 0 is set to 0% among the parameters (EAR) representing the aspect ratio of the patient's eyes at each time. Therefore, the value of the vertical axis (Y axis) of FIG. 7 represents the degree of eye opening [%] calculated from the parameter (EAR) representing the aspect ratio of the patient's eyes. For example, as shown in FIG. 8, the time width W80width when the degree of eye opening is 80% and the time width W20width when the degree of eye opening is 20% are calculated as the parameter (width) representing the time width of the closed eyes in one blink.

[0056] (4) A parameter that represents the percentage of time the eyes are closed per unit time (fcnt) A parameter (fcnt) representing the percentage of eyes closed per unit time is calculated according to the distance between feature points near the eyes as shown in Fig. 7. For example, a value obtained by dividing the sum of the durations during which the degree of eye opening [%] is 80% or less within a given time by the given time is calculated as a parameter (W80fcnt) representing the percentage of eyes closed per unit time. Also, a value obtained by dividing the sum of the durations during which the degree of eye opening [%] is 20% or less within a given time by the given time is calculated as a parameter (W20fcnt) representing the percentage of eyes closed per unit time.

[0057] (5) Parameter representing the blink interval (b_inv) A parameter (b_intv) representing the blink interval is calculated according to the distance between feature points near the eyes as shown in Fig. 7. For example, as shown in Fig. 8, the time span (b_inv) from a certain time when the eye opening degree [%] falls below 20% to the next time when the parameter (EAR) representing the aspect ratio of the patient's eyes falls below 20% is calculated as the parameter representing the blink interval. Then, the average (b_intv_avg) of the parameter representing the blink interval (b_inv) of the patient's eyes is calculated as is calculated.

[0058] (6) Parameter representing the number of blinks per unit time (Close) A parameter (Close) representing the number of blinks per unit time is calculated according to the feature points near the eyes as shown in Fig. 7. For example, the parameter (Close) representing the number of blinks per unit time is calculated by dividing the number of times the degree of eye opening falls below 20% within a predetermined time by the predetermined time.

[0059] (7) Parameter representing the speed at which the eyes close (grad_close) A parameter (grad_close) representing the speed at which the eyes are closed is calculated according to the feature points near the eyes as shown in FIG. 7. The value of the vertical axis (Y axis) in FIG. 9 represents the degree of eye opening [%] calculated from the parameter (EAR) representing the aspect ratio of the patient's eyes, as in FIG. 8. The speed at which the eyes are closed corresponds to the slope (grad_close) of the decrease in the degree of eye opening as shown in FIG. 9. In addition, a parameter (grad_close_avg) representing the average speed at which the eyes are closed (grad_close) is calculated by averaging the slope (grad_close) of the decrease in the degree of eye opening.

[0060] (8) Parameter representing the speed at which the eyes open (grad_open) A parameter (grad_open) representing the speed at which the eyes are opened is calculated according to the feature points near the eyes as shown in Fig. 7. The speed at which the eyes are opened corresponds to the gradient (grad_open) of the increase in the degree of eye opening as shown in Fig. 9. In addition, a parameter (grad_open_avg) representing the average speed at which the eyes are opened (grad_open) is calculated by averaging the gradient (grad_open) of the increase in the degree of eye opening.

[0061] (9) Parameter that represents the length of time the eyes are closed (area) A parameter (area) representing the length of time the eyes are closed is calculated according to the feature points near the eyes as shown in Fig. 7. For example, as shown in Fig. 9, the area of ​​the region formed by the length of time the degree of opening of the eyes is 80% or less, the parameter (grad_close) representing the speed of closing the eyes, and the parameter (grad_open) representing the speed of opening the eyes may be calculated as the parameter (area) representing the length of time the eyes are closed. Alternatively, the parameter (area) representing the length of time the degree of opening of the eyes is 80% or less may simply be calculated as the parameter (area) representing the length of time the eyes are closed.

[0062] (10) Parameter representing the average speed of opening and closing the eyes (grad_avg) A parameter (grad_avg) representing an average speed when the eyes are opened and closed is calculated according to the feature points near the eyes as shown in Fig. 7. For example, a parameter (grad_avg) representing an average speed when the eyes are opened and closed is calculated based on a parameter (grad_close) representing a speed when the eyes are closed and a parameter (grad_open) representing a speed when the eyes are opened.

[0063] When a patient is in a state of heart failure, swelling occurs in the patient's face. In this case, moisture tends to accumulate in parts of the face that have relatively more space (in other words, loose parts of the face). For example, the eyelids are usually a part that has more space, and when moisture accumulates there, the eyelids swell, making it difficult to open the eyes and making the eyes smaller (more specifically, the opening of the eyes decreases and the aspect ratio changes). In addition, when a patient is in a state of heart failure, the patient feels fatigued and has difficulty in keeping the eyes open. Due to these changes, the patient's eyes are closed for longer periods of time, and they blink more frequently and take longer to blink. While normal people blink instantly, people in a state of heart failure blink slowly. Furthermore, people in a state of heart failure try to look forward with eyes that are difficult to open due to swollen eyelids, so they tend to turn their faces upwards (specifically, with their chins protruding). For this reason, the heart failure estimation system 10 of the present embodiment extracts parameters related to the patient's eyes from an image of the patient's face region, and estimates the patient's heart failure based on the parameters.

[0064] [4. Patient oral parameters] In this embodiment, parameters related to the patient's mouth are utilized. The parameters related to the patient's mouth extracted in this embodiment are at least one of the parameters in Table 2 below.

[0065] [Table 2]

[0066] Fig. 10 is a diagram for explaining each parameter listed in the above Table 2. As shown in Fig. 10, by the above-mentioned processing, for example, feature points near the mouth are extracted. Note that the numbers in the figure are indexes for identifying feature points near the mouth.

[0067] (1) Parameter that represents the aspect ratio of the patient's mouth (m_OUTLINE) A parameter (m_OUTLINE) representing the aspect ratio of the patient's mouth is calculated according to the feature points near the mouth as shown in Fig. 10. For example, in the example shown in Fig. 10, a value (m_OUTLINE) is calculated by dividing the average value of the length of the line segment connecting feature point 37 and feature point 84, the length of the line segment connecting feature point 0 and feature point 17, and the length of the line segment connecting feature point 267 and feature point 314 by the length of the line segment connecting feature point 61 and feature point 291.

[0068] (2) Parameter (m_OPEN) that indicates the degree to which the patient’s mouth is open A parameter (m_OPEN) that indicates the degree of opening of the patient's mouth is calculated according to the feature points near the mouth as shown in Fig. 10. For example, in the example shown in Fig. 10, the length of the line segment connecting feature point 82 and feature point 87, the length of the line segment connecting feature point 13 and feature point 14, and the length of the line segment connecting feature point 312 and feature point 313 are calculated. The average length (m_OPEN) of the line segments connecting the feature points 317 is calculated.

[0069] (3) Parameter representing the left corner of the mouth (m_gradL) A parameter (m_gradL) representing the left corner of the mouth is calculated according to the feature points near the mouth as shown in Fig. 10. For example, in the example shown in Fig. 10, the angle between a straight line (dashed line) set in the left-right direction with respect to the mouth and a straight line representing the left corner of the mouth set by the feature points near the left corner of the mouth is calculated as the parameter (m_gradL) representing the left corner of the mouth. Note that the line connecting feature points 0 and 17 in the upper part of Fig. 10 is set as a vertical reference line, and the line that intersects this reference line at a feature point near the upper end of the lower lip (dashed line shown in the lower part of Fig. 10) is set as a horizontal reference line.

[0070] (4) Parameter representing the right corner of the mouth (m_gradR) A parameter (m_gradR) representing the right corner of the mouth is calculated according to the feature points near the mouth as shown in Fig. 10. For example, in the example shown in Fig. 10, the angle between a straight line (dashed line) set in the left-right direction with respect to the mouth and a straight line representing the right corner of the mouth set by the feature points near the right corner of the mouth is calculated as the parameter (m_gradR) representing the right corner of the mouth.

[0071] As described above, when a patient has heart failure, the patient must open his / her eyes by lifting his / her heavy eyelids that are swollen. Therefore, a patient with heart failure needs to make a certain effort to open his / her eyelids. In addition, because the patient tries to face forward with eyes that are difficult to open due to swollen eyelids, the patient's face tends to turn upward. Furthermore, in a state of heart failure, moisture also accumulates in the cheeks near the corners of the mouth, which may act as a weight due to gravity and push the corners of the mouth down. Due to these influences, when a patient has heart failure, the effort required to open the eyes may result in a so-called clenching of teeth, the face may turn upward, or the swelling of the cheeks may become a weight, causing the corners of the mouth to deform or displace downward. In addition, as with the eyelids, the thickness and shape of the lips change due to the occurrence of swelling. For this reason, the heart failure estimation system 10 of this embodiment extracts parameters related to the patient's mouth from an image of the patient's face area, and estimates the patient's heart failure based on the parameters.

[0072] [5. Parameters that represent the results of near-infrared analysis of the amount of moisture on the patient's face] In this embodiment, a parameter representing the result of the analysis of the moisture content of the patient's face by near-infrared is used. It is known that water has an absorption peak in the near-infrared wavelength range as a spectral characteristic. By irradiating the patient's face with light including near-infrared rays and observing the scattered light, the degree of light absorption by water in the skin, i.e., the moisture content, can be estimated. Specifically, a parameter representing the moisture content contained in the patient's face is calculated based on an image captured by the NIR camera 13. Specifically, a parameter representing the moisture content contained in the patient's face is calculated by analyzing the image captured by the NIR camera 13 using near-infrared spectroscopy. At this time, the effect of uneven illumination in the image may be removed by using a known differential absorption imaging method. For example, when focusing on the absorption peak of water at 980 nm in the wavelength band of 700 nm to 1100 nm, the wavelength band of 700 nm, in which the absorbance of water is sufficiently lower than that at 980 nm and changes with the illumination light intensity, may be removed. In addition, since the camera output signal in each wavelength band may contain noise components due to the dark current of the image sensor itself, the true light intensity may be obtained by subtracting the camera output signal captured by blocking the light from the camera. In this case, for example, A calculated by the following formula diff is calculated as a parameter that represents the analysis result of the moisture content. 980 is the camera output signal strength in the 980 nm wavelength band when the patient's face is photographed, and Idark 980 is the camera output signal strength at 980 nm when the camera is blocked from light, and I 700 is the camera output signal intensity in the 700 nm wavelength band when the patient's face is photographed, and Idark 700 is the camera output signal strength in the 700 nm wavelength band when the camera is shielded from light. Note that the above is just one example of the wavelength band used for imaging, and a nearby wavelength band or another water absorption peak wavelength, such as 1450 nm, may be used.

[0073]

number

[0074] In addition, the facial regions for which the moisture content is analyzed are divided into regions shown in FIG. 11-1 based on the above-mentioned feature points in the patient's face, and parameters representing the moisture content are calculated for each region. The facial regions in FIG. 11-1 are identified based on the feature points detected by the above-mentioned process. As a result of the analysis of each region on the entire face, it has been confirmed that there is a high correlation between the change in the parameter representing the moisture content and the degree of exacerbation of heart failure, particularly in the region of the upper and lower eyelids of the patient and the region of the cheek of the patient shown in FIG. 11-1.

[0075] As mentioned above, patients with heart failure may have facial swelling that increases their facial contours. However, some patients with heart failure do not have a very strong facial swelling, and in this case, the change in the patient's facial contour may be difficult to see. This is also seen in the progression of heart failure, and generally, when the progression of heart failure is mild, the change in the facial contour is difficult to see. On the other hand, as the progression of heart failure becomes more severe, the amount of water in the body increases, and the change in the facial contour becomes more noticeable.

[0076] Basically, fluid retention is often evident in both the front and side of the face. On the other hand, there are patients in whom fluid retention is evident only in the front of the face (in other words, protruding forward). While fluid retention evident in the side of the face can be evaluated by extracting a two-dimensional facial contour, when fluid retention is evident in the front of the face, it may be difficult to estimate the degree of heart failure even if the above-mentioned facial contour-related feature parameters are used.

[0077] Therefore, the heart failure estimation system 10 of the present embodiment extracts parameters related to the water content obtained by the near-infrared analysis, and estimates the state of heart failure of the patient based on the parameters. This makes it possible to measure the water content itself in the target part of the face. That is, even if the water content appears in the lateral direction of the face, even if the water content appears in the front direction of the face, and even if the water content does not yet affect the change in the facial contour, if the water content exists in the face, it is possible to capture the water content. In fact, in some cases of heart failure, it is difficult to judge the state of heart failure based on the facial contour, etc. However, even in such cases, it has been confirmed that the water content obtained by the near-infrared analysis changes and decreases with treatment, and it has been confirmed that it works complementarily with the morphological changes of the face. As described above, the wavelength band of 700 nm to 1100 nm used in the analysis of water content by near-infrared has low absorbance for melanin and hemoglobin, which mainly affect the color of human skin. Therefore, the analysis of water content by near-infrared is not easily affected by differences in skin color due to race. In addition, indoor lighting, mainly fluorescent lamps and LEDs, contains very low or no wavelength components and does not affect the analysis. In order to obtain parameters that represent the results of the near-infrared analysis of moisture content, a marker for normalizing the signal level may be attached to the patient's face or placed near the face. The reflectance of the marker is known, so the signal level can be normalized.

[0078] [6. Parameters representing the 3D shape data of the patient's face] In this embodiment, parameters representing the three-dimensional shape data of the patient's face are used. Specifically, the parameters representing the three-dimensional shape data of the patient's face are calculated based on an image captured by a depth camera (not shown). Of the above feature parameters, when calculating the parameters related to the patient's facial shape, the parameters related to the patient's eyes, and the parameters related to the patient's mouth, the feature points detected by Facemesh, etc. can be simply projected onto a two-dimensional plane to calculate the distance and area between the feature points. On the other hand, if the feature points have a depth direction (Z) as measured by a camera or estimated by Facemesh, etc., they can also be calculated as the distance and area between the feature points in three-dimensional space. Figure 11-2 shows an example of facial feature point detection in three-dimensional space.

[0079] The above characteristic parameters are used in this embodiment. A list of the above characteristic parameters is shown in FIG.

[0080] The above characteristic parameters can be used as numerical values, and the degree of heart failure can also be evaluated as a numerical value. Since the degree of heart failure is calculated as an objective numerical value, it can be used as a skill-less index that transcends the boundaries of specialization and occupations. In addition, the progression of the disease can be evaluated from the rate of change. For example, a patient whose number representing the degree of heart failure increases by three numbers from 2 to 5 in one day is judged to be more rapidly deteriorating than a patient whose number representing the degree of heart failure increases by one number from 1 to 2 in one day. In this way, it is possible to evaluate not only whether the patient's condition is bad or good, but also how quickly the condition is deteriorating or improving.

[0081] The output unit 29 outputs the estimation result estimated by the estimation unit 28. Note that the output unit 29 may output the score itself indicating the degree of heart failure as the estimation result.

[0082] The display device 16 displays the estimation result output from the output unit 29.

[0083] A medical professional or a user operating the heart failure prediction device 14 checks the prediction results displayed on the display device 16 and confirms the possibility that the patient has heart failure.

[0084] The heart failure estimation system 10 of this embodiment is assumed to be used, for example, under a situation as shown in FIG.

[0085] In the example of FIG. 14, a medical worker H such as a doctor holds a tablet terminal which is an example of the heart failure estimation system 10. The medical worker H collects video data of the patient U using a camera (not shown) equipped in the tablet terminal. The tablet terminal then estimates whether the patient U has heart failure or not based on the video data of the patient U, and outputs the estimation result to a display unit (not shown). The medical worker H determines the degree of heart failure of the patient U with reference to the estimation result displayed on the display unit (not shown) of the tablet terminal.

[0086] The heart failure estimation device 14 can be realized by, for example, a computer 50 shown in Fig. 15. The computer 50 includes a CPU 51, a memory 52 as a temporary storage area, and a non-volatile storage unit 53. The computer 50 also includes an input / output interface (I / F) 54 to which an external device, an output device, etc. are connected, and a read / write (R / W) unit 55 that controls reading and writing of data from and to a recording medium. The computer 50 also includes a network I / F 56 that is connected to a network such as the Internet. The CPU 51, the memory 52, the storage unit 53, the input / output I / F 54, the R / W unit 55, and the network I / F 56 are connected to one another via a bus 57.

[0087] The storage unit 53 can be realized by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. The storage unit 53 as a storage medium stores a program for operating the computer 50. The CPU 51 reads the program from the storage unit 53, expands it in the memory 52, and sequentially executes the processes contained in the program.

[0088] [Operation of the heart failure estimation system of the first embodiment]

[0089] Next, a description will be given of a specific operation of the heart failure estimation system 10 of the first embodiment. The heart failure estimation device 14 of the heart failure estimation system 10 executes each process shown in FIG.

[0090] First, in step S100, the acquisition unit 20 acquires a video of the patient captured by the camera 12. Then, the acquisition unit 20 stores the video of the patient in the image data storage unit 22. The acquisition unit may also acquire a video of the patient captured by the NIR camera 13 and store it in the image data storage unit 22.

[0091] Next, in step S102, the extraction unit 26 reads out the moving image stored in the image data storage unit in step S100, and extracts one or more feature parameters from the images at each time that constitute the moving image.

[0092] In step S104, the estimation unit 28 estimates the degree of heart failure of the patient based on the one or more characteristic parameters extracted in step S102 and the computational model stored in the reference data storage unit 24.

[0093] In step S106, the estimation unit 28 outputs the degree of heart failure of the patient estimated in step S104 as a result.

[0094] Then, the output unit 29 outputs the estimation result by the estimation unit 28. The display device 16 displays the estimation result output from the output unit 29. A medical professional or a user operating the heart failure estimation device 14 checks the estimation result output from the display device 16, and the medical professional or the user checks the degree of heart failure.

[0095] As described above, the heart failure estimation device 14 of the heart failure estimation system 10 of the first embodiment extracts one or more feature parameters from an image or video obtained by capturing an image of a patient's face. Then, the heart failure estimation device 14 estimates the degree of heart failure of the patient based on the extracted one or more feature parameters. This makes it possible to estimate the degree of heart failure of the patient from an image or video of the patient's face area.

[0096] Heart failure not only has a poor prognosis, but also has a large number of patients. In addition, heart failure patients suffer from an intractable chronic disease that repeats remission and exacerbation, and the recurrence rate after discharge is high. On the other hand, early intervention in the event of exacerbation can prevent recurrence and progression, so early detection of exacerbation of heart failure is important in the management of heart failure, even if the patient is discharged from the hospital. However, BNP (blood test), chest X-ray, and cardiac ultrasound, which are widely used to quantitatively evaluate heart failure, are mainly performed in hospitals, and interpretation of the results requires expertise. In addition, it is not realistic to perform them frequently due to their invasiveness, cost, and expertise. Other conventional technologies include systems that non-invasively monitor the patient's periodic respiratory variation, heart rate variation, electrocardiogram, etc. using wearable or portable sensors, but all of these technologies require the installation of equipment to estimate the state of heart failure, information collection by contact, implantation into the body, and long-term continuous recording to acquire data. This poses issues such as the effects of body movement, postural changes, and artifacts (human and mechanical errors) due to poor contact, restrictions on body position for data acquisition, and physical burdens.For this reason, currently there is no technology that enables non-contact, non-invasive, simple, and rapid understanding of the pathology of heart failure (for example, early detection of exacerbation) at home or remotely, across the boundaries of specialties and occupations.

[0097] In addition, efforts have been made to remotely monitor weight or vital signs in order to detect early signs of worsening heart failure at home. However, early detection of worsening heart failure has not been successful. The reason for this is that heart failure patients take various drugs that affect hemodynamics (for example, drugs that lower blood pressure or heart rate, or drugs that expel water from the body), making it difficult to detect changes in vital signs (for example, blood pressure or pulse rate). In addition, due to complex pathology or frailty (here, frailty refers to a state in which muscle mass and fat mass are reduced and the patient is emaciated), there are many cases in which muscle or fat mass is reduced even without weight gain, and instead fluid retention such as pleural effusion progresses. For this reason, it has been impossible for patients and medical professionals to notice worsening heart failure before it becomes severe, especially in living environments where access to medical care is limited.

[0098] On the other hand, in everyday clinical practice, even if there are no changes in the patient's vital signs or weight, it is not uncommon to detect poor health from changes in the patient's facial expression and notice worsening heart failure, or conversely, to be convinced that treatment is working based on the degree of improvement in the patient's facial expression.

[0099] The heart failure estimation system of the present embodiment has been developed in consideration of the above circumstances, and estimates the degree of heart failure of a patient based on one or more feature parameters extracted from an image or video obtained by capturing an image of the patient's face. This makes it possible to estimate the degree of heart failure of a patient from an image or video of the patient's face region.

[0100] In this way, information showing changes in the patient's face area is useful for early detection of worsening heart failure. However, the diagnostic skills of medical professionals who capture these changes vary from person to person, and it is not easy to verbalize and communicate them. Therefore, if these changes can be captured as objective digital indicators, as in this embodiment, it is believed that they can be widely used in heart failure treatment as a new, highly versatile, evidence-based, skill-free indicator regardless of specialty, occupation, person, or location.

[0101] The data used in this embodiment is an image or video of the patient's face, which is the most easily recognizable object in daily life. The patient's face is an exposed part even if the patient is dressed or lying in bed. Therefore, it is possible to continuously evaluate the state of heart failure using common indicators, such as facial features, even at home after the patient is discharged from the hospital, in a non-cardiology specialist's outpatient clinic, or in a nursing home, and it is possible to comprehensively perform medical intervention before the worsening of heart failure in a community.

[0102] Furthermore, by accumulating quantitative data on the process of heart failure aggravation after the patient is discharged from the hospital, it is possible to share data on the aggravation of heart failure, the course of treatment, the course of recuperation, and the transition of recuperation at home or in a facility (e.g., a nursing care facility or a rehabilitation facility, etc.) that has not been clear until now. These data can be used as a common index connecting the personal health record and the electronic medical record by evaluating them together with the conventional indexes such as vital signs and activity level, and the conventional heart failure indexes obtained in hospitals and clinics (electronic medical records such as blood tests, chest X-rays, and echocardiographic findings), and can contribute to further elucidation of the pathology of heart failure and development of diagnosis and treatment methods for heart failure. In addition, the index calculated in this embodiment is a continuous variable, and it is possible to objectively visualize daily changes regardless of the evaluator, and it is also possible to evaluate the disease activity of heart failure (e.g., the speed of aggravation and improvement, etc.) from the degree of change.

[0103] In addition, blood sampling, X-rays, and cardiac ultrasound examinations for diagnosing a patient's heart failure require specialized techniques and special examination systems (devices), and there are problems with invasiveness, radiation exposure, examination costs, and expertise, making it difficult to perform them repeatedly. In particular, X-rays and cardiac ultrasound examinations require special examination devices, making it difficult to perform the examinations at clinics or at home. As for blood sampling, most of the examinations are outsourced to external institutions, so the results cannot be confirmed in real time. Furthermore, these and other heart failure evaluation indices require expertise to interpret the results. In contrast, the heart failure estimation system of this embodiment makes it possible to repeatedly obtain the patient's condition in a simple, non-contact, and non-invasive manner. Specifically, the patient's image is captured by a camera in a short time of less than one minute per session, and the results regarding the state of heart failure are automatically determined in real time. In addition, the indices used in this embodiment are based on the pathology of heart failure and are well related to widely used heart failure indices (e.g., NYHA cardiac function classification, BNP value, weight change, degree of pulmonary congestion or pleural effusion on chest X-ray, cardiac ultrasound test results, etc.), and capture facial changes supported by medical evidence, so the results obtained are easy to interpret and explain. Therefore, the physical and mental burden on not only patients but also medical professionals is very light, which is an advantage over other heart failure evaluation indices.

[0104] Furthermore, by combining the heart failure estimation system according to this embodiment with a smartphone app, a smartphone's face recognition function, a PC's face recognition or camera function, an online conference system, an online remote medical system, a home security system, a smart home, a smart home appliance, an in-vehicle camera, a work environment monitoring system, a metaverse, a digital twin, etc., it becomes possible to more easily monitor the pathology of heart failure, which makes it possible to optimize and individualize medical care in the treatment or prevention of heart failure.

[0105] If it becomes possible to evaluate heart failure using a smartphone app, it will be possible to reduce the risk of developing and recurring heart failure in users, and to evaluate the time series of the heart failure state. In addition, the heart failure estimation system according to this embodiment can be incorporated into the life of patients or users as part of daily activities. This will encourage patients or users to change their awareness and behavior, leading to a lifestyle that does not burden heart failure, and it can also be used as a digital therapeutic device or educational tool.

[0106] <Heart failure estimation system according to the second embodiment>

[0107] Next, a second embodiment will be described. Note that, among the configuration of the heart failure estimation system of the second embodiment, the parts having the same configuration as the first embodiment will be given the same reference numerals and the description thereof will be omitted.

[0108] 17 shows a heart failure estimation system 310 according to the second embodiment. As shown in FIG 17, the heart failure estimation system 310 includes a user terminal 18 and a heart failure estimation device 314. The heart failure estimation device 314 further includes a communication unit 30.

[0109] The heart failure estimation device 314 of the heart failure estimation system 310 estimates the degree of heart failure of the user based on an image or video of the user captured by the camera 12 provided in the user terminal 18.

[0110] The heart failure estimation system 310 of the second embodiment is assumed to be used, for example, under the conditions shown in FIGS.

[0111] 18, a medical professional H such as a doctor operates a heart failure estimation device 314, and a user U who is a subject operates a user terminal 18. The user U captures an image or video of himself / herself with the camera 12 of the user terminal 18 which he / she operates. Then, the user terminal 18 transmits the image or video to the heart failure estimation device 314 via a network 19 such as the Internet.

[0112] The heart failure estimation device 314 receives images or videos of the user U transmitted from the user terminal 18. Then, the heart failure estimation device 314 estimates the degree of heart failure of the user U based on the received images or videos, and outputs the estimation result to a display unit 315 of the heart failure estimation device 314. The medical worker H judges the degree of heart failure of the user U by referring to the estimation result displayed on the display unit 315 of the heart failure estimation device 314.

[0113] On the other hand, in the example of FIG. 19, a user U, who is a subject, takes an image or video of himself / herself with the camera 12 of a user terminal 18 operated by the user. Then, the user terminal 18 transmits the image or video to a heart failure estimation device 314 via a network 19 such as the Internet. The heart failure estimation device 314 receives the image or video of the user U transmitted from the user terminal 18. Then, the heart failure estimation device 314 estimates the degree of heart failure of the user U based on the received image or video, and transmits the estimation result to the user terminal 18. The user terminal 18 receives the estimation result transmitted from the heart failure estimation device 314, and displays the estimation result on a display unit (not shown). The user checks the estimation result and confirms the degree of his / her heart failure.

[0114] The heart failure inferring device 314 executes an information processing routine similar to that shown in FIG.

[0115] As described above, the heart failure inferring system of the second embodiment can estimate the degree of heart failure of a user by using the heart failure inferring device 314 installed on the cloud.

[0116] By using the heart failure estimation system of the second embodiment, it becomes possible to estimate the heart failure of a user even outside a hospital. Incidentally, being able to estimate heart failure outside a hospital has many advantages and is of great social significance. For example, a patient who has been diagnosed with heart failure once is likely to have their heart failure worsen or recur outside the hospital (for example, at home, at a hospital transfer, or at a nursing facility) and to be readmitted repeatedly even after being treated in a hospital and discharged. In such a case, if it is possible to detect the symptoms of the patient's worsening heart failure as early as possible and take early measures, it is expected that readmission due to worsening heart failure or progression of heart failure can be prevented. Furthermore, if treatment can be intervened earlier even if heart failure worsens, the patient is more likely to recover quickly even if hospitalized. Furthermore, such a technology may be applicable to early detection and management of a serious disease such as heart failure even when it is difficult to visit a hospital, such as during a disaster or an infectious disease epidemic. In general, in order to detect heart failure, it is necessary to monitor the state of heart failure including the patient's blood pressure, pulse, blood sampling, X-ray, and cardiac ultrasound examination results, but it is difficult to monitor the state of the patient sequentially outside the hospital and in a situation where medical and human resources are limited. In contrast, the heart failure estimation system of this embodiment allows non-medical personnel, regardless of their specialty or occupation, to repeatedly monitor the state of heart failure easily, inexpensively, non-invasively, and non-contact. In addition, since the evaluation items used in this system are composed of multiple indices that are not easily affected by racial differences such as skin color and age, it can be used on a global scale regardless of age or gender. In addition, in recent years, the treatment results of congenital heart disease have improved, and the number of patients with congenital heart disease who reach adulthood is increasing. Even in such patients, the heart failure estimation system of this embodiment can continuously monitor the state of heart failure from childhood to adulthood.

[0117] In addition, according to the heart failure estimation system of the present embodiment, the degree of the user's heart failure can be calculated based on the user's face image or video, so that the user can check the degree of heart failure by himself / herself even outside the hospital, such as at home, a nursing facility, or an evacuation site. Furthermore, by using the heart failure estimation system of the present embodiment, even doctors other than doctors specializing in cardiology, other medical professionals (pharmacists, nurses, physical therapists, etc.), or caregivers can determine the user's heart failure. This allows early detection of changes in the degree of the patient's heart failure. Furthermore, since the heart failure estimation system of the present embodiment can easily determine the state of heart failure regardless of place or time, it can be used as an educational tool to improve the user's health literacy and deepen the knowledge of many medical professionals and caregivers involved in the treatment of heart failure by feeding back the results as so-called "teacher data" or evaluating the results together with the user's symptoms, physical findings, and test findings. EXAMPLES

[0118] Next, an embodiment will be described. In this embodiment, a parameter representing the shape of the patient's face, a parameter representing the patient's face vector, a parameter representing the patient's eyes, and a parameter representing the patient's mouth are used to estimate heart failure using a decision tree algorithm called LightGBM. Figure 20 shows an example of the results of estimating the degree of heart failure of a patient using LightGBM.

[0119] The horizontal axis (X-axis) of the graph in Figure 20 represents the value of the New York Heart Association (NYHA) cardiac function classification diagnosed by a doctor and recorded in the medical record, and the vertical axis (Y-axis) of the graph represents the estimated value of the NYHA cardiac function classification estimated by the trained LightGBM.

[0120] The NYHA cardiac function classification (excerpt from the 2017 revised edition of the Joint Guidelines of the Japanese Circulation Society and the Japanese Heart Failure Society: Guidelines for the Treatment of Acute and Chronic Heart Failure) is shown below.

[0121] NYHA1: Cardiac disease is present, but physical activity is not limited. Ordinary physical activity does not cause significant fatigue, palpitations, dyspnea, or angina. NYHA2: Mild to moderate limitation of physical activity. Asymptomatic at rest. Ordinary physical activity causes fatigue, palpitations, dyspnea, or angina. NYHA3: Severe physical activity is limited. No symptoms at rest. Exercise less than normal physical activity causes fatigue, palpitations, dyspnea, or angina. NYHA4: Any physical activity is restricted due to heart disease. Symptoms of heart failure and angina are present even at rest. These symptoms worsen with the slightest exertion.

[0122] Specifically, the horizontal axis (X-axis) of the graph in FIG. 20 indicates the value of the NYHA functional classification at the time of data acquisition of the case evaluated by a cardiologist, and the vertical axis (Y-axis) indicates the estimated value of the NYHA functional classification output from the trained LightGBM. Normally, the NYHA functional classification is classified into the above four categories, but in daily clinical practice, the state of heart failure of a patient cannot be clearly classified into four categories, and it may be evaluated as NYHA1-2 or NYHA2-3. In this case, for convenience, NYHA1-2 is converted to 1.5, NYHA2-3 to 2.5, and NYHA3-4 to 3.5 for analysis. For this reason, the horizontal axis (X-axis) of the graph in FIG. 20 is a value in 0.5 increments. On the other hand, the estimated value of the NYHA functional classification on the vertical axis (Y-axis) of the graph in FIG. 20 is displayed as a decimal value because it is output as a continuous variable from the trained LightGBM.

[0123] The left side of Fig. 20 shows the results when LightGBM is trained using data other than a specific patient case (G-HF023) as training data, and the trained LightGBM is used to estimate the NYHA cardiac function classification value of the training data. On the other hand, the graph on the right side of Fig. 20 shows the results when the trained LightGBM is used to estimate the NYHA cardiac function classification value of a specific patient case (G-HF023) using the data of the specific patient case (G-HF023) as test data. Fig. 20 also shows the mean absolute percentage error (mape).

[0124] As shown in the right panel of Figure 20, in this patient's case (G-HF023), although the magnitude of the values ​​differed, it can be seen that the progression of the severity of heart failure could be estimated with the NYHA cardiac function classification with an extremely high correlation coefficient of R = 0.99.

[0125] In addition, Figure 21 is a graph showing the importance of feature parameters obtained when training LightGBM. This shows that many parameters related to the shape of the face, mouth, and eyes, as well as near-infrared analysis, contribute to the estimation of NYHA cardiac function classification. In addition, since there are individual differences in which feature parameters are likely to change for each patient, further accuracy improvement can be expected by changing the weighting of each feature parameter according to facial features.

[0126] On the other hand, FIG. 22-1 shows an example of a statistical model by multivariate analysis for the NYHA cardiac function classification using data from 37 patients (case numbers G-HF002 to G-HF038, which will be described later). In this statistical model, the NYHA cardiac function classification is calculated using four factors: emdiff_dod, W80fcnt, Pitch, and Area_all. The NYHA prediction formula was calculated to predict the NYHA function classification. The horizontal axis (X axis) of the graph in Figure 22-1 shows the NYHA function classification at the time of data acquisition of the case evaluated by a cardiologist, and the vertical axis (Y axis) shows the NYHA function classification estimated from the NYHA prediction formula, which is a statistical model. The four figures below the NYHA prediction formula show the effect of embdiff_dod, W80fcnt, Pitch, and Area_all on NYHA prediction (referred to as prediction effect in Figure 22-1) from the left. The correlation diagram below shows the result of verifying the correlation between the NYHA actual value (X axis) and the NYHA predicted value (Y axis) in the case of G-HF001, which was not used to create the NYHA prediction formula, using the obtained NYHA prediction formula. As shown in the figure, the prediction formula calculated from the multivariate analysis model can predict the NYHA function classification with a strong correlation coefficient of R = 0.88.

[0127] Figures 22-2 and 22-3 show graphs showing the importance of the feature parameters (note that parameters related to complexion are excluded) obtained when LightGBM was trained using facial moisture content obtained from near-infrared analysis, as well as the results of estimating the NYHA cardiac function classification using data from a specific patient's case (G-HF017) as test data. The characteristic parameters in FIG. 22-2 are as follows: JPEG0007672033000005.jpg6283 This shows that facial shape, eye and mouth parameters, and hydration parameters all contribute widely to the estimation of the NYHA cardiac function classification, and as shown in Figures 20 and 21, the progression of the severity of heart failure can be estimated with an extremely high correlation coefficient of R=0.97. Furthermore, the straight line shown in Figure 22-4 corresponds to the prediction formula calculated from the statistical model shown in the following formula. NYHA prediction formula = 4.21+0.083*(Area: face area)-0.23*(EAR: degree of eye opening)+0.21*(m_outline: aspect ratio of mouth)+0.00075*(m_grad: corners of mouth)+0.025*(pitch: face direction)+0.42*(w80fcnt: time eyes are closed)-2.9*(w149: face width)-5.5*(Adiff: hydration)+4.4*(embdiff_dod: face vector) When constructing the above NYHA prediction equation, the data from G-HF012 to G-HF060 were used to create a prediction equation from data other than G-HF017, and it was verified using the data from G-HF017. As a result, a strong correlation was found between the actual NYHA functional classification values ​​and the predicted values ​​(correlation coefficient R = 0.97). FIG. 23 is a diagram showing a comparison between a subjective assessment of the degree of heart failure by a specialist and an estimation result of the degree of heart failure when a uniform determination threshold is set for each characteristic parameter.

[0128] The first, second, and third columns from the left in the "Subjective Evaluation" column in Figure 23 are the results of visually determining whether there were any changes in the facial contour, eyes, and cheeks between the time the patient was hospitalized due to heart failure and the time he or she was discharged. Changes are recorded as "1," no changes (or it is unclear whether there were any changes) are recorded as "0." If any changes were observed, a "1" is recorded in the "OR for each item" in the fourth column from the left in the "Subjective Evaluation Items" column, and if none were observed, a "0" is recorded.

[0129] The left column of the "Area_all" column in Figure 23 shows the amount of change in the parameter that represents the area of ​​the patient's face between hospitalization and discharge due to heart failure, and the right column shows the judgment result regarding the presence or absence of change when a threshold value (=0.06) is set for that. Specifically, if it is equal to or greater than the set threshold, it is written as "changed '1'", and if it is less than the threshold, it is written as "no change '0'" (see embdiff_dod (The same definition is used for

[0130] The left column of the "embdiff_dod" column in Fig. 23 shows the results of the parameter (embdiff_dod) that represents the change in the patient's face vector, and the right column shows the judgment result regarding the presence or absence of change when a threshold value (=0.29) is set for that. In the "EAR" column in Fig. 23, if the parameter representing the aspect ratio of the patient's eyes has increased at the time of discharge compared to when hospitalized for heart failure, it is entered as "1", and if it has decreased or remained unchanged, it is entered as "0".

[0131] The "Adiff correlation (weight)" and "Adiff correlation (BNP)" columns in Figure 23 are the results of threshold judgment for the correlation coefficient between the parameters representing the results of near-infrared analysis of the water content of the patient's face and the patient's weight and BNP value (if near-infrared analysis has not been performed, it is left blank). A correlation coefficient of 0.6 or greater is entered as "1", and a correlation coefficient less than 0.6 is entered as "0". The "Detection accuracy" column is the sum (maximum value = 5) of the threshold judgment results for the above (Area_all, embdiff_dod, EAR, Adiff correlation (weight), and Adiff correlation (BNP)), and the higher the value, the more changes have been detected in the feature parameters before and after heart failure treatment.

[0132] As shown in FIG. 23, it is found that the subjective evaluation and the judgment results based on each characteristic parameter are consistent in many cases. Furthermore, it is also clear that each characteristic parameter is sensitive to detect changes associated with heart failure that cannot be detected by subjective evaluation. For example, in G-HF002, G-HF004, G-HF010, G-HF011, G-HF014, G-HF019, G-HF020, G-HF022, G-HF024, and G-HF025, the facial changes associated with heart failure cannot be detected visually at all, but significant changes are detected in multiple items among the correlations of Area_all, embdiff_dod, EAR, and Adiff (FIG. 23). Furthermore, it is also clear that in these cases, some of the characteristic parameters have a strong correlation with NYHA cardiac function classification, body weight, and BNP (brain natriuretic peptide) (FIGS. 24 to 26, which will be described later). As a result, this heart failure estimation system is not only able to objectively support the subjective assessment of heart failure by cardiologists that can be seen with the naked eye, but is also able to detect changes that cannot be seen with the naked eye from multiple angles using many parameters. If we were to quantitatively indicate the accuracy, 100% of changes that doctors can recognize can be captured by one of the feature parameters, and even for slight signs that the human eye cannot detect, it is possible to detect them without missing them using one of the feature amounts, although the false positives will increase by adjusting the sensitivity.

[0133] 24 to 26 are diagrams showing the correlation between each characteristic parameter and the NYHA cardiac function classification, body weight, and BNP (brain natriuretic peptide), which are widely used as indexes indicating the degree of heart failure.

[0134] Figure 24 shows the correlation between the NYHA cardiac function classification and each characteristic parameter. The leftmost column lists the case number, and the third to eleventh columns from the left list the correlation coefficient between each characteristic parameter and the NYHA cardiac function classification. Blank spaces indicate parameters that have not been measured. Cases where a relationship with the state of heart failure is theoretically recognized in the correct direction and the correlation coefficient is moderate or higher (a correlation coefficient of "0.6 or higher" for a positive correlation, and a correlation coefficient of "-0.6 or lower" for a negative correlation) are shaded. For example, the width and area of ​​the face theoretically increase with the worsening of heart failure, and are therefore considered to show a positive correlation. Based on this, the sum of the number of characteristic parameters with a correlation coefficient of moderate or higher is listed as the score in the second column from the left. For example, in case G-HF001, eight parameters, namely the face vector parameter, the mouth parameter, two eye parameters, the face direction, the two face width parameters, and the face area parameter, showed a high correlation with the NYHA cardiac function classification with a correlation coefficient of 0.6 or more, and "8" is entered in the score column as the sum of the characteristic parameters with a correlation coefficient of 0.6 or more.

[0135] Similarly, Fig. 25 shows the correlation between body weight and each characteristic parameter, and Fig. 26 shows the correlation between BNP and each characteristic parameter and the sum of each characteristic parameter with a high correlation coefficient. Blank spaces indicate that the parameter was not measured and could not be evaluated.

[0136] As shown in Figures 24 to 26, many of the characteristic parameters have a moderate to strong correlation with the NYHA cardiac function classification, body weight, and BNP. Furthermore, it is noteworthy that even in cases where the total sum of correlation coefficients is low, which is moderate or higher, there are cases where there is a strong correlation with one characteristic parameter, and this heart failure estimation system, which evaluates using multiple parameters, is effective in reducing the risk of overlooking the worsening of heart failure. For example, in case G-HF036, the correlation between the parameters related to the face shape, mouth, eyes, face direction, and face area / width and the NYHA cardiac function classification, body weight, and BNP is less than moderate, but the near-infrared parameters have a "strong" correlation with the NYHA, body weight, and BNP, and the system has succeeded in accurately capturing the state of heart failure (or not overlooking the worsening of heart failure). Similarly, in case G-HF013, the feature parameters related to the mouth, eye and face direction, near infrared, and face area and width were less than moderately correlated with the NYHA cardiac function classification and body weight, but the face vector parameters were "strongly" correlated with these. In this way, this heart failure estimation system uses multiple feature parameters that sensitively reflect the state of heart failure, and each works complementarily to accurately grasp the degree of heart failure. Figures 27-1 to 27-3 are diagrams showing the correlation between each characteristic parameter calculated using the 3D shape data of the face as shown in Figure 11-2 and the NYHA cardiac function classification, weight, and BNP, while increasing the number of cases from 30 shown in Figures 24 to 26 to 60. Note that blank spaces indicate cases where the parameters have not been measured and therefore cannot be evaluated. As shown in Figures 27-1 to 27-3, even when the number of cases is increased, it is presumed that there is a correlation between each characteristic parameter and the NYHA cardiac function classification, weight, and BNP. Furthermore, it has been confirmed that these characteristic parameters are also related to the degree of congestion and pleural effusion on chest X-rays, which are established as evaluation indicators of heart failure, and the results of cardiac ultrasound examinations. For example, in the case of G-HF009, the pulmonary congestion score and pleural effusion score shown on the X-axis decreased (the lower the score, the more improved the congestion and pleural effusion, and 0 is a state in which heart failure is compensated for without any congestion or pleural effusion), while the NYHA cardiac function class, which is an indicator of heart failure symptoms shown on the Y-axis, improved (the lower the score, the more mild the symptoms, and NYHA cardiac function class I is a state in which heart failure is compensated and there are no symptoms) (top of Figure 27-4). Similarly, as the pulmonary congestion score and pleural effusion score shown on the X-axis decreased (improved), the EAR, which is an indicator of eye opening, increased (improved), and the face width decreased (swelling improved) (middle and bottom of Figure 27-4). The correlation coefficient is moderate or higher. Similarly, the pulmonary congestion score and pleural effusion score decreased (improved), while Adiff, a near-infrared index representing water volume, increased (water volume decreased), and embdiff_dod, an index of change in facial vectors, decreased (heart failure improved) (Figure 27-5). Furthermore, it has been confirmed that each characteristic parameter has a moderate or higher correlation with the results of echocardiography. For example, Figure 27-6 shows the correlation between embdiff_dod, an index of the change in face vector shown on the Y axis, EAR, an index of the degree of eye opening, and the E / A ratio and IVC (inferior vena cava: diameter of the inferior vena cava, mm) of echocardiography shown on the X axis in the case of G-HF017. E / A is an index of cardiac diastolic function and is calculated from the blood flow velocity waveform (TMF) of the mitral valve orifice. The E wave is an early diastolic wave (E wave) generated by active blood inflow from the left atrium to the left ventricle when the mitral valve opens. The A wave is an atrial systolic wave (A wave) generated by blood inflow from the left atrium to the left ventricle due to atrial contraction in late diastole. Generally, in the TMF of middle-aged and elderly people, the E / A ratio is 1 or less, but when heart failure worsens, natural blood inflow becomes difficult due to left ventricular diastolic dysfunction, and the proportion of blood inflow due to atrial contraction increases, resulting in a greater increase in the A wave (a greater decrease in the E / A ratio). In contrast, if the condition of heart failure is improved by heart failure treatment, the cardiac diastolic function also improves, the proportion of blood naturally flowing from the left atrium to the left ventricle increases, and the proportion of assistance from atrial contraction decreases, resulting in an increase in the E / A ratio. On the other hand, the IVC diameter indicates the degree of water retention in the body, especially in the blood vessels, and increases when heart failure worsens and decreases when heart failure improves. In the G-HF017 case, the E / A ratio of the TMF increases and the IVC diameter decreases due to the improvement of heart failure, but in accordance with this, embdiff_dod, which is an index of change in face vectors, decreases (improves), and EAR, which is an index of eye opening, increases (improves). In addition, it has been confirmed that the feature parameters obtained from facial images and videos are correlated with E / e' (mitral valve E wave velocity divided by mitral annular e' velocity) and tricuspid regurgitation velocity obtained by cardiac ultrasound examination. In this way, each feature parameter obtained from facial images and videos has been established as an evaluation index for heart failure, and it has been confirmed that it is correlated with medically proven heart failure symptoms (NYHA functional class), weight gain / edema, blood test findings (BNP value), chest X-ray findings, and cardiac ultrasound findings. As an index specific to heart failure, the obtained results can be medically explained, and will also contribute to the construction of explainable AI. Therefore, the present disclosure provides a heart failure estimation device, a heart failure estimation system, a heart failure estimation method, and a heart failure estimation program that can estimate the degree of heart failure of a patient from an image or video of the patient's face region in a non-contact, non-invasive, simple, and short time, and can solve the problems of the conventional techniques, such as the need to wear equipment and surgery required for measurement, the problem of artifacts caused by body movement, body position, and contact, and the physical and economic burden caused by monitoring. In addition, according to the present disclosure, since the degree of heart failure of a patient can be estimated from an image or video of the patient's face region, it is possible to obtain the effect of eliminating the need to wear equipment for measurement, avoiding measurement and evaluation errors caused by body movement, body position, and contact, and reducing the burden on the patient at the same time. Furthermore, the present disclosure obtains the effect of sensitively detecting changes in condition associated with heart failure that cannot be captured by conventional heart failure indicators by converting subjective evaluations such as the "notice and intuition" of a skilled doctor into an objectively assessable index.

[0137] The technology of the present disclosure is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the disclosure.

[0138] In the above embodiment, a case where a plurality of characteristic parameters are used has been described as an example, but the present invention is not limited to this. For example, the degree of heart failure may be estimated using at least one of the plurality of characteristic parameters.

[0139] Furthermore, the trained model or statistical model used to estimate the degree of heart failure may be a model prepared in advance for each patient attribute (such as by age, gender, or disease). For example, trained models or statistical models for men in their twenties, women in their twenties, men in their thirties, patients with respiratory diseases, and patients undergoing maintenance dialysis are prepared in advance. For example, if the patient is a man in his twenties, the trained model or statistical model for men in their twenties is used to estimate the degree of heart failure. This allows the degree of heart failure to be estimated with high accuracy according to the patient's attributes.

[0140] The degree of aggravation of heart failure of a patient may be estimated based on time series data of a score representing the degree of heart failure or time series data of one or more characteristic parameters.

[0141] For example, Figure 27-7 shows an example of the time course of near-infrared parameters related to the worsening of heart failure after discharge. In the example presented, the near-infrared parameter (Adiff), which indicates the amount of water, significantly decreased due to heart failure treatment from the time of admission to the time of discharge, and regarding the state of heart failure, the weight decreased from 78 kg at admission to 57 kg at the time of discharge, the BNP value, which is blood sampling data indicating the degree of heart failure, decreased from 1788 pg / ml at admission to 282 pg / ml at the time of discharge, and the NYHA cardiac function classification, which is an index of heart failure symptoms, improved from NYHA 4 to 1, and the patient was discharged. However, during the outpatient clinic after discharge, the heart failure worsened again, the weight increased to 63 kg (+7 kg), the BNP value increased again to 1494 pg / ml, the NYHA cardiac function classification was also 1-2, and mild shortness of breath was observed. At this time, Adiff, an index of the amount of water obtained from the face image, also increased again, confirming that the worsening of heart failure was accurately captured. After that, the patient was given additional treatment for heart failure, and at the next outpatient visit, all of the heart failure indicators (weight, BNP value, NYHA cardiac function classification) improved, and Adiff, an indicator of water volume, was also confirmed to have decreased. After that, the state of heart failure stabilized, and Adiff, an indicator of water volume, was also confirmed to have remained stable. Similarly, with regard to embdif_dod, a parameter of the face vector, and EAR, an indicator of eye shape, as heart failure improved with hospitalization, embdiff_dod (left side of the Y axis in the figure, square mark) decreased (since the time of discharge is used as the reference, the difference between the face vector observed at the time of discharge and at the time of admission improved), and EAR (right side of the Y axis in the figure, circle mark) increased (eye opening improved), and when heart failure worsened during the outpatient visit, emdiff_dod increased again (the difference from the face vector at the time of discharge reappeared), and EAR decreased (eye opening decreased again) (Figure 27-8). In this way, the characteristic parameters used in this heart failure estimation system have been shown to sensitively detect not only changes in heart failure during hospitalization, but also changes in heart failure in outpatient care. On the other hand, when heart failure suddenly worsens, there is also intestinal edema and loss of appetite, and the nutritional status declines insidiously, and muscle and fat mass often decrease and are replaced by water instead. For this reason, if fluid removal is promoted through treatment, the patient will lose a large amount of weight at the time of discharge, and the so-called malnutrition state will become apparent (sunken cheeks and a thin face). If heart failure is subsequently managed, fluid retention will not be recognized and muscle and fat will return, improving the nutritional status. This system is capable of capturing not only changes in the state of heart failure that reflect changes in so-called "water content" due to fluid retention during the worsening of heart failure and fluid removal during the acute phase of heart failure treatment, but also complex changes in the state of heart failure, such as improvements in nutritional status as heart failure progresses. For example, in the graph of the change in face vector and the transition of the parameters (embdiff_dod and EAR) representing the degree of eye opening shown in Figure 27-8, the change (increase in water retention) is detected and changes sensitively when the heart failure worsens shortly after discharge, but from the remission stage to the chronic / maintenance stage, the face shape changes differently from when the patient was discharged, reflecting the improvement in nutritional status (change from malnutrition to a "plump" state), and is maintained. In fact, the above Adiff, which represents the amount of water in the face, reflects the remission of heart failure and then remains stable, confirming that the change in face shape observed from the chronic to maintenance stage is not due to water retention associated with heart failure, but is due to the improvement and change in nutritional status. In this way, by skillfully combining and interpreting multiple feature parameters, it is possible to evaluate not only the state of heart failure, but also frailty and nutritional status, which are closely related to heart failure. FIG. 27-9 is an example of the time-dependent transition of Adiff, which represents the facial moisture amount, and EAR. FIG. 27-10 is an example of the time-dependent transition of Adiff, which represents the facial moisture amount, and embdiff_dod, which represents the facial vector change. FIG. 27-11 is an example of the time-dependent transition of Adiff, which represents the facial moisture amount, and face width w14-9. As can be seen from FIG. 27-9, FIG. 27-10, and FIG. 27-11, the time-dependent transition of Adiff, which represents the facial moisture amount, and the time-dependent transition of each feature parameter are related to each other.

[0142] Similarly, as shown in FIG. 28, a graph may be created in which the horizontal axis (X-axis) represents the date and time, and the vertical axis (Y-axis) represents the score representing the degree of heart failure. In this case, the smaller the value on the vertical axis (Y-axis), the greater the severity (degree of aggravation), and the steeper the slope, the stronger (or faster) the disease progression (speed of aggravation). For example, a score representing the degree of heart failure of patient A, whose degree of aggravation of heart failure is small and whose progression is gradual, is shown by a solid line, and a score representing the degree of heart failure of patient B, whose condition worsened early, is shown by a dashed line. Such a time series graph may be generated, and the degree of aggravation of the patient's heart failure may be estimated based on the graph. Alternatively, the value of the characteristic parameter may be plotted on the vertical axis (Y-axis), and the degree of aggravation of the patient's heart failure may be estimated from the time series data. In particular, the parameters used in this heart failure prediction system are continuous variables, and it is possible to predict the momentum of the disease progression, such as aggravation or improvement of heart failure, from the degree of change within a day or between days. For example, embdiff_dod in Figure 6 shows the daily change in value over time, and the magnitude of the slope makes it possible to grasp how quickly heart failure is worsening or improving. By taking this information into account, it is possible to more appropriately warn of worsening heart failure.

[0143] In the above embodiment, various machine learning models such as Facemesh, FaceNet, and LightGBM are used, but the present invention is not limited to this. Other models may be used in addition to the machine learning models. For example, other applicable regression models may include Ridge regression, Lasso, Elastic-Net, Least-angle regression (LARS), LARS Lasso, Orthogonal Matching Pursuit (OMP), Bayesian Regression, Logistic regression, or RandomForest regression.

[0144] In addition, in the above embodiment, an example was described in which 468 feature points were extracted from a facial image using Facemesh, but this is not limited to this and any number of feature points may be extracted from a facial image.

[0145] For example, although the present specification has been described as an embodiment in which a program is pre-installed, the program may also be provided by being stored on a computer-readable recording medium.

[0146] In the above embodiment, the process executed by the CPU after reading the software (program) may be executed by various processors other than the CPU. In this case, examples of the processor include a PLD (Programmable Logic Device) such as an FPGA (Field-Programmable Gate Array) whose circuit configuration can be changed after manufacture, and a dedicated electric circuit such as an ASIC (Application Specific Integrated Circuit) which is a processor having a circuit configuration designed exclusively for executing a specific process. Alternatively, a GPGPU (General-purpose graphics processing unit) may be used as the processor. Each process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, a plurality of FPGAs, a combination of a CPU and an FPGA, etc.). The hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0147] In the above-described embodiments, the program is stored (installed) in advance in the storage device. However, the present invention is not limited to this. The program may be provided in a form stored in a non-transitory storage medium such as a Digital Read Only Memory (DVD-ROM), a Digital Versatile Disk Read Only Memory (DVD-ROM), or a Universal Serial Bus (USB) memory. The program may also be provided in a form downloaded from an external device via a network.

[0148] Furthermore, each process of the present embodiment may be implemented by a computer or server equipped with a general-purpose processor and a storage device, and each process may be executed by a program. This program is stored in a storage device, and may be recorded on a recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or may be provided through a network. Of course, any other components do not have to be implemented by a single computer or server, and may be distributed and implemented on multiple computers connected by a network.

[0149] (Additional Note) The following additional notes are given regarding aspects of the present disclosure. (Appendix 1) An acquisition unit that acquires an image or video obtained by capturing an image of a patient's face; an extraction unit that extracts one or more feature parameters from the image or video acquired by the acquisition unit; an estimation unit that estimates a degree of heart failure of the patient based on the one or more characteristic parameters extracted by the extraction unit; an output unit that outputs the result estimated by the estimation unit; A heart failure estimation device comprising: (Appendix 2) The estimation unit is Based on the one or more feature parameters extracted by the extraction unit and a trained model that has been machine-learned in advance, the one or more feature parameters are input to the trained model to obtain a score that represents a degree of heart failure of the patient output from the trained model, and the degree of heart failure of the patient is estimated based on the score. 2. A heart failure estimation device as described in appendix 1. (Appendix 3) The estimation unit is based on the one or more characteristic parameters extracted by the extraction unit and a statistical model generated in advance, the one or more characteristic parameters are input as explanatory variables of the statistical model, thereby acquiring a target variable obtained from the statistical model as a score representing a degree of heart failure of the patient, and estimating the degree of heart failure of the patient based on the score; 2. A heart failure estimation device as described in appendix 1. (Appendix 4) The characteristic parameters are: the parameters including at least one of a parameter relating to the shape of the patient's face, a parameter relating to the complexion of the patient, a parameter relating to the patient's eyes, a parameter relating to the patient's mouth, a parameter representing a result of a near-infrared analysis of the amount of moisture in the patient's face, and a parameter representing three-dimensional shape data of the patient's face. The heart failure estimation device according to any one of Supplementary notes 1 to 3. (Appendix 5) The parameters relating to the shape of the patient's face are: at least one of a parameter representing an area of ​​the face, a parameter representing a lateral width of the face, a parameter representing a direction of the face, and a parameter representing a change in a face vector representing the facial features; 5. A heart failure estimation device according to claim 4. (Appendix 6) The parameters relating to the complexion of the patient are: At least one of a parameter representing a hue of the patient's face and a parameter representing a degree of greenness of the patient's face, 5. A heart failure estimation device according to claim 4. (Appendix 7) The patient's eye parameters include At least one of a parameter representing the aspect ratio of the patient's eyes, a parameter representing the difference between the aspect ratios of the patient's left and right eyes, a parameter representing the duration of eye closure in one blink, a parameter representing the rate at which the eyes are closed per unit time, a parameter representing the interval between blinks, a parameter representing the number of blinks per unit time, a parameter representing the speed at which the eyes are closed, a parameter representing the speed at which the eyes are opened, a parameter representing the length of time the eyes are closed, a parameter representing the average speed at which the eyes are opened and closed, and a parameter representing the vertical orientation of the face, 5. A heart failure estimation device according to claim 4. (Appendix 8) The parameters relating to the patient's mouth are: At least one of a parameter representing an aspect ratio of the patient's mouth, a parameter representing an opening of the patient's mouth, a parameter representing a left corner of the mouth, and a parameter representing a right corner of the mouth. 5. A heart failure estimation device according to claim 4. (Appendix 9) The trained model is a trained model trained in advance for each patient attribute, 3. A heart failure estimation device as described in appendix 2. (Appendix 10) The statistical model is a statistical model generated in advance for each patient attribute. 4. A heart failure estimation device according to claim 3. (Appendix 11) The estimation unit is estimating a degree of exacerbation of heart failure of the patient based on the time series data of the score or the time series data of the one or more characteristic parameters; 4. A heart failure estimation device according to claim 2 or 3. (Appendix 12) The extraction unit extracts feature points representing positions of each part in the face from the image or video acquired by the acquisition unit, and extracts the one or more feature parameters based on the feature points. 4. A heart failure estimation device according to claim 2 or 3. (Appendix 13) A heart failure inference system including a user terminal equipped with a camera and the heart failure inference device according to any one of Supplementary Note 1 to Supplementary Note 3, the user terminal transmits the image or the video captured by the camera to the heart failure estimation device; The acquisition unit of the heart failure estimation device acquires the image or the video transmitted from the user terminal, the estimation unit of the heart failure estimation device estimates a degree of heart failure of the patient based on the image or the video. Heart failure estimation system. (Appendix 14) The characteristic parameters are: The parameters include parameters related to the patient's facial shape. The heart failure estimation device according to any one of Supplementary notes 1 to 3. (Appendix 15) The characteristic parameters are: The parameters include parameters related to the complexion of the patient. The heart failure estimation device according to any one of Supplementary notes 1 to 3. (Appendix 16) The characteristic parameters are: The parameters include parameters related to the patient's eye. The heart failure estimation device according to any one of Supplementary notes 1 to 3. (Appendix 17) The characteristic parameters are: parameters including parameters related to the patient's mouth; The heart failure estimation device according to any one of Supplementary notes 1 to 3. (Appendix 18) The characteristic parameters are: The parameters include parameters representing the results of a near-infrared analysis of the moisture content of the patient's face. The heart failure estimation device according to any one of Supplementary notes 1 to 3. (Appendix 19) The characteristic parameters are: The parameters include parameters representing three-dimensional shape data of the patient's face. The heart failure estimation device according to any one of Supplementary notes 1 to 3. (Appendix 20) Acquire an image or video obtained by capturing an image of the patient's face; Extracting one or more feature parameters from the captured images or video; estimating a degree of heart failure of the patient based on the one or more extracted characteristic parameters. A heart failure estimation method in which processing is performed by a computer. (Appendix 21) Acquire an image or video obtained by capturing an image of the patient's face; Extracting one or more feature parameters from the captured images or video; estimating a degree of heart failure of the patient based on the one or more extracted characteristic parameters. A heart failure estimation program for causing a computer to execute the processing.

[0150] The disclosure of Japanese Patent Application No. 2023-019491, filed on February 10, 2023, is incorporated herein by reference in its entirety. All documents, patent applications, and technical standards described herein are incorporated herein by reference to the same extent as if each individual document, patent application, and technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. An acquisition unit that acquires an image or video obtained by capturing an image of a patient's face; an extraction unit that extracts one or more feature parameters from the image or video acquired by the acquisition unit; an estimation unit that estimates a degree of heart failure of the patient based on the one or more characteristic parameters extracted by the extraction unit; an output unit that outputs the result estimated by the estimation unit; Including, The feature parameters include parameters related to the shape of the patient's face, parameters related to the patient's eyes, parameters related to the patient's mouth, and near-infrared water absorption parameters for the patient's face. The parameters include at least one of a parameter representing a result of a quantity analysis and a parameter representing three-dimensional shape data of the patient's face. Heart failure estimation device.

2. The estimation unit is Based on the one or more feature parameters extracted by the extraction unit and a trained model that has been machine-learned in advance, the one or more feature parameters are input to the trained model to obtain a score that represents a degree of heart failure of the patient output from the trained model, and the degree of heart failure of the patient is estimated based on the score. The heart failure prediction device according to claim 1 .

3. The estimation unit is based on the one or more characteristic parameters extracted by the extraction unit and a statistical model generated in advance, the one or more characteristic parameters are input as explanatory variables of the statistical model, thereby acquiring a target variable obtained from the statistical model as a score representing a degree of heart failure of the patient, and estimating the degree of heart failure of the patient based on the score. The heart failure prediction device according to claim 1 .

4. The characteristic parameters are: The parameters further include a parameter related to the complexion of the patient. The heart failure estimation device according to any one of claims 1 to 3.

5. The parameters relating to the shape of the patient's face are: At least one of a parameter representing an area of ​​the face, a parameter representing a width in a lateral direction of the face, a parameter representing a direction of the face, and a parameter representing a change in a face vector representing the features of the face. Also includes one The heart failure estimation device according to any one of claims 1 to 4.

6. The parameters relating to the complexion of the patient are: At least one of a parameter representing a hue of the patient's face and a parameter representing a degree of greenness of the patient's face, The heart failure prediction device according to claim 4 .

7. The patient's eye parameters include A parameter representing the aspect ratio of the patient's eyes, a parameter representing the difference between the aspect ratios of the patient's left and right eyes, a parameter representing the duration of eye closure in one blink, a parameter representing the duration of eye closure per unit time, A parameter that represents the proportion of blinks, a parameter that represents the blink interval, a parameter that represents the number of blinks per unit time, a parameter that represents the speed at which the eyes are closed, and a parameter that represents the speed at which the eyes are opened. a parameter representing the length of time the eyes are closed, a parameter representing the average speed of opening and closing the eyes, and a parameter representing the vertical orientation of the face, The heart failure estimation device according to any one of claims 1 to 6.

8. The parameters relating to the patient's mouth are: At least one of a parameter representing an aspect ratio of the patient's mouth, a parameter representing an opening of the patient's mouth, a parameter representing a left corner of the mouth, and a parameter representing a right corner of the mouth. 、 The heart failure estimation device according to any one of claims 1 to 7.

9. The trained model is a trained model trained in advance for each patient attribute, The heart failure prediction device according to claim 2 .

10. The statistical model is a statistical model generated in advance for each patient attribute. The heart failure prediction device according to claim 3 .

11. The estimation unit is estimating a degree of exacerbation of heart failure of the patient based on the time series data of the score or the time series data of the one or more characteristic parameters; The heart failure prediction device according to claim 2 or 3.

12. the extraction unit extracts feature points representing positions of each part in the face from the image or video acquired by the acquisition unit, and extracts the one or more feature parameters based on the feature points; The heart failure prediction device according to claim 2 or 3.

13. A heart failure inference system including a user terminal equipped with a camera and the heart failure inference device according to any one of claims 1 to 3, the user terminal transmits the image or the video captured by the camera to the heart failure estimation device; The acquisition unit of the heart failure estimation device acquires the image or the video transmitted from the user terminal, The estimation unit of the heart failure estimation device estimates a degree of heart failure of the patient based on the image or the video. Heart failure estimation system.

20. Acquire an image or video obtained by capturing an image of the patient's face; Extracting one or more feature parameters from the captured images or video; estimating a degree of heart failure of the patient based on the one or more extracted characteristic parameters; The feature parameters include parameters related to the shape of the patient's face, parameters related to the patient's eyes, parameters related to the patient's mouth, and near-infrared water absorption parameters for the patient's face. The parameters include at least one of a parameter representing a result of a quantity analysis and a parameter representing three-dimensional shape data of the patient's face. A heart failure estimation method in which processing is performed by a computer.

21. Acquire an image or video obtained by capturing an image of the patient's face; Extracting one or more feature parameters from the captured images or video; estimating a degree of heart failure of the patient based on the one or more extracted characteristic parameters; The feature parameters include parameters related to the shape of the patient's face, parameters related to the patient's eyes, parameters related to the patient's mouth, and near-infrared water absorption parameters for the patient's face. The parameters include at least one of a parameter representing a result of a quantity analysis and a parameter representing three-dimensional shape data of the patient's face. A heart failure estimation program for causing a computer to execute the processing.

23. An acquisition unit that acquires an image or video obtained by capturing an image of a patient's face; an extraction unit that extracts one or more feature parameters from the image or video acquired by the acquisition unit; Based on the one or more feature parameters extracted by the extraction unit and a trained model that has been machine-learned in advance, the one or more feature parameters are input to the trained model. By doing so, a score representing the degree of heart failure of the patient is obtained, which is output from the trained model, and the degree of aggravation of the heart failure of the patient is calculated based on the time series data of the score. An estimation unit that performs estimation; an output unit that outputs the result estimated by the estimation unit; A heart failure estimation device comprising:

24. An acquisition unit that acquires an image or video obtained by capturing an image of a patient's face; an extraction unit that extracts one or more feature parameters from the image or video acquired by the acquisition unit; Based on the one or more feature parameters extracted by the extraction unit and a statistical model generated in advance, the one or more feature parameters are input as explanatory variables of the statistical model. By inputting the statistical model, a target variable obtained from the statistical model is obtained as a score representing the degree of heart failure of the patient, and based on the time series data of the score, the patient's heart failure aggravation is calculated. An estimation unit that estimates the degree of an output unit that outputs the result estimated by the estimation unit; A heart failure estimation device comprising:

29. Acquire an image or video obtained by capturing an image of the patient's face; Extracting one or more feature parameters from the captured images or video; Based on the one or more extracted feature parameters and a trained model that has been machine-learned in advance, the one or more feature parameters are input to the trained model, Obtaining a score representing the degree of heart failure of the patient output from the trained model, and estimating the degree of exacerbation of heart failure of the patient based on time series data of the score; Output the estimated results, A heart failure estimation method in which processing is performed by a computer.

30. Acquire an image or video obtained by capturing an image of the patient's face; Extracting one or more feature parameters from the captured images or video; Based on the one or more extracted feature parameters and a statistical model generated in advance, the one or more feature parameters are input as explanatory variables of the statistical model. The objective variable obtained from the statistical model is obtained as a score representing the degree of heart failure of the patient, and the degree of aggravation of the heart failure of the patient is estimated based on time series data of the score. 、 Output the estimated results, A heart failure estimation method in which processing is performed by a computer.

35. Acquire an image or video obtained by capturing an image of the patient's face; Extracting one or more feature parameters from the captured images or video; Based on the one or more extracted feature parameters and a trained model that has been machine-learned in advance, the one or more feature parameters are input to the trained model, Obtaining a score representing the degree of heart failure of the patient output from the trained model, and estimating the degree of exacerbation of heart failure of the patient based on time series data of the score; Output the estimated results, A heart failure estimation program for causing a computer to execute the processing.

36. Acquire an image or video obtained by capturing an image of the patient's face; Extracting one or more feature parameters from the captured images or video; Based on the one or more extracted feature parameters and a statistical model generated in advance, the one or more feature parameters are input as explanatory variables of the statistical model. The objective variable obtained from the statistical model is obtained as a score representing the degree of heart failure of the patient, and the degree of aggravation of the heart failure of the patient is estimated based on time series data of the score. 、 Output the estimated results, A heart failure estimation program for causing a computer to execute the processing.

Images