Edema evaluating method, evaluating device, learning device, learning model producing method, trained model, and program

Abstract

To provide a technology capable of improving accuracy while suppressing the influence of individual characteristics in the evaluation of pitting edema.SOLUTION: A learning model producing method for evaluating pitting edema in the lower leg of a body includes the steps of: acquiring a pair image 210 consisting of a post-compression image 240 after applying pressure to a partial area of the lower leg for a predetermined period and a pre-compression image 230 before applying the pressure for each of a plurality of subjects (S11); and acquiring a correct label of an indentation intensity D in the post-compression image 240 for each of a plurality of subjects (S12). This producing method further includes step (S13) of performing machine learning on a learning model to establish a relation between the pair image 210 and the indentation intensity D using teacher data that comprises the pair image 210 and the correct label of the indentation intensity D for each subject.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a technique for evaluating edema (pitting edema) and related techniques. [Background technology]

[0002] "Edema" (swelling) can appear as a precursor to heart failure, and it is desirable to detect edema at an early stage as an exacerbation symptom of heart failure, etc. For example, "pitting edema" can be evaluated by continuously pressing (compressing) the front of the lower leg with a finger for a predetermined period of time, leaving an indentation after the finger is released. It is desirable to evaluate pitting edema objectively.

[0003] An example of such an evaluation technique is described in Patent Document 1. Patent Document 1 describes evaluating pitting edema based on the recovery time until the indentation on the lower leg disappears or the depth of the indentation. The presence or depth of the indentation is detected by analyzing an image containing the indentation. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2022 / 065391 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology of Patent Document 1, the state of an indentation (presence or absence, depth, etc.) at a certain point in time is detected based on a single image (frame image, etc.) at that point in time.

[0006] There are individual differences in the condition of the skin surface (such as the condition of wrinkles) of the lower leg of a subject. When the condition of an indentation at a certain point in time is detected based on only a single image, as in the technology of Patent Document 1, the condition of the indentation may not always be detected appropriately due to the influence of the individual differences (individual characteristics).

[0007] Therefore, an object of the present invention is to provide a technology that can improve accuracy in the evaluation of pitting edema by suppressing the influence of individual characteristics. [Means for solving the problem]

[0008] In order to solve the above problem, the learning model production method of the present invention is a learning model production method for producing a learning model for evaluating pitting edema of the lower leg of a body, and is characterized by comprising the steps of: a) acquiring paired images for each of a plurality of subjects, each paired image consisting of a post-compression image after compression has been applied to a portion of the lower leg for a predetermined period of time and a pre-compression image before the compression was applied; b) acquiring correct labels for the indentation intensity in the post-compression image for each of the plurality of subjects; and c) using training data consisting of the paired images of each subject and the correct labels for the indentation intensity of each subject, machine learning the relationship between the paired images and the indentation intensity in the learning model.

[0009] The pre-compression image and the post-compression image may each be a color image.

[0010] The pre-compression image and the post-compression image may each be a three-channel image in which each pixel has an RGB value, and the pair of images may be a six-channel image.

[0011] The learning model may be a deep learning model.

[0012] The learning model is a deep learning model, and the deep learning model includes a first deep learning model that accepts three-channel input, and a second deep learning model that has a structure in which an additional convolutional layer is added to the input side of the first deep learning model, and the additional convolutional layer may be a convolutional layer that involves conversion from six channels to three channels.

[0013] The first deep learning model may be a learning model that has been pre-trained using learning data other than the training data, and the second deep learning model may be machine-learned by fine-tuning using the training data.

[0014] The correct label may be a value indicating which of a plurality of stages the indentation strength is.

[0015] In order to solve the above problem, the trained model of the present invention is characterized in that it is a trained model produced using any of the above trained model production methods.

[0016] In order to solve the above problem, the present invention provides a program for causing a computer to execute any one of the above learning model production methods.

[0017] In order to solve the above problems, the evaluation method of the present invention is an evaluation method for evaluating pitting edema of the lower leg of the body using a learning model, and is characterized by comprising the steps of: a) obtaining a post-compression image after applying compression to a portion of the lower leg of the person being evaluated for a predetermined period of time and a pre-compression image before the compression was applied; and b) inputting a pair of images consisting of the pre-compression image and the post-compression image into the learning model and calculating an evaluation value of the pitting edema of the person being evaluated based on the output value output from the learning model.

[0018] The learning model may be a learning model that takes the paired images as input and outputs the indentation intensity in the post-compression image, and may be a learning model that has been trained using multiple training data consisting of the paired images of multiple subjects and correct labels of the indentation intensity of each of the multiple subjects.

[0019] The pre-compression image and the post-compression image may each be a color image.

[0020] In order to solve the above problems, the present invention provides a program for causing a computer to execute any one of the above evaluation methods.

[0021] In order to solve the above problem, the learning device of the present invention includes a control unit that performs machine learning to create a learning model for evaluating pitting edema in the lower leg of the body, and the control unit acquires paired images for each of a plurality of subjects, each consisting of a post-compression image after compression has been applied to a portion of the lower leg for a predetermined period of time and a pre-compression image before the compression was applied, acquires correct labels for the indentation intensity in the post-compression images for each of the plurality of subjects, and uses training data consisting of the paired images of each subject and the correct labels for the indentation intensity of each subject to machine learn the relationship between the paired images and the indentation intensity in the learning model.

[0022] In order to solve the above problem, the evaluation device of the present invention includes a control unit that calculates an evaluation value for pitting edema of the lower leg of the person being evaluated, and the control unit acquires a post-compression image after applying compression to a portion of the lower leg of the person being evaluated for a predetermined period of time and a pre-compression image before the compression was applied, inputs paired images consisting of the pre-compression image and the post-compression image into a learning model, and calculates an evaluation value for the pitting edema of the person being evaluated based on the output value output from the learning model. [Effects of the Invention]

[0023] According to the present invention, it is possible to improve the accuracy of the evaluation of pitting edema by suppressing the influence of individual characteristics. [Brief explanation of the drawings]

[0024] [Figure 1] FIG. 1 is a schematic diagram showing an evaluation system for evaluating edema (pitting edema). [Figure 2] FIG. 2 is a diagram illustrating functional blocks of the evaluation device. [Figure 3] FIG. 2 is a diagram illustrating functional blocks of a terminal device. [Figure 4] FIG. 1 is a diagram illustrating the processing in the learning stage and the processing in the inference stage. [Figure 5] 10 is a flowchart showing the process of the learning stage. [Figure 6] 10 is a flowchart showing the processing at the inference stage. [Figure 7] Flowchart showing the process of capturing a moving image [Figure 8] FIG. 2 is a diagram showing the imaging conditions of a subject. [Figure 9] FIG. 2 is a diagram showing the timing of various processes related to image capture. [Figure 10] 10A to 10C are diagrams showing pre-compression images and the like. [Figure 11] FIG. 10 is a diagram showing images after compression. [Figure 12] FIG. 1 is a diagram showing criteria for indentation strength. [Figure 13] FIG. 10 is a diagram illustrating a specific configuration example of a learning model. [Figure 14] FIG. 1 illustrates three-fold cross-validation. [Figure 15] 1 is a graph showing the results of Experiment 1. [Figure 16] FIG. 1 shows the verification results of Experiment 1 in the form of a confusion matrix for two-label classification. [Figure 17] 10 is a graph showing the results of Experiment 2. [Figure 18] FIG. 10 shows the verification results of Experiment 2 in the form of a confusion matrix for two-label classification. [Figure 19]FIG. 10 shows pre-compression and post-compression images for four improved examples. [Figure 20] FIG. 10 is a diagram showing a learning model according to a modified example. [Figure 21] FIG. 10 is a diagram showing a learning model according to another modified example. [Figure 22] FIG. 10 is a diagram showing a specific configuration of a learning model according to a comparative example. DETAILED DESCRIPTION OF THE INVENTION

[0025] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0026] <1. System Overview> Fig. 1 is a schematic diagram showing an evaluation system 1 for evaluating edema (pitting edema). As shown in Fig. 1, the evaluation system 1 includes an evaluation device 30 for evaluating edema (pitting edema) and a plurality of terminal devices 70 (see also Figs. 2 and 3). Fig. 2 is a diagram showing the functional blocks of the evaluation device 30, and Fig. 3 is a diagram showing the functional blocks of the terminal devices 70.

[0027] The evaluation device 30 and each terminal device 70 can communicate with each other via a network (including the Internet, etc.). The evaluation device 30 is configured as a server device (such as a cloud server), and the terminal device 70 is configured as a client device. In other words, the evaluation system 1 is configured as a client-server system.

[0028] The terminal device 70 is placed in a hospital, a home, or the like, and generates a captured image by capturing an image of the lower leg or the like of a person present near the terminal device 70. The terminal device 70 transmits the captured image to the evaluation device 30, and the evaluation device 30 executes a learning process of a learning model and an evaluation process using the learning model based on the captured image.

[0029] The evaluation device 30 is equipped with a learning model 410 (FIG. 2) for evaluating pitting edema of the lower legs of the body. The learning model 410 after being trained by machine learning is also referred to as a trained model 420. Specifically, the learning parameters of the learning model 410 (learner) are adjusted using a predetermined machine learning method, and a trained learning model 410 (trained model 420) is generated (see the upper part of FIG. 4). Furthermore, the trained model 420 is used to estimate (infer) the state of edema of the lower legs of the person to be evaluated (evaluation subject) (see the lower part of FIG. 4). Note that FIG. 4 is a diagram showing the processing in the learning stage and the processing in the inference stage.

[0030] As the learning model 410, for example, a neural network model (particularly, a deep neural network model) configured with multiple layers may be used (see FIG. 13). Weighting coefficients and the like (learning parameters) for the multiple layers (input layer, (one or more) intermediate layers, and output layer) in the neural network model may be adjusted using a predetermined machine learning method. As the neural network model, for example, a model (described later) that applies an efficient net (such as EfficientNet-B0) may be used.

[0031] Here, a post-compression image 240 (see FIG. 11) after pressure is applied with a finger (thumb) to a partial region 242 (see FIG. 11) of the lower leg of each subject for a predetermined period of time is used, and a pre-compression image 230 (see FIG. 10) before the pressure is applied is also used. In detail, the pre-compression image 230 is an image (photographed image) of an area including the partial region 242 before the pressure (immediately before the pressure), and the post-compression image 240 is an image (photographed image) of an area including the partial region 242 after the pressure (immediately after the pressure). A set (pair) of images consisting of the pre-compression image 230 and the post-compression image 240 is used as a pair of images 210.

[0032] Specifically, the learning model 410 is a model that learns the relationship between the paired images 210 (the pre-compression image 230 and the post-compression image 240) for each of a plurality of subjects and the indentation intensity D in the post-compression image 240. In more detail, the learning model 410 is a model that receives the paired images 210 (the pre-compression image 230 and the post-compression image 240) as input and outputs the indentation intensity D. The indentation intensity D is used as an evaluation value for pitting edema.

[0033] In the machine learning of the learning model 410, the indentation intensity D in the post-compression image 240 of each subject is used as correct data (also referred to as correct label or simply label). In particular, a combination (dataset) of the paired images 210 (pre-compression image 230 and post-compression image 240) of each subject and the indentation intensity D of each subject is used as training data (labeled data) in the machine learning of the learning model 410. In other words, the relationship between the paired images 210 and the indentation intensity D in the learning model 410 is machine-learned using training data configured to include the paired images 210 of each subject and the correct label of the indentation intensity D of each subject.

[0034] In this embodiment, not only the post-compression image 240 but also the pre-compression image 230 is used. This makes it possible to suppress the influence of individual characteristics (improve accuracy).

[0035] <2. Evaluation device 30> The evaluation device 30 is configured as an information processing device. The evaluation device 30 is a device that evaluates (estimates) the state of pitting edema, and is also referred to as an edema evaluation device. The evaluation device 30 is also a device that evaluates (measures (quantifies)) the strength of an indentation (indentation strength), and is also referred to as an indentation strength evaluation device (indentation strength measuring device or indentation strength quantification device).

[0036] In this embodiment, the evaluation device 30 executes an inference process (evaluation process) using the learning model 410, and also executes a generation process (learning process) for the learning model 410. Therefore, the evaluation device 30 is also expressed as a learning model generation device or a learning device.

[0037] As shown in FIG. 2, the evaluation device 30 includes a controller 31 (also referred to as a control unit), a storage unit 32, a communication unit 34, and an operation unit 35.

[0038] The controller 31 is a control device that is built into the evaluation device 30 and controls the operation of the evaluation device 30 .

[0039] The controller 31 is configured as a computer system including one or more hardware processors (e.g., a central processing unit (CPU) and a graphics processing unit (GPU)). The controller 31 performs various processes by executing, in the CPU or the like, a predetermined software program (hereinafter also simply referred to as a program) stored in a storage unit (a non-volatile storage unit such as a ROM and / or a hard disk) 32. The program (more specifically, a group of program modules) (also referred to as a "program product") may be recorded on a portable recording medium such as a USB memory, read from the recording medium, and installed in the evaluation device 30. Alternatively, the program may be downloaded via a communication network or the like and installed in the evaluation device 30.

[0040] The controller 31 executes processing related to the learning stage of machine learning.

[0041] Specifically, the controller 31 first executes a process of acquiring training data. Specifically, the controller 31 acquires paired images 210 (pre-compression image 230 and post-compression image 240) of each of a plurality of subjects, and also acquires data indicating the indentation intensity of each of the plurality of subjects. That is, the controller 31 acquires a combination of the paired images 210 of each subject and the indentation intensity D of each subject as training data.

[0042] Next, the controller 31 executes a process of optimizing learning parameters for the learning device (learning model 410) based on the training data, and generates a trained model 420. More specifically, machine learning is performed using data indicating a known relationship between each paired image 210 of a plurality of subjects and each indentation intensity D of the plurality of subjects as training data, and the learning parameters are optimized. As a result, the trained model 420 is generated.

[0043] Furthermore, the controller 31 executes processing related to the inference stage of machine learning. Specifically, using the learning model 410 (trained model 420) in which the learning parameters have been adjusted, processing is executed to calculate an evaluation value of pitting edema of the lower leg of the evaluation subject (estimation processing (inference processing) for estimating the indentation intensity D) based on the paired images 210 of the evaluation subject. In other words, evaluation processing is executed to evaluate the state of pitting edema.

[0044] The storage unit 32 is configured with a storage device such as a hard disk drive (HDD) and / or a solid state drive (SSD), etc. The storage unit 32 stores a learning model 410 (including learning parameters and programs related to the learning model) (and thus a trained model 420), etc.

[0045] The communication unit 34 is capable of performing network communication via a network. This network communication utilizes various protocols, such as TCP / IP (Transmission Control Protocol / Internet Protocol). By utilizing this network communication, the evaluation device 30 can exchange various data with a desired counterpart (for example, a terminal device 70). For example, a moving image 200 ( FIG. 1 ) captured by the terminal device 70 is transmitted from the terminal device 70 to the evaluation device 30, whereby the evaluation device 30 can receive the moving image 200 (including a pre-compression image 230 and a post-compression image 240).

[0046] The operation unit 35 includes an operation input unit 35a that accepts operation inputs to the evaluation device 30, a display unit 35b that displays and outputs various information, and an audio input / output unit 35e that inputs and outputs audio. A mouse and a keyboard are used as the operation input unit 35a, and a display (such as a liquid crystal display) is used as the display unit 35b. A touch panel that functions as both a part of the operation input unit 35a and a part of the display unit 35b may also be provided. The audio input / output unit 35e includes an audio input device (audio input unit) such as a microphone and an audio output device (audio output unit) such as a speaker.

[0047] <3. Terminal Device 70> Each terminal device 70 is an information input / output terminal device (information processing device) capable of network communication with the evaluation device 30. Each terminal device 70 is configured as a smartphone, a tablet terminal, or a personal computer (which may be fixed (desktop) or portable), etc. FIG. 1 illustrates a terminal device 70 configured as a tablet terminal.

[0048] The terminal device 70 exchanges various types of information with the evaluation device 30. The terminal device 70 generates a captured image (moving image 200) capturing the lower legs or the like of a person present near the terminal device 70, and transmits the captured image to the evaluation device 30. The terminal device 70 also receives information from the evaluation device 30 and displays the information on its display unit 75b or the like.

[0049] FIG. 3 is a functional block diagram showing a schematic configuration of the terminal device 70. As shown in FIG.

[0050] As shown in the functional block diagram of Figure 3, the terminal device 70 includes a controller 71, a memory unit 72, a communication unit 74, an operation unit 75, and an imaging unit 76, and various functions are realized by operating these units in combination.

[0051] The controller (control unit) 71 is a control device that controls the terminal device 70 .

[0052] The controller 71 has the same hardware configuration as the controller 31. The controller 71 executes, in a CPU or the like, a predetermined software program stored in a storage unit (a non-volatile storage unit such as a ROM and / or a hard disk) 72, thereby realizing various processes. The program (more specifically, a group of program modules) (also referred to as a "program product") may be recorded on a portable recording medium such as a USB memory, read from the recording medium, and installed in the terminal device 70. Alternatively, the program may be downloaded via a communication network or the like and installed in the terminal device 70.

[0053] The controller 71 executes the program and performs the following various processes.

[0054] Specifically, during the learning stage of the learning model 410, the controller 71 performs processes such as capturing moving images 200 (including pre-compression images 230 and post-compression images 240) for each subject, and also performs processes such as transmitting the moving images 200 for each subject to the evaluation device 30.

[0055] Furthermore, the controller 71 captures (generates) a video 200 (including a pre-compression image 230 and a post-compression image 240) of the person to be evaluated in the inference stage using the learning model 410 (more specifically, the trained model 420). The controller 71 also transmits the video 200 of the person to be evaluated to the evaluation device 30. Furthermore, the controller 71 acquires, from the evaluation device 30, an estimation result of the indentation strength D by the evaluation device 30 using the trained model 420, and displays the estimation result on the display unit 75b.

[0056] The storage unit 72 has the same hardware configuration as the storage unit 32 .

[0057] The communication unit 74 has the same hardware configuration as the communication unit 34. By utilizing network communication via the communication unit 74, the terminal device 70 can exchange various data with a desired counterpart (e.g., the evaluation device 30).

[0058] The operation unit 75 includes an operation input unit 75a that accepts operation input to the terminal device 70, a display unit 75b that displays and outputs various types of information, and an audio input / output unit 75e that inputs and outputs audio. A display (such as a liquid crystal display) is used as the display unit 75b. Here, a touch panel 75c (FIG. 1) is provided that functions as both a part of the operation input unit 75a and a part of the display unit 75b. A mouse, a keyboard, or the like may also be used as the operation input unit 75a. The audio input / output unit 75e includes an audio input device (audio input unit) such as a microphone and an audio output device (audio output unit) such as a speaker. The input audio is recognized as a character string or the like by voice recognition processing.

[0059] The photographing unit 76 is configured to include a camera 76c, etc. The photographing unit 76 is capable of generating photographed images (pre-compression image 230, post-compression image 240, etc.) by the camera 76c. Specifically, the camera 76c has an RGB image sensor or the like that captures visible light images (color images, etc.), and is capable of capturing color moving images (and color still images).

[0060] <4. Learning Stage Processing> Below, the processes in the learning stage of the learning model 410 and the inference stage using the learning model 410 (trained model 420) will be explained in order.

[0061] First, the learning stage processing will be described.

[0062] 5 is a flowchart showing the process of the learning stage. The process shown in FIG. 5 is executed by the controller 31 and the like.

[0063] 5 is also a diagram showing a method for generating a trained model. In this application, generating a trained model 420 means producing (manufacturing) the trained model 420, and the "method for generating a trained model" means the "method for producing a trained model."

[0064] The learning stage processing in FIG. 5 is roughly divided into a preparatory stage processing for teacher data (steps S11 to S13) and a machine learning processing of the learning model 410 using the teacher data (step S14).

[0065] Furthermore, prior to the processing of steps S11 to S13, the processing of Fig. 7 (steps S1 to S5) is executed. Fig. 7 is a flowchart showing the processing of capturing a moving image 200 including a pre-compression image 230 and a post-compression image 240. The processing of Fig. 7 is executed by the terminal device 70 (under the control of the controller 71). First, the processing of Fig. 7 will be described below.

[0066] Fig. 8 is a diagram showing the state of a subject (a person undergoing an edema test) during the processing of Fig. 7. As shown in Fig. 8, the subject (person not being photographed) is photographed while sitting in a chair 83. The terminal device 70 used for photographing is supported by a stand 85 and set at a predetermined height in front of the subject, as shown in Fig. 8, for example. The terminal device 70 is set so that its photographing unit 76 captures the lower leg (particularly the anterior shin) of the subject within its field of view.

[0067] In step S1 of Fig. 7, the terminal device 70 displays preparation guidance 101 (see Fig. 9) on the display unit 75b. Specifically, the guidance 101 reads, "Take off your shoes, place your right foot on the footrest, and align your foot with the footprint mark (reference position) on the footrest. When you are ready, press the OK button or say that you are ready." The guidance 101 is displayed on the display unit 75b and is also output as voice.

[0068] The subject places his / her bare right foot on the footrest 81 in accordance with the guidance 101, as shown in FIG. 8, and aligns the foot position on the footrest 81 with the reference position. A footprint graphic 82 (a graphic that simplifies footprints (see FIG. 10, etc.)) indicating the reference position is drawn on the top surface of the footrest 81. The subject can align his / her foot with the reference position by positioning his / her foot so that it aligns with the footprint graphic 82. Thereafter, the subject notifies the terminal device 70 that he / she is ready by pressing the OK button displayed on the display unit 75b (or by saying "Ready").

[0069] When the terminal device 70 confirms that the OK button has been pressed (or that "ready" has been spoken), the process proceeds to step S2.

[0070] In step S2, the terminal device 70 starts capturing the video 200 (at time T1 (see also FIG. 9) immediately after time T0 when it is confirmed that preparation is complete). Note that FIG. 9 is a diagram showing the timing of various processes related to image capturing.

[0071] In step S3, the terminal device 70 displays guidance 102 (FIG. 9) for instructing to apply pressure (for instructing to start applying pressure) on the display unit 75b (times T2 to T3). Specifically, the following messages are sequentially displayed and output as audio: "At the start signal, please apply firm pressure with your right thumb to the front of your right foot for 3 seconds (enough to leave a mark), then release your thumb. The countdown will now begin.", "5 seconds left," "3," "2," "1," and "Start." It is preferable that an image showing the pressure position in detail (such as an image of a foot with the position of the fingers superimposed during pressure) is also displayed on the display screen.

[0072] In response to the guidance 102 (see FIG. 9) for instructing the subject to start compression, the subject extends his / her right hand to the front shin of his / her right leg and compresses (presses hard) the front shin with the thumb of his / her right hand for a predetermined period M1 (e.g., 3 seconds).

[0073] Then, at time T4, when a predetermined period M1 has elapsed since the start instruction (time T3) (for example, 3 seconds after time T3), guidance 103 (Figure 9) for instructing the user to end compression, such as "Please release your finger," is displayed on display unit 75b and output as voice.

[0074] Then, in step S4, the terminal device 70 finishes capturing the video 200 (at time T5, which is a few seconds (for example, 5 seconds) after time T4).

[0075] In the next step S5, the terminal device 70 transmits the captured video 200 to the evaluation device 30. Information (date, hour, minute, second, etc.) about the capture start time (recording start time) T1 and each of the times T2, T3, and T4 for the video 200 is recorded as information about each time within the video 200, attached to the video 200, and transmitted to the evaluation device 30. Alternatively, information about the elapsed time from time T1 to each of the times T2, T3, and T4 may be recorded as information about each time within the video 200, attached to the video 200, and transmitted to the evaluation device 30.

[0076] The moving image 200 includes information on a pre-compression image 230 (see FIG. 10) and information on a post-compression image 240 (see FIG. 11). As will be described later, the pre-compression image 230 and the post-compression image 240 are extracted from the moving image 200 by the evaluation device 30.

[0077] Next, returning to FIG. 5, the preparatory processing of teacher data by the evaluation device 30 (steps S11 to S13) will be described.

[0078] In step S11, the evaluation device 30 acquires a pre-compression image 230 and a post-compression image 240 as a pair of images 210. The post-compression image 240 is an image taken after applying pressure to a partial region 242 of the subject's lower leg for a predetermined period M1 (e.g., 3 seconds). On the other hand, the pre-compression image 230 is an image taken before applying pressure to the partial region 242 for the predetermined period M1. The pre-compression image 230 and the post-compression image 240 are each images of an area including the partial region 242.

[0079] Specifically, the evaluation device 30 extracts (acquires) a post-compression image 240 from the moving image 200. A portion of a certain frame image among a plurality of frame images taken during a period from time T4 (see FIG. 9 ) immediately after the completion of compression to time T5 when image capture ends is extracted as the post-compression image 240. For example, within one frame image taken at time T21 (see FIG. 9 ), a certain time ΔT (the time required for the finger to leave the foot) has elapsed since time T4 immediately after the completion of compression, a region including a partial region 242 that was pressed by the finger until just before time T4 is extracted as the post-compression image 240 ( FIG. 11 ). Time T21 can also be expressed as the time immediately after the finger leaves the foot. The post-compression image 240 ( FIG. 11 ) is extracted (acquired) as a partial region of the high-resolution moving image 200 (more specifically, one frame image within the moving image 200), specifically, as a region (rectangular region) having a predetermined size (for example, 448×448 pixels). Furthermore, the partial region 242 that has been pressed by the finger may be identified (automatically) by image analysis processing, or may be identified (manually) by a person (e.g., a user). Then, a region including the partial region 242 is extracted as the post-compression image 240. The post-compression image 240 may be extracted automatically by image analysis processing or the like, or may be extracted (manually) by a region designation operation or the like by a person (e.g., a user).

[0080] Here, the post-compression image 240 is extracted as a three-channel image (a color image having an R image, a G image, and a B image) in which each pixel has an RGB value. The post-compression image 240 is extracted as voxel data having a pixel size of, for example, 448 (width W) * 448 (height H) * 3 (number of channels C).

[0081] The evaluation device 30 also extracts (acquires) a pre-compression image 230 from the moving image 200. Specifically, a portion of a certain frame image among a plurality of frame images taken during a period from the imaging start time T1 to a time T3 immediately before the start of compression (preferably until the start time T2 of the compression start instruction guidance) is extracted as the pre-compression image 230. For example, within one frame image taken at the start time T2 of the compression start instruction guidance (or the time immediately before that), a partial region (a region including the partial region 242 described above) corresponding to the post-compression image 240 (see FIG. 11 ) is extracted as the pre-compression image 230 (see FIG. 10 ). The partial region corresponding to the post-compression image 240 may be identified by image analysis processing or by a human (e.g., a user). In other words, the pre-compression image 230 may be extracted automatically by image analysis processing or the like, or may be extracted (manually) by a human (e.g., a user) specifying an area. The pre-compression image 230 is extracted (acquired) as a region having the same pixel size as the post-compression image 240. Note that the partial region extracted as the pre-compression image 230 and the partial region extracted as the post-compression image 240 do not need to be regions in exactly the same place (position) (regions that completely match), and may be regions in roughly the same place (regions that partially overlap, etc.).

[0082] Here, the pre-compression image 230 is also extracted as a three-channel image (a color image having an R image, a G image, and a B image) in which each pixel has an RGB value, similar to the post-compression image 240. The pre-compression image 230 is extracted as voxel data having a pixel size of, for example, 448*448*3.

[0083] 11 is a diagram showing the position of the post-compression image 240 in the moving image 200 (specifically, one frame image), etc. Also, FIG. 10 is a diagram showing the position of the pre-compression image 230 (partial region corresponding to the post-compression image 240 (corresponding region before compression)) in the moving image 200 (specifically, another frame image).

[0084] A pre-compression image 230 and a post-compression image 240 of one subject are acquired as described above. The pre-compression image 230 and the post-compression image 240 are acquired as a pair of images (paired images 210) containing information on changes in state before and after compression. In other words, the pre-compression image 230 and the post-compression image 240 constitute the paired images 210. This paired image 210 is a 6-channel image (see the top of FIG. 13 ). The paired images 210 are configured as voxel data having a pixel size of, for example, 448*448*6.

[0085] In the next step S12, the evaluation device 30 acquires a correct label for the indentation intensity D corresponding to the paired images 210 of the subject (more specifically, the indentation intensity D in the post-compression image 240 of the paired images 210). The correct label is assigned by a doctor or the like (a label assigner). Specifically, the doctor or the like determines which of a plurality of levels (e.g., five levels) the state of the indentation (indentation intensity) in the post-compression image 240 is in based on predetermined criteria as the indentation intensity D. The indentation intensity D related to this determination result is assigned as the correct label. The correct label is a value indicating which of a plurality of levels (e.g., five levels) the indentation intensity D is in. The correct label is assigned in advance (into the evaluation device 30) by a user (e.g., a doctor) prior to step S12 and input in advance. Alternatively, the correct label may be assigned (input) at the time of processing in step S12. Then, the evaluation device 30 acquires the assigned (input) correct label in step S12.

[0086] Fig. 12 is a diagram showing the criteria for determining the indentation strength D. Fig. 12 shows the criteria for labeling the indentation strength D on a five-level scale (the state of each level (stage) of the indentation strength D). Of the five levels (0 to 4) of the indentation strength D, an indentation strength of "0" represents the smallest (weakest) strength, and an indentation strength of "4" represents the largest (strongest) strength.

[0087] An indentation strength of "4" indicates a state in which a "deep depression is formed" (however, the depression is deeper (more clearly defined) than an indentation strength of "3" described below).

[0088] An indentation strength of "3" indicates a state in which "a depression is formed covering the area pressed by the finger" (more specifically, a state in which a wide (but shallow) depression is formed over the entire oval-shaped pressure area (for example, from the area covered by the left end of the thumb to the area covered by the right end of the thumb)).

[0089] An indentation strength of "2" indicates that "a depression is formed in part of the area pressed with the finger" (more specifically, a depression is formed in only part of the oval-shaped pressure area (such as only one side of the area covered by the left end of the thumb or the area covered by the right end of the thumb)).

[0090] An indentation strength of "1" indicates a state in which "there is no indentation, but a color change is observed."

[0091] An indentation strength of "0" indicates a state in which "neither a dent nor a color change is observed."

[0092] Based on such criteria, the doctor or the like assigns a correct label of the indentation intensity D to one pair of images 210 (particularly the post-compression image 240) related to one subject, and inputs the correct label (value) to the evaluation device 30. The evaluation device 30 acquires the correct label input by the doctor or the like.

[0093] In this way, the evaluation device 30 obtains the correct label of the indentation intensity D in the paired images 210 (specifically, the post-compression image 240) of one subject.

[0094] Although the indentation strength D is divided into five levels here, the present invention is not limited to this. For example, the indentation strength D may be divided into two levels ("weak" and "strong") or another number of levels.

[0095] In step S13, the evaluation device 30 generates training data for the learning model 410. Specifically, the training data is generated including the paired images 210 of one subject and the correct label of the indentation intensity D of the subject.

[0096] The processing of steps S11 to S13 as described above is repeatedly performed for a plurality of subjects, thereby generating a plurality of training data (a plurality of training data consisting of paired images of the plurality of subjects and correct labels of indentation strength for each of the plurality of subjects). If it is determined in step S14 that the processing of steps S11 to S13 has not yet been completed for any subject, the process returns to step S11, and the processing of steps S11 to S13 is performed for the next subject. On the other hand, if it is determined in step S14 that the processing of steps S11 to S13 has been completed for all of the plurality of subjects, the process proceeds to step S15.

[0097] Note that, although the processes of steps S11 to S13 are executed for each subject here, this is not limiting. For example, processes for multiple subjects may be executed (collectively) in steps S11 to S13.

[0098] In step S15, the evaluation device 30 performs machine learning of the learning model 410 using a plurality of training data generated in steps S11 to S13, etc. The learning model 410 is a learning model that receives the paired images 210 as input and outputs the indentation intensity D in the post-compression image 240. Using training data configured with the paired images 210 of each subject and the correct label of the indentation intensity D of each subject as described above, the relationship (input / output relationship) between the paired images 210 and the indentation intensity D in the learning model 410 is machine learned. Specifically, in the learning model 410, machine learning is performed so as to reduce the error (a loss function indicating the error) between the output value for the indentation intensity D of each subject (the output value from the learning model 410) and the correct label of the indentation intensity D of each subject. As the loss function, for example, a cross-entropy error is used. Alternatively, a square sum error or the like may be used as the loss function.

[0099] A trained model 420 (trained learning model 410) is generated by such machine learning.

[0100] Fig. 13 is a diagram showing a specific configuration example of the learning model 410. The learning model 410 (also referred to as 410A) in Fig. 13 is constructed as a deep learning model that uses (applies) EfficientNet-B0 (see M. Tan, Q.V. Le: "EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks," ICML, 2019) as a base model.

[0101] This learning model 410A includes a basic part 450 of an efficient net (EfficientNet-B0). The basic part 450 (also referred to as the first deep learning model (described later)) is configured to include "Block 1" to "Block 7" and a convolutional layer 451 (Conv 3x3) on the input side of "Block 1" in EfficientNet-B0. The convolutional layer 451 is constructed as a layer that accepts three-channel input.

[0102] Furthermore, the learning model 410A has a structure in which another convolutional layer 471 is added before (further on the input side) the convolutional layer 451 on the input side. This convolutional layer 471 is a convolutional layer that involves conversion from 6 channels to 3 channels. By providing the convolutional layer 471 before (on the input side of) the convolutional layer 451, it is possible to make a base model that corresponds to a 3-channel input image correspond to a 6-channel input image.

[0103] Furthermore, the convolution layer 471 also has, strictly speaking, a pooling layer (a function for converting pixel size) (in addition to being a pure convolution layer (which performs only convolution operations)). For example, the pixel size on the input side of the convolution layer 451 is 224*224, and the pixel size on the input side of the convolution layer 471 is 448*448. The convolution layer 471 also has a function for converting the pixel size from 448*448 pixels to 224*224 pixels.

[0104] Thus, learning model 410A has a structure including a first deep learning model 450 (a base model having a convolutional layer 451 on the input side that accepts three-channel inputs) that accepts three-channel inputs (image inputs). In detail, learning model 410A is a deep learning model (also referred to as a second deep learning model) that has a structure in which a further convolutional layer 471 is added to the input side of the first deep learning model 450.

[0105] Similarly to the first deep learning model 450, the second deep learning model 410 (410A) functions as a classifier or the like by providing a fully connected layer (FC layer) 462 on the output side. For example, by providing a fully connected layer 462 that classifies the indentation intensity D into five levels ("0" to "4") based on a 1280-dimensional feature vector (output from the basic part 450), it is possible to classify the indentation intensity D into five levels ("0" to "4"). Alternatively, by providing a fully connected layer 462 (see FIG. 13) that classifies the indentation intensity D into two levels ("weak" and "strong") based on the 1280-dimensional feature vector (output from the basic part 450), it is possible to classify the indentation intensity D into two levels ("weak" and "strong"). However, the fully connected layer 462 may be configured to classify the indentation intensity D into other levels (e.g., three levels, four levels, six levels, seven levels, etc.).

[0106] Here, the learning model 410A (second deep learning model) is illustrated as an example of a mode in which it is machine-learned by fine tuning using the training data acquired in steps S11 to S13.

[0107] Specifically, the first deep learning model 450 (base model) is trained in advance using training data (other training data) different from the training data (training data acquired in steps S11 to S13). As the first deep learning model 450, for example, EfficientNet-B0 (the basic part thereof) trained by machine learning using more than one million images in the ImageNet database is used.

[0108] Then, a second deep learning model (learning model 410) including the first deep learning model 450 is fine-tuned using the training data (training data consisting of the paired images 210 and the ground truth data of the indentation intensity D) acquired in steps S11 to S13. For example, in addition to adjusting the parameter P0 of the convolution layer 471 and the parameter PZ of the output-side fully connected layer 462, the parameters of all layers of the first deep learning model 450 are updated (readjusted) (by fine-tuning using the learned parameter values ​​as initial values).

[0109] <5. Inference stage processing> Next, the process of the inference stage using the trained model 420 (the process of estimating the indentation strength D for a new subject) will be described.

[0110] Fig. 6 is a flowchart showing the processing of the inference stage. The processing shown in Fig. 6 is executed by the controller 31, etc. By the processing of the inference stage, an indentation intensity D (an evaluation value of pitting edema) of the evaluation subject (an estimated subject) is estimated based on paired images 210 of the evaluation subject.

[0111] The subject of evaluation may be, for example, a person other than the subject whose data was used in the learning stage. However, the subject of evaluation may also be a person (subject) whose data was used in the learning stage but whose state of pitting edema is unknown at a certain time after the data acquisition. Inference processing based on the trained model 420 may be used to estimate the pitting edema evaluation value of that person at that time.

[0112] First, in step S31, a pair of images 210 of the person to be evaluated is acquired. In step S31, the same processing as that in step S11 is performed for the person to be evaluated. Specifically, the terminal device 70 of the person to be evaluated generates (captures) a moving image 200 of the person to be evaluated (person to be photographed) and transmits the moving image 200 to the evaluation device 30. Based on the moving image 200, the evaluation device 30 acquires a pre-compression image 230 and a post-compression image 240 of the person to be evaluated as a pair of images 210. Specifically, a post-compression image 240 after compression has been applied to a portion of the lower leg of the person to be evaluated for a predetermined period of time M1, and a pre-compression image 230 before the compression have been applied are acquired as the pair of images 210.

[0113] In the next step S32, when the paired images 210 relating to the subject to be evaluated are input to the trained model 420, the indentation intensity D (classification value) is calculated as an estimation result by the trained model 420 and output from the trained model 420. Here, the value of the indentation intensity D (output value from the trained model 420) is calculated as the evaluation value of the pitting edema of the subject to be evaluated.

[0114] In step S33, the inference result (estimated value of indentation strength D) by the trained model 420 (trained learning model 410) is transmitted from the evaluation device 30 to the terminal device 70, and the inference result (estimated value of indentation strength D) is displayed on the terminal device 70 (display unit 75b). In other words, the evaluation device 30 transmits the inference result to the terminal device 70, and causes the terminal device 70 to display the inference result.

[0115] For example, on the display unit 75b of the terminal device 70 that has received the estimated value of the indentation strength D, the estimated value of the indentation strength D (evaluation value of the pitting edema) is displayed as, for example, "Your evaluation value of pitting edema is 4 out of 0 to 4," or "Your indentation strength D is strong out of strong and weak."

[0116] 6. Experimental Example <Summary> Below, the verification results of the generalization performance of the trained model 420 are shown.

[0117] In this verification, data on the combination of paired images 210 (pre-compression image 230 and post-compression image 240) of 247 people and the correct label of indentation intensity D was obtained in the same manner as steps S1 to S4 and S11 to S13 (FIGS. 7 and 5), and three-fold cross-validation (see FIG. 14) was performed using this data. Also, to facilitate verification, the correct labels were classified into two levels (weak / strong) instead of five levels (0 to 4). Then, two experiments (Experiment 1 and Experiment 2) were conducted in which the data to be used and the two-level classification method were changed.

[0118] In Experiment 1, data from 247 people with labeling results for indentation strength D (21 results for "0" (21 people), 62 results for "1" (79 people), 50 results for "3" (50 people), and 35 results for "4" (35 people)) were used. Data for indentation strengths of "0" and "4" (simply put, data from both extremes) were used. Indentation strength "0" was classified into the "weak" group, and indentation strength "4" was classified into the "strong" group. In other words, the data from 21 people with indentation strength "0" was used in the "weak" group, and the data from 35 people with indentation strength "4" was used in the "strong" group. In this way, in Experiment 1, data from a total of 56 people was classified into two levels of indentation strength D, "weak" and "strong," and used. In Experiment 1, the data from a total of 56 people was divided into three groups (18 people, 19 people, and 19 people), and three-fold cross-validation was performed. Note that the data from the remaining 191 people was not used in Experiment 1.

[0119] In Experiment 2, data for indentation intensities "0," "1," "3," and "4" are used from the data containing the labeling results of the indentation strength D of the 247 people. Indentation intensities "0" and "1" are classified into the "weak" group, and indentation intensities "3" and "4" are classified into the "strong" group. In other words, the data of 21 people with indentation strength "0" and the data of 62 people with indentation strength "1" are used as the data of the "weak" group (data of a total of 83 people), and the data of 50 people with indentation strength "3" and the data of 35 people with indentation strength "4" are used as the data of the "strong" group (data of a total of 85 people). In this way, in Experiment 2, the data of a total of 168 people is classified into two levels of indentation strength D, "weak" and "strong," and used. Furthermore, in Experiment 2, the data of the total 168 people is divided into three groups (56 people each) and three-fold cross-validation is performed. Note that the data of the remaining 79 people is not used in Experiment 2.

[0120] Furthermore, in order to compare the learning model 910 according to the comparative example (FIG. 22) with the above-mentioned learning model 410 (FIG. 13), a similar verification was also performed on the learning model 910 according to the comparative example. Specifically, in both Experiment 1 and Experiment 2, verification of the learning model 910 according to the comparative example and verification of the above-mentioned learning model (learning model according to the embodiment) 410 were performed, respectively.

[0121] Here, the learning model 910 according to the comparative example is a model similar to the learning model 410 according to the embodiment (see FIG. 13 ). However, the learning model 910 according to the comparative example differs from the learning model 410 in that it does not have the input-side convolution layer 471 and accepts only one of the post-compression images 240 as an input image, rather than the pair of images 210 (both the pre-compression image 230 and the post-compression image 240).

[0122] Furthermore, learning by fine tuning is also performed on the learning model 910 (see FIG. 22) according to the comparative example, as with the learning model 410. Specifically, parameters other than the convolutional layer 471 (not present in the learning model 910), i.e., the parameter PZ of the fully connected layer 462 on the output side and the parameters of all layers of the first deep learning model 450, are updated (readjusted) (by fine tuning using the learned parameter values ​​as initial values).

[0123] In three-fold cross-validation (FIG. 14), all target data (for example, data from 56 people in Experiment 1 (data from 168 people in Experiment 2)) is divided into three parts: first divided data, second divided data, and third divided data. Then, in the first round, the first divided data and the second divided data are used as training data to train a learning model (410 or 910), and the third divided data is used as verification data to verify the trained learning model (more specifically, the accuracy rate is calculated). In the second round, the first divided data and the third divided data are used as training data to train a learning model, and the second divided data is used as verification data to verify the trained learning model. In the third round, the second divided data and the third divided data are used as training data to train a learning model, and the first divided data is used as verification data to verify the trained learning model. Then, the average accuracy rate for the three rounds is obtained as the verification result.

[0124] <Results of Experiment 1> FIG. 15 is a graph showing the results of Experiment 1. In each of the upper and lower graphs in FIG. 15, the horizontal axis represents the number of training trials and the vertical axis represents the accuracy rate (correct answer rate). The upper graph in FIG. 15 shows the verification results of the learning model 910 according to the comparative example, and the lower graph in FIG. 15 shows the verification results of the learning model 410 according to this embodiment. Each graph also shows a curve L indicating the accuracy rate for the training data and a curve V indicating the accuracy rate for the verification data. Specifically, the upper graph shows an accuracy rate curve L9 of the learning model 910 for the training data and an accuracy rate curve V9 of the learning model 910 for the verification data. Similarly, the lower graph shows an accuracy rate curve L1 of the learning model 410 for the training data and an accuracy rate curve V1 of the learning model 410 for the verification data. The accuracy rate curves L1 and L9 are curves that show the accuracy rate when training data is re-input into the training model as a function of the number of training runs, and the accuracy rate curves V1 and V9 are curves that show the accuracy rate when verification data is input into the training model as a function of the number of training runs.

[0125] As shown in the upper part of Figure 15, the accuracy rate of learning model 910 after 200 training cycles (see V9) is 76.6% (the average value of the three accuracy rates (correct answer rates for each of the three divided data) for the three validation data in the three-fold cross-validation). On the other hand, as shown in the lower part of Figure 15, the accuracy rate of learning model 410 after 200 training cycles (see V1) is 85.8% (the average value of the three accuracy rates (correct answer rates for each of the three divided data) for the three validation data (divided data) in the three-fold cross-validation).

[0126] In Experiment 1, it can be seen that by adopting learning model 410 instead of learning model 910, the accuracy rate (correct answer rate) improved from 76.6% to 85.8%.

[0127] FIG. 16 shows the verification results for Experiment 1 (specifically, the verification results using the verification data after 200 training runs) in the form of a confusion matrix (2 × 2) for two-label classification. The upper table in FIG. 16 shows the verification results for learning model 910, and the lower table in FIG. 16 shows the verification results for learning model 410. In each table, the right and left columns are divided into columns based on whether the correct answer was "strong" or "weak," and the upper and lower rows are divided into columns based on whether the estimated value of the learning model was "strong" or "weak." The values ​​in each column represent the number (number of people), and columns with correct estimation results are surrounded by a bold line. Based on the upper table, an accuracy rate of 76.8% (= (25 + 18) / 56) was calculated, and based on the lower table, an accuracy rate of 85.7% (= (27 + 21) / 56) was calculated. The decimal point discrepancies from the accuracy rates shown in FIG. 15 are due to errors when calculating the average values.

[0128] <Results of Experiment 2> 17 is a graph showing the results of Experiment 2. FIG. 17 is a graph similar to FIG.

[0129] As shown in the upper part of Figure 17, the accuracy rate of the learning model 910 after 200 training cycles (see V9) is 69.0% (the average value of the three accuracy rates (correct answer rates for each of the three divided data) for the three validation data (divided data) in the three-fold cross-validation). On the other hand, as shown in the lower part of Figure 17, the accuracy rate of the learning model 410 after 200 training cycles (see V1) is 81.7% (the average value of the three accuracy rates (correct answer rates for each of the three divided data) for the three validation data in the three-fold cross-validation).

[0130] In Experiment 2, it can be seen that the accuracy rate (correct answer rate) improved from 69.0% to 81.7% by adopting learning model 410 instead of learning model 910. The reason why the accuracy rate in Experiment 2 was lower than that in Experiment 1 is thought to be because not only extreme data (data with correct labels of "0" and "4" for indentation strength D) but also other data (data with correct labels of "1" and "3" for indentation strength D) was used in Experiment 2.

[0131] FIG. 18 is a diagram similar to FIG. 16. In FIG. 18, the verification results for Experiment 2 (specifically, the verification results using verification data after 200 training runs) are shown in the form of a confusion matrix (2 × 2) for two-label classification. The upper table in FIG. 18 shows the verification results for learning model 910, and the lower table in FIG. 18 shows the verification results for learning model 410. Based on the upper table, an accuracy rate of 69.0% (= (52 + 64) / 168) is calculated, and based on the lower table, an accuracy rate of 81.5% (= (69 + 68) / 168) is calculated. Note that the discrepancy after the decimal point from the accuracy rate shown in FIG. 17 is due to errors in calculating the average value.

[0132] Moreover, here, a relatively small amount of data is used to train and verify (infer) the learning model 410. By using a relatively large amount of data to train the learning model 410, it is possible to train the learning model 410 more appropriately and further improve the accuracy rate of the learning model 410. Furthermore, by using a relatively large amount of data, even when the number of stages of the indentation strength D is increased to, for example, five stages, it is possible to train the learning model 410 appropriately and obtain a high accuracy rate of the learning model 410.

[0133] <7. Effects of the embodiment> In the above embodiment, in the learning stage of the learning model 410, the relationship between the paired images 210 and the indentation intensity D in the learning model 410 is machine-learned using training data configured to include the paired images 210 of each subject and the correct label for the indentation intensity D for that subject (see FIG. 4, etc.). In particular, the learning model 410 (420) is generated by machine learning the relationship between the paired images 210 and the indentation intensity D using training data configured to include the paired images 210 including not only the post-compression images 240 but also the pre-compression images 230 and the correct label for the indentation intensity D. Therefore, it is possible to generate a learning model 410 that can improve the accuracy of pitting edema evaluation by suppressing the influence of individual characteristics (compared to the learning model 910, etc., according to the comparative example). Consequently, it is possible to improve the accuracy of pitting edema evaluation by suppressing the influence of individual characteristics.

[0134] Specifically, the paired images 210, including not only the post-compression image 240 but also the pre-compression image 230, are input into the learning model 410 (420), and the evaluation value of the pitting edema of the subject (estimated value of the indentation intensity D) is calculated based on the output value obtained. Therefore, it is possible to improve the accuracy of the evaluation of pitting edema by suppressing the influence of individual characteristics. For example, it is possible to reduce the influence of morphological characteristics of the skin surface (wrinkle characteristics, etc.). Furthermore, when the pre-compression image 230 and the post-compression image 240 are color images, it is possible to reduce the influence of individual color-related characteristics (e.g., skin color, etc.).

[0135] Furthermore, evaluation based on the learning model 410 can eliminate variations due to the subjectivity of the evaluator, making it possible to perform objective evaluation.

[0136] 19 is a diagram showing four examples in which correct inference results (correct answers) were obtained in the above-described verification experiment by changing the learning model 910 to the learning model 410, in other words, by changing the input image from only the post-compression image 240 to the paired image 210 (both the pre-compression image 230 and the post-compression image 240). In FIG. 19, these four examples are divided into four rows, one above the other. The pre-compression image 230 is shown in the left column, and the post-compression image 240 is shown in the right column.

[0137] In all four cases, the learning model 910, which inputs only the post-compression image 240, did not obtain the correct inference result ("weak"), but instead obtained an incorrect inference result ("strong"). On the other hand, the learning model 410, which inputs the paired images 210 (both the pre-compression image 230 and the post-compression image 240), obtained the correct inference result ("weak").

[0138] In the post-compression image 240 in the top row (FIG. 19(a)) of Fig. 19, there is a "streak" (or "dimple") 240a (a morphological feature of the skin surface). It is considered that the learning model 910, to which only the post-compression image 240 is input, estimated that the indentation strength D was "strong" based on the streak 240a (in the post-compression image 240) (obtaining an incorrect judgment result).

[0139] In contrast, a "streak" (or "depression") 230a also exists in the pre-compression image 230 in the top row (FIG. 19(a)) of Fig. 19. It is considered that the learning model 410, to which the pre-compression image 230 is also input, estimated that the indentation strength D was "weak" (a correct answer was obtained) based on the presence of streaks (230a, 240a) in both the post-compression image 240 and the pre-compression image 230.

[0140] In detail, the learning model 410 is trained using training data relating to a combination of a pair of images 210 having streaks (230a, 240a) in both the post-compression image 240 and the pre-compression image 230 and a correct label of indentation intensity D "weak." Therefore, if streaks (230a, 240a) exist in both the post-compression image 240 and the pre-compression image 230, it is considered that the presence of the streaks alone will not determine the indentation intensity D as "strong" (rather, it will be determined as "weak").

[0141] Similarly, in the post-compression image 240 in the second row from the top (FIG. 19(b)) of Fig. 19, there is a "streak" 240b extending vertically. It is considered that the learning model 910, to which only the post-compression image 240 is input, estimated that the indentation strength D is "strong" based on the streak 240b (in the post-compression image 240).

[0142] In contrast, a "streak" 230b extending vertically also exists in the pre-compression image 230 in the second row from the top (FIG. 19(b)) of Fig. 19. It is considered that the learning model 410, to which the pre-compression image 230 is also input, estimated that the indentation strength D was "weak" (a correct answer was obtained) based on the presence of streaks (230b, 240b) in both the post-compression image 240 and the pre-compression image 230.

[0143] In the post-compression image 240 in the third row from the top of Fig. 19 (Fig. 19(c)), a "mole" 240c is present. It is considered that the learning model 910, to which only the post-compression image 240 is input, estimated that the indentation strength D is "strong" based on the mole 240c (in the post-compression image 240).

[0144] In contrast, a "mole" 230c also exists in the pre-compression image 230 in the third row from the top (FIG. 19(c)) of Fig. 19. It is considered that the learning model 410, to which the pre-compression image 230 is also input, estimated that the indentation strength D was "weak" (a correct answer was obtained) based on the presence of moles (230c, 240c) in both the post-compression image 240 and the pre-compression image 230.

[0145] In detail, the learning model 410 is trained using training data relating to a combination of a pair of images 210 having moles (230c, 240c) in both the post-compression image 240 and the pre-compression image 230, and a correct answer label of indentation intensity D "weak." Therefore, when moles (230c, 240c) are present in both the post-compression image 240 and the pre-compression image 230, it is considered that the presence of the streak alone will not determine the indentation intensity D as "strong" (rather, it will be determined as "weak").

[0146] In the post-compression image 240 in the bottom row (fourth row from the top) of Figure 19 (Figure 19(d)), there is a "mottled pattern" (black spots 240d, etc.) on the skin surface. It is considered that the learning model 910, to which only the post-compression image 240 is input, estimated that the indentation intensity D is "strong" based on the black spots 240d (in the post-compression image 240). In other words, it is considered that the indentation intensity D is estimated to be "strong" due to the existence of variations in brightness in the post-compression image 240.

[0147] In contrast, the pre-compression image 230 in the bottom row of Fig. 19 (Fig. 19(d)) also has a "mottled pattern" (black spots 230d, etc.) on the skin surface. It is considered that the learning model 410, to which the pre-compression image 230 is also input, estimated that the indentation intensity D is "weak" based on the presence of black spots (230d, 240d) ("mottled pattern") in both the post-compression image 240 and the pre-compression image 230 (indicating that the skin texture is not smooth). In other words, it is considered that the indentation intensity D is estimated to be "weak" (a correct judgment was obtained) based on the presence of variations in brightness in both the pre-compression image 230 and the post-compression image 240.

[0148] In detail, the learning model 410 is trained using training data relating to a combination of a pair of images 210 having black spots (230d, 240d) or the like in both the post-compression image 240 and the pre-compression image 230, and a correct answer label of indentation intensity D "weak." Therefore, if black spots (230d, 240d) or the like are present in both the post-compression image 240 and the pre-compression image 230, it is considered that the presence of the black spots alone does not determine that the indentation intensity D is "strong."

[0149] In this way, by using the above-mentioned learning model 410, it is possible to reduce the influence of wrinkle characteristics on the skin surface, suppress the influence of individual characteristics, and improve the accuracy of evaluating pitting edema.

[0150] Furthermore, in the above embodiment, when a doctor assigns a correct label for the indentation intensity D based on a predetermined standard (see FIG. 12 ), the indentation intensity D is assigned as a value with high objectivity. Then, by training the learning model 410 based on such an indentation intensity D, the objectivity of the output value (estimated value of the indentation intensity D) from the learning model 410 is improved. Therefore, an objective index that can be used in diagnosing heart failure, etc., can be (automatically) provided by evaluation (edema evaluation) using the learning model 410. Consequently, not only doctors but also non-doctor users (patients, etc.) can obtain an objective index (evaluation) regarding pitting edema.

[0151] Furthermore, by labeling the indentation strength D in five stages as shown in FIG. 12, it is possible to obtain a more precise index value (compared to when it is labeled in two stages).

[0152] <8. Modifications, etc.> Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described contents.

[0153] For example, in the above-described embodiment, pressure is applied to a region of the lower leg of the body using a finger, but this is not limited thereto, and pressure may be applied using, for example, a pressure tool or device.

[0154] In the above embodiment, the predetermined period M1 during which pressure is applied with the finger is 3 seconds, but this is not limiting. For example, the predetermined period M1 may be 5 seconds, 10 seconds, or another period of time.

[0155] Furthermore, in the above-described embodiment and the like, the pre-compression image 230 and the post-compression image 240 are extracted by the evaluation device 30 in steps S11 and S31, but this is not limiting, and the pre-compression image 230 and the post-compression image 240 may be extracted by the terminal device 70. The terminal device 70 may then transmit the pre-compression image 230 and the post-compression image 240, and the evaluation device 30 may receive and acquire the pre-compression image 230 and the post-compression image 240 transmitted from the terminal device 70.

[0156] In the above-described embodiment, the paired images 210 are captured using the terminal device 70, but this is not limiting. For example, the paired images 210 (the pre-compression image 230 and the post-compression image 240) may be captured using an evaluation device 30 further provided with an imaging unit.

[0157] In addition, in the above embodiments, the pre-compression image 230 and the post-compression image 240 are extracted from the moving image 200, but this is not limited to this, and the pre-compression image 230 and the post-compression image 240 may each be generated (captured) as a still image from the time of capture.

[0158] In the above-described embodiment, the inference result (estimated value of indentation strength D) by the learning model 410 is transmitted from the evaluation device 30 to the terminal device 70, and the inference result (estimated value of indentation strength D) is displayed on the terminal device 70 (display unit 75b). However, the present invention is not limited to this. For example, the inference result may be displayed on the evaluation device 30.

[0159] In the above-described embodiment, the evaluation device 30 executes both the learning process and the inference process, but this is not limiting. For example, the learning process (the process of generating a learning model) may be executed by a device other than the evaluation device 30.

[0160] Furthermore, the learning model 410 is not limited to the model (410A) shown in FIG. 13. For example, the learning model 410 may be a learning model 410B shown in FIG. 20. FIG. 20 illustrates a learning model 410B obtained by modifying the convolutional layer 451 (see FIG. 13) of the first deep learning model 450, which accepts three-channel input, to a convolutional layer 461 which accepts six-channel input. In other words, the learning model 410B is a model in which a new convolutional layer 461 is provided on the input side of a model (base model) 450B of the first deep learning model 450 excluding the convolutional layer 451. In the learning model 410B, the convolutional layer 461 accepts six-channel input, so there is no need to provide a separate convolutional layer 471. The learning model 410B may be trained by fine-tuning using the learning model 450B as a base model.

[0161] Alternatively, the learning model 410 may be a learning model 410C shown in FIG. 21 or the like. The learning model 410C in FIG. 21 has a structure in which two first deep learning models 450 (450C, 450D) arranged in parallel are connected by a fully connected layer 462 on the output side. Of the paired images 210, which are six-channel images, three-channel images (a pre-compression image 230 and a post-compression image 240) are input separately to the first deep learning models (base models) 450C, 450D. Specifically, the pre-compression image 230 is input to one first deep learning model 450C, and the post-compression image 240 is input to the other first deep learning model 450D. A fully connected layer 463 is provided to connect the output of one first deep learning model 450C with the output from the other first deep learning model 450D. The fully connected layer 463 is a layer that classifies the indentation strength D into multiple levels (e.g., five levels) based on two feature maps (e.g., feature vectors with a total of 2560 (= 1280 × 2) dimensions) output from the two first deep learning models 450C, 450D.

[0162] In the above embodiments, an efficient net (such as EfficientNet-B0) is mainly used as the first deep learning model 450 (base model), but the first deep learning model 450 is not limited to this. For example, various convolutional neural networks (CNNs) such as VGG16 and Residual Network (ResNet) (residual network) may be used as the first deep learning model 450.

[0163] In the above-described embodiment, the learning model 410 is trained using fine tuning, but this is not limiting. For example, the learning model 410 may be trained by machine learning using a normal learning method without using fine tuning.

[0164] In addition, although the above embodiments and the like have exemplified a classification model as the learning model, the learning model is not limited to this, and may be constructed as a regression model. In other words, the indentation strength D may be estimated as a continuous value (a real number (non-discrete value) having a value after the decimal point, such as "3.8").

[0165] In addition, in the above-described embodiments, the pre-compression image 230 and the post-compression image 240 are each a color image, but this is not limiting. For example, the pre-compression image 230 and the post-compression image 240 may each be a grayscale image (one-channel image). In this case, two-channel images may be used as the paired images 210 instead of six-channel images. [Explanation of symbols]

[0166] 1. Rating System 30 Evaluation equipment 70 Terminal Equipment 81 Footrest 82 footprints 200 Videos 210 paired images 230 Pre-compression image 240 Post-compression image 242 Some areas 410, 410B, 410C Learning Model 450, 450B, 450C, 450D base model 910 (Comparative Example) Learning Model

Claims

1. A method for producing a learning model for evaluating pitting edema of the lower leg of the body, a) For each of several subjects, a step of acquiring a pair of images consisting of a post-compression image after applying compression to a portion of the lower leg for a predetermined period of time and a pre-compression image before applying the compression, b) A step of obtaining the correct label for the indentation intensity in the post-compression image of each of the multiple subjects, c) A step of machine learning the relationship between the paired images and the indentation intensity in the learning model using training data comprising the paired images of each subject and the correct labels for the indentation intensity of each subject, A method for producing a learning model, characterized by comprising the following features.

2. The method for producing a learning model according to claim 1, characterized in that the pre-compression image and the post-compression image are both color images.

3. The pre-compression image and the post-compression image are, respectively, three-channel images in which each pixel has an RGB value. The method for producing a learning model according to claim 2, characterized in that the paired images are 6-channel images.

4. The method for producing a learning model according to claim 1, characterized in that the learning model is a deep learning model.

5. The aforementioned learning model is a deep learning model, The deep learning model includes a first deep learning model that accepts three channels of input, and a second deep learning model having a structure in which a further convolutional layer is added to the input side of the first deep learning model. The method for producing a learning model according to claim 3, characterized in that the further convolutional layer is a convolutional layer that performs a conversion from 6 channels to 3 channels.

6. The first deep learning model is a learning model that has been pre-trained using training data separate from the training data, The method for producing a learning model according to claim 5, characterized in that the second deep learning model is machine-trained by fine-tuning using the training data.

7. The method for producing a learning model according to claim 1, characterized in that the correct label is a value indicating which of a plurality of stages the indentation intensity is.

8. A trained model produced using the training model production method described in any one of claims 1 to 7.

9. A program for causing a computer to execute the learning model production method described in any one of claims 1 to 7.

10. An evaluation method for assessing pitting edema of the lower leg using a learning model, a) A step of obtaining a post-compression image and a pre-compression image of a portion of the lower leg of the person being evaluated, after applying compression for a predetermined period of time. b) A step of inputting the paired images, consisting of the pre-compression image and the post-compression image, into the learning model and calculating the evaluation value of the pitting edema of the person being evaluated based on the output value output from the learning model, An evaluation method characterized by comprising the following:

11. The evaluation method according to claim 10, characterized in that the learning model is a learning model that takes the paired images as input and outputs the indentation intensity in the compressed image, and is a learning model that has been trained using a plurality of training data consisting of the paired images of a plurality of subjects and the correct labels of the indentation intensity of each of the plurality of subjects.

12. The evaluation method according to claim 10, characterized in that the pre-compression image and the post-compression image are both color images.

13. A program for causing a computer to execute the evaluation method described in any one of claims 10 to 12.

14. A learning device, A control unit that performs machine learning to create a learning model for evaluating pitting edema in the lower leg of the body. Equipped with, The control unit, For each of the multiple subjects, a pair of images is obtained consisting of a post-compression image after applying compression to a portion of the lower leg for a predetermined period of time, and a pre-compression image before applying the compression. The correct labels for the indentation intensity in the post-compression images of each of the aforementioned multiple subjects are obtained. A learning device characterized by using training data comprising the paired images of each subject and the correct labels for the indentation intensity of each subject to machine learn the relationship between the paired images and the indentation intensity in the learning model.

15. An evaluation device, A control unit that calculates the evaluation value for pitting edema in the lower leg of the person being evaluated. Equipped with, The control unit, After applying compression to a portion of the lower leg of the person being evaluated for a predetermined period of time, a post-compression image and a pre-compression image are obtained. An evaluation device characterized by inputting paired images, consisting of a pre-compression image and a post-compression image, into a learning model and calculating an evaluation value for the pitting edema of the person being evaluated based on the output value output from the learning model.

Images