Trained model generation method, trained model generation system, inference device, inference system, and computer program

Abstract

To automate an evaluation of whether an examinee suffers from delirium or not while reducing the degree of contribution of personal subjectivity.SOLUTION: First data D1 corresponding to a dynamic image in which an examinee 41 is captured is acquired. Second data corresponding to specular changes at relative positions of a plurality of feature points in a body of the examinee 41 captured in the dynamic image is generated on the basis of the first data D1. Third data D3 indicating a result of determination of whether the examinee 41 suffers from delirium performed based on the dynamic image is generated. A neural network learns using the second data D2 and the third data D3, thereby generating a learnt model M applied to an inference device for inferring a probability that the examinee suffers from delirium on the basis of the dynamic image in which the examinee is captured.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a method for generating a trained model to be applied to an inference device that infers the probability that a subject has developed delirium based on a video image that includes the subject, and a system for generating the trained model.The present invention also relates to a computer program executable in the system.The present invention also relates to the inference device and a computer program executable by a processor of the inference device.The present invention also relates to an inference system including the inference device and an image processing device that processes the video image. [Background technology]

[0002] Patent Document 1 discloses a technique for detecting whether a subject is suffering from delirium based on an image in which the subject is captured. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2014-528314 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to automate the evaluation of whether a subject has developed delirium while reducing the contribution of individual subjectivity. [Means for solving the problem]

[0005] A first aspect for achieving the above object is a method for generating a trained model that is applied to an inference device that infers a probability that a subject has developed delirium based on a video image that includes the subject, the method comprising: acquiring first data corresponding to a moving image in which the subject is captured; generating second data corresponding to a change over time in relative positions of a plurality of feature points on the body of the subject captured in the moving image based on the first data; generating third data indicating a determination result of whether the subject has developed delirium based on the moving image; generating a trained model by training a neural network using the second data and the third data; Contains:

[0006] A second aspect for achieving the above object is a system for generating a trained model to be applied to an inference device that infers a probability that a subject has developed delirium based on a video image in which the subject is captured, the system comprising: an image processing device that generates, based on first data corresponding to a moving image in which the subject is captured, second data corresponding to a change over time in relative positions of a plurality of feature points on the body of the subject captured in the moving image; a model generation device that generates a trained model by training a neural network using the second data and third data indicating a determination result of whether the subject has developed delirium, the determination result being made based on the video; and It is equipped with:

[0007] A third aspect for achieving the above object is a computer program executable in a system for generating a trained model to be applied to an inference device that infers a probability that a subject has developed delirium based on a video image in which the subject is captured, the computer program comprising: By being executed, an image processing device included in the system generates, based on first data corresponding to a moving image in which the subject is captured, second data corresponding to a change over time in relative positions of a plurality of feature points on the body of the subject captured in the moving image; A model generation device included in the system generates a trained model by training a neural network using the second data and third data indicating the judgment result of whether the subject has developed delirium based on the video image.

[0008] Delirium is observed in many patients in intensive care units and is associated with increased mortality, prolonged hospital stays, and long-term cognitive impairment. However, the onset of the condition tends to be underdiagnosed in clinical practice. Although standard assessment methods such as the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU) are available, assessment still relies on the subjectivity of the individual assessor.

[0009] According to the configurations of the first to third aspects, by preparing a large number of sets of subject behaviors and judgment results related to the onset of delirium and training a neural network, a trained model can be generated as an algorithm that can automate the judgment related to the onset of delirium while reducing the contribution of individual subjectivity. By implementing such a trained model in an inference device, it is possible to automate the evaluation of whether a subject has developed delirium while reducing the contribution of individual subjectivity.

[0010] Furthermore, the second data input to the neural network together with the third data as training data corresponds to the change over time in the relative positions of multiple feature points on the subject's body extracted from the video images acquired by the imaging device, and therefore contains a significantly reduced amount of information compared to the first data corresponding to the video images, thereby enabling efficient learning using data sets obtained from a large number of subjects while suppressing an increase in processing load.

[0011] A fourth aspect for achieving the above object is an inference device, comprising: an input interface that receives behavior data corresponding to changes over time in relative positions of a plurality of feature points on the body of a subject captured in a moving image that is generated based on image data corresponding to the moving image in which the subject is captured; a processor that inputs the behavior data into a trained model generated by the generation method according to the first aspect, thereby obtaining inference data corresponding to the probability that the subject has developed delirium; an output interface for outputting the inference data; It is equipped with:

[0012] A fifth aspect for achieving the above object is a computer program executable by a processor of an inference device, comprising: When executed, the inference device receiving behavior data corresponding to changes over time in relative positions of a plurality of feature points on the body of the subject captured in a moving image generated based on image data corresponding to the moving image in which the subject is captured; By inputting the behavior data into a trained model generated by the generation method according to the first aspect, inference data corresponding to the probability that the subject has developed delirium is obtained; The inference data is output.

[0013] A sixth aspect of the present invention for achieving the above object is an inference system, comprising: an image processing device that generates behavior data corresponding to changes over time in relative positions of a plurality of feature points on the body of a subject captured in a moving image based on image data corresponding to the moving image in which the subject is captured; an inference device that inputs the behavior data into a trained model generated by the generation method according to the first aspect, thereby acquiring inference data corresponding to the probability that the subject has developed delirium, and outputs the inference data; It is equipped with:

[0014] According to the configurations of the fourth to sixth aspects, an inference device incorporating a trained model receives behavioral data indicating changes over time in the relative positions of multiple feature points on the body of a subject captured in a video captured by an imaging device. The behavioral data contains significantly less information than the image data corresponding to the video. This reduces the processing load required to infer the probability that the subject has developed delirium.

[0015] According to the configurations of each of the first to sixth aspects, rather than the moving image itself in which the subject appears, information relating to the change over time in the relative positions of multiple feature points on the subject's body extracted from the moving image is used for learning and inference, thereby respecting the subject's privacy. [Brief explanation of the drawings]

[0016] [Figure 1] 1 illustrates an example of the configuration of an inference system according to an embodiment. [Figure 2] 1 illustrates an example of the configuration of a model generation system according to an embodiment. [Figure 3] 3 illustrates a specific configuration of the model generation system of FIG. 2. [Figure 4] 3 illustrates a skeletal model that may be used by the image processing devices of FIGS. 1 and 2; [Figure 5] 3 illustrates an example facial expression detection model that may be used by the image processing devices of FIGS. 1 and 2. DETAILED DESCRIPTION OF THE INVENTION

[0017] Examples of embodiments are described in detail below with reference to the accompanying drawings.

[0018] 1 illustrates an example of the configuration of an inference system 10 according to one embodiment. The inference system 10 is configured to infer the probability that a subject 20 has developed delirium based on a video image in which the subject 20 appears.

[0019] The inference system 10 includes an imaging device 11. The imaging device 11 is a device that acquires moving images in which the subject 20 is captured and outputs image data IM corresponding to the moving images. An example of the imaging device 11 is a video camera that is sensitive at least in the visible wavelength range. If a video camera that is sensitive in the near-infrared wavelength range is used, moving images of the subject 20 can be acquired even at night. A range imaging camera or a thermal imaging camera can also be used.

[0020] The inference system 10 includes an image processing device 12. The image processing device 12 is configured to generate behavior data BH corresponding to changes over time in the relative positions of multiple feature points on the body of a subject 20 captured in a moving image acquired by an imaging device 11, based on image data IM.

[0021] Specifically, the image processing device 12 performs processing on the image data IM to detect predetermined feature points on the body of the subject 20 that appear in each of the multiple frame images that make up the moving image. In the example shown in Fig. 1, the multiple feature points detected are the hands and head of the subject 20. By acquiring the positions of the multiple feature points for the multiple frame images, information indicating changes over time in the relative positions of the feature points can be obtained. The behavior data BH indicates this information.

[0022] Reasoning system 10 includes reasoning device 13. Reasoning device 13 has input interface 131, processor 132, and output interface 133.

[0023] The input interface 131 is configured to receive the behavior data BH generated by the image processing device 12. The behavior data BH may be received in real time from the image processing device 12 via wired or wireless communication, or may be received non-real time via a storage medium in which the behavior data BH is stored.

[0024] The processor 132 is configured to perform processing to acquire the probability that the subject 20 has developed delirium by inputting the behavior data BH into the trained model M. The trained model M is an inference algorithm generated through machine learning using a neural network, which will be described later. The trained model M is configured to receive the behavior data BH corresponding to the behavior of the subject 20 as input, and to output the probability that the subject 20 has developed delirium as an inference result. The inference result may be associated with a score (e.g., any value from 1 to 5) corresponding to the inferred probability.

[0025] The processor 132 is configured to output inference data IF corresponding to the probability inferred by the trained model M from the output interface 133.

[0026] The inference system 10 includes a notification device 14. The notification device 14 is configured to notify the user of the result of inference by the trained model M based on the inference data IF received from the inference device 13. The notification is performed using at least one of a visual notification, an auditory notification, and a tactile notification.

[0027] The trained model M described above is generated by a model generation system 30 illustrated in Fig. 2. That is, the model generation system 30 is configured to generate the trained model M to be applied to the inference device 13 described above.

[0028] The model generation system 30 includes an imaging device 31. The imaging device 31 is a device that acquires moving images in which the subject is captured and outputs first data D1 corresponding to the moving images. An example of the imaging device 31 is a video camera that is sensitive at least in the visible wavelength range. If a video camera that is sensitive in the near-infrared wavelength range is used, moving images of the subject can be acquired even at night. A range imaging camera or a thermal imaging camera can also be used.

[0029] The model generation system 30 includes an image processing device 32. The image processing device 32 is configured to generate, based on the first data D1, second data D2 corresponding to changes over time in the relative positions of a plurality of feature points on the body of the subject captured in the moving image acquired by the imaging device 31.

[0030] Specifically, the image processing device 32 performs a process on the first data D1 to detect predetermined feature points on the subject's body that appear in each of the multiple frame images that make up the moving image. In the example shown in Fig. 2, the multiple feature points detected are the subject's hands and head. By acquiring the positions of the multiple feature points for the multiple frame images, information indicating changes over time in the relative positions of the feature points can be obtained. The second data D2 indicates this information.

[0031] The acquisition of first data D1 and the generation of second data D2 are performed for multiple subjects 41 to 4n. Note that first data D1 for the same subject acquired at different times is treated as having been acquired from different subjects. As a result, first data D1 for n subjects is acquired and second data D2 for n subjects is generated. The first data D1 for n subjects may be acquired by the same imaging device 31 or may be acquired by multiple different imaging devices 31.

[0032] Note that an appropriate normalization process may be used to convert information relating to the positional relationships between multiple feature points, which are originally defined in a three-dimensional space, into information relating to their relative positions in a two-dimensional image. Furthermore, when the first data D1 for n people is acquired by multiple image capture devices 31, a process for normalizing different image capture environments between the devices may also be performed as appropriate.

[0033] The second data D2 generated for each of the plurality of subjects 41 to 4n is configured to include information indicating from which subject the first data D1 was acquired and on which basis the second data D2 was generated.

[0034] The model generation system 30 includes a teacher data generation device 33. The teacher data generation device 33 is configured to generate third data D3 indicating the result of a determination made based on the moving images acquired by the imaging device 31 as to whether each of the plurality of subjects 41 to 4n has developed delirium.

[0035] As illustrated in FIG. 3, the teacher data generation device 33 includes an input interface 331, a user interface 332, a processor 333, and an output interface 334.

[0036] The input interface 331 is configured to receive the first data D1 output from the imaging device 31.

[0037] The user interface 332 is configured to accept a judgment JG made by the medical professional 50 based on the video corresponding to the first data D1 as to whether the subject appearing in the video is experiencing delirium. The judgment JG may be provided as a binary option of "has delirium occurred" or "has not occurred," or may be provided as a delirium assessment score in accordance with a predetermined standard. Examples of such standards include the Confusion Assessment Method for the Intensive Care Unit (CAM-ICU) and the Intensive Care Delirium Screening Checklist (ICDSC). The judgment JG may be input via an input device such as a keyboard or a mouse, or may be input using voice recognition or gesture recognition technology.

[0038] A medical professional who is familiar with delirium patients can input a judgment JG as to whether a subject is suffering from delirium by referring only to the video image. However, it is often difficult to judge whether a subject is suffering from delirium by referring only to the video image. Therefore, it is preferable that medical professionals input a judgment JG as to whether a subject is suffering from delirium by referring not only to the video image of the subject but also to the subject's background information (medication information, gender, age, etc.).

[0039] The processor 333 is configured to generate third data D3 by associating the judgment JG input through the user interface 332 with the first data D1. As a result, third data D3 for n subjects is generated based on the first data D1 for n subjects. In other words, the third data D3 is configured to include information indicating from which subject the first data D1 acquired was used to determine whether the subject has developed delirium.

[0040] The output interface 334 is configured to output the third data D3 generated by the processor 333.

[0041] 2 and 3, the model generation system 30 includes a model generation device 34. The model generation device 34 includes an input interface 341, a processor 342, and an output interface 343.

[0042] The input interface 341 is configured to receive the second data D2 from the image processing device 32 and the third data D3 from the teacher data generation device 33.

[0043] The processor 342 is configured to train a neural network using the second data D2 and the third data D3, thereby generating a trained model M. The trained model M is generated as a processing algorithm that receives as input data corresponding to changes over time in the relative positions of multiple feature points on the body of a subject captured in a moving image acquired by an imaging device, and outputs data corresponding to the probability that the subject has developed delirium.

[0044] Well-known supervised learning techniques are appropriately used to train the neural network. That is, the third data D3 is used as training data to generate the trained model M. The third data D3 indicates what changes over time in the relative positions of multiple feature points on the subject's body indicate whether the subject is suffering from delirium (or not). By identifying the second data D2 and the third data D3 that have information related to the same first data D1, the second data D2 and the third data D3 derived from a common video are associated with each other.

[0045] The output interface 343 is configured to output the trained model M generated by the processor 342 in a form that can be implemented in the inference device 13.

[0046] Delirium is observed in many patients in intensive care units and is associated with increased mortality, prolonged hospital stays, and long-term cognitive impairment. However, signs of delirium tend to be underdiagnosed in clinical practice. Although standard assessment methods such as the CAM-ICU are available, the reality is that assessment still relies on the subjectivity of the individual assessor.

[0047] According to the above configuration, by preparing a large number of sets of judgment results related to the subject's behavior and the onset of delirium and training the neural network, it is possible to generate a trained model M as an algorithm that can automate judgments related to the onset of delirium while reducing the contribution of individual subjectivity. By implementing such a trained model M in the inference device 13, it is possible to automate the evaluation of whether a subject has developed delirium while reducing the contribution of individual subjectivity.

[0048] Furthermore, the second data D2 input to the neural network together with the training data corresponds to the change over time in the relative positions of multiple feature points on the subject's body extracted from the video images acquired by the imaging device 31, and therefore contains a much smaller amount of information than the first data D1 corresponding to the video images. Therefore, it is possible to efficiently learn using data sets obtained from a large number of subjects while suppressing an increase in processing load.

[0049] As described with reference to Figure 1, behavior data BH indicating changes over time in the relative positions of multiple feature points on the body of subject 20 captured in moving images acquired by imaging device 11 is input to inference device 13 equipped with trained model M. The behavior data BH contains significantly less information than image data IM corresponding to the moving images. This makes it possible to suppress an increase in the processing load required to infer the probability that subject 20 has developed delirium.

[0050] Furthermore, since the information used for learning and inference is not the video image itself in which the subject appears, but rather information related to the changes over time in the relative positions of multiple feature points on the subject's body extracted from the video image, the subject's privacy can be respected.

[0051] The image processing device 32 of the model generation system 30 can be configured to generate second data D2 by applying a skeletal model SM exemplified in Fig. 4 to first data D1. The expression "applying a skeletal model" used in this specification means detecting multiple feature points defined in the skeletal model from the body of the subject captured in a moving image acquired by the imaging device 31, and connecting the multiple feature points with multiple skeletal connecting lines defined in the skeletal model.

[0052] The skeletal model SM includes a left shoulder feature point LU1, a left elbow feature point LU2, and a left wrist feature point LU3. The left shoulder feature point LU1 corresponds to the left shoulder of the model human body. The left elbow feature point LU2 corresponds to the left elbow of the model human body. The left wrist feature point LU3 corresponds to the left wrist of the model human body. The left shoulder feature point LU1 and the left elbow feature point LU2 are connected by a skeleton connection line. The left elbow feature point LU2 and the left wrist feature point LU3 are connected by a skeleton connection line.

[0053] The skeletal model SM includes a right shoulder feature point RU1, a right elbow feature point RU2, and a right wrist feature point RU3. The right shoulder feature point RU1 corresponds to the right shoulder of the model human body. The right elbow feature point RU2 corresponds to the right elbow of the model human body. The right wrist feature point RU3 corresponds to the right wrist of the model human body. The right shoulder feature point RU1 and the right elbow feature point RU2 are connected by a skeleton connection line. The right elbow feature point RU2 and the right wrist feature point RU3 are connected by a skeleton connection line.

[0054] The skeletal model SM includes a left hip feature point LL1, a left knee feature point LL2, and a left ankle feature point LL3. The left hip feature point LL1 corresponds to the left part of the hip of the model human body. The left knee feature point LL2 corresponds to the left knee of the model human body. The left ankle feature point LL3 corresponds to the left ankle of the model human body. The left hip feature point LL1 and the left knee feature point LL2 are connected by a skeletal connection line. The left knee feature point LL2 and the left ankle feature point LL3 are connected by a skeletal connection line.

[0055] The skeletal model SM includes a right hip feature point RL1, a right knee feature point RL2, and a right ankle feature point RL3. The right hip feature point RL1 corresponds to the right part of the hip of the model human body. The right knee feature point RL2 corresponds to the right knee of the model human body. The right ankle feature point RL3 corresponds to the right ankle of the model human body. The right hip feature point RL1 and the right knee feature point RL2 are connected by a skeletal connection line. The right knee feature point RL2 and the right ankle feature point RL3 are connected by a skeletal connection line.

[0056] The skeletal model SM includes a facial feature point F and a neck feature point N. The facial feature point F corresponds to the face of the model human body. The neck feature point N corresponds to the neck of the model human body. The neck feature point N is connected to each of the facial feature point F, left shoulder feature point LU1, right shoulder feature point RU1, left waist feature point LL1, and right waist feature point RL1 by skeleton connection lines.

[0057] By applying such a skeletal model SM to the first data D1, second data D2 that more accurately describe the subject's behavior can be generated. As a result of using such second data D2, a trained model M can be generated that can improve the inference of the probability that the subject is experiencing delirium.

[0058] The skeletal model SM illustrated in FIG. 4 is merely an example. A well-known skeletal model can be used, in which the number and positions of feature points and skeletal connection lines are appropriately determined depending on the behavior of the subject to be detected. However, the skeletal model used to generate the second data D2 in the image processing device 32 of the model generation system 30 and the skeletal model used to generate the behavior data BH in the image processing device 12 of the inference system 10 must be the same. That is, when the skeletal model SM of FIG. 4 is used in the image processing device 32, the image processing device 12 generates the behavior data BH by applying the skeletal model SM to the image data IM. This allows the trained model M to provide inference results that reflect the learning content using the skeletal model SM.

[0059] In addition to or instead of the skeletal model SM, a facial expression detection model EM as illustrated in FIG. 5 may be used. The facial expression detection model EM is used in a well-known image recognition technique that detects facial expressions based on changes over time in the relative positions of multiple feature points set on a model face. That is, multiple feature points for generating second data D2 may be set on the face of the subject. In FIG. 5, black circles represent each feature point.

[0060] It is known that the facial expression of the subject is an important factor in assessing delirium. With the above configuration, by generating a trained model M using second data D2 containing information related to the subject's facial expression, it is possible to automate the assessment of delirium based on video images of the facial expression of the subject 20 acquired by the imaging device 11.

[0061] As illustrated in FIG. 2, the model generation system 30 may include a sensor 35. The sensor 35 is configured to output fourth data D4 corresponding to at least one of the subject's biometric information, the subject's body movement, and the subject's voice. Examples of the biometric information include body temperature, blood pressure, heart rate, and transcutaneous arterial oxygen saturation (SpO2). The subject's body movement may be measured by an acceleration sensor or a vibration sensor attached to the subject or a bed used by the subject. The subject's voice may be acquired by an imaging device 31 or by a sound-collecting microphone attached to the subject or a bed used by the subject.

[0062] The detection of various pieces of information by the sensor 35 is performed in parallel with the acquisition of moving images of the subject by the imaging device 31. As a result, fourth data D4 for n people is generated in parallel with the acquisition of first data D1 for n people. The fourth data D4 is configured to include information indicating from which subject the first data D1 was acquired and together with which the fourth data D4 was generated.

[0063] The fourth data D4 is input to the model generating device 34. That is, the input interface 341 of the model generating device 34 can be configured to also accept the fourth data D4.

[0064] The processor 342 of the model generation device 34 may be configured to train the neural network using the fourth data D4 in addition to the second data D2 and the third data D3. By identifying the second data D2, the third data D3, and the fourth data D4 that have information related to the same first data D1, the second data D2, the third data D3, and the fourth data D4 that originate from a common video are correlated with each other.

[0065] This configuration strengthens the correlation between the subject's behavior during training and delirium assessment. In particular, when multiple feature points on the subject's body cannot be accurately detected from video images, the information acquired through sensor 35 can play a supplementary role.

[0066] In this case, the inference system 10 is configured to include a sensor 15 having the same configuration as the sensor 35 described above. The sensor 15 is configured to output supplementary data SP corresponding to the fourth data D4. The supplementary data SP is received by an input interface 131 of the inference device 13. The processor 132 of the inference device 13 is configured to input the supplementary data SP, in addition to the behavior data BH, into the trained model M to obtain inference data IF.

[0067] According to this configuration, the trained model M can be made to provide an inference result that reflects the learning content that also uses the fourth data D4.

[0068] In addition to or instead of the above-described sensor 35, the model generation system 30 may include a background information input device 36. The background information input device 36 is provided as a user interface that accepts input of background information of the subject from a medical professional. Examples of the background information include the subject's gender, age, height, weight, medical history, medication information, etc. The background information input device 36 is configured to generate fifth data D5 including the background information.

[0069] The generation of the fifth data D5 by the background information input device 36 is performed prior to or in parallel with the acquisition of moving images of the subject by the imaging device 31. As a result, the fifth data D5 for n people is generated in association with the acquisition of the first data D1 for n people. The fifth data D5 is configured to include information indicating the subject from whom the first data D1 was acquired and together with which the fifth data D5 was generated.

[0070] The fifth data D5 is input to the model generating device 34. That is, the input interface 341 of the model generating device 34 can be configured to also accept the fifth data D5.

[0071] The processor 342 of the model generation device 34 may be configured to train the neural network using the fifth data D5 in addition to the second data D2 and the third data D3. By identifying the second data D2, the third data D3, and the fifth data D5 that have information related to the same first data D1, the second data D2, the third data D3, and the fifth data D5 that originate from a common video are correlated with each other.

[0072] With this configuration, information that does not directly appear in the subject's behavior can be reflected in learning by the neural network.

[0073] In this case, inference system 10 is configured to include a background information input device 16 having the same configuration as background information input device 36 described above. Background information input device 16 is configured to output background data BG corresponding to fifth data D5. The background data BG is received by input interface 131 of inference device 13. Processor 132 of inference device 13 is configured to input background data BG in addition to behavior data BH into trained model M to obtain inference data IF.

[0074] According to this configuration, the trained model M can be made to provide an inference result that reflects the learning content that also uses the fifth data D5.

[0075] The inference system 10 illustrated in FIG. 1 and the model generation system 30 illustrated in FIG. 2 are provided as independent systems, but may share parts as needed.

[0076] For example, image data IM output from the imaging device 11 of the inference system 10 can be received by the teacher data generation device 33 of the model generation system 30. In this case, a medical professional 50 can input a judgment JG as to whether a subject 20 appearing in a moving image corresponding to the image data IM is suffering from delirium into the user interface 332. When inputting the judgment JG, it does not matter whether or not it is necessary to refer to the inference result by the inference device 13. In other words, the teacher data generation device 33 generates third data D3 based on the image data IM.

[0077] The third data D3 generated based on the image data IM is received by the input interface 341 of the model generation device 34. On the other hand, the behavior data BH generated by the image processing device 12 of the inference system 10 is also received by the input interface 341.

[0078] With this configuration, the processor 342 of the model generation device 34 can cause the neural network to perform new learning related to delirium assessment using a combination of the behavior data BH and the third data D3 generated based on video images acquired by the imaging device 11 of the inference system 10. Because the behavior data BH is provided for inference by the inference device 13, the inference algorithm of the inference device 13 can be strengthened or modified through the new learning.

[0079] It should be noted that instead of the configuration in which the behavior data BH output from the image processing device 12 of the inference system 10 is input to the model generation device 34, a configuration in which the image data IM output from the imaging device 11 of the inference system 10 is input to the image processing device 32 of the model generation system 30 may also be adopted. In this case, the second data D2 and third data D3 generated based on the image data IM are used for learning by the neural network.

[0080] The processor 333 of the teacher data generation device 33, the processor 342 of the model generation device 34, and the processor 132 of the inference device 13, each having the above-described functions, may be realized by a general-purpose microprocessor operating in cooperation with general-purpose memory. Examples of general-purpose microprocessors include a CPU, an MPU, and a GPU. Examples of general-purpose memory include a ROM and a RAM. In this case, the ROM may store a computer program that executes the above-described process. The ROM is an example of a non-transitory computer-readable medium that stores a computer program. The general-purpose microprocessor specifies at least a portion of the program stored in the ROM, expands it on the RAM, and executes the above-described process in cooperation with the RAM. The computer program may be pre-installed in the general-purpose memory or may be downloaded from an external server via a communication network and then installed in the general-purpose memory. In this case, the external server is an example of a non-transitory computer-readable medium that stores a computer program.

[0081] The processor 333 of the teacher data generation device 33, the processor 342 of the model generation device 34, and the processor 132 of the inference device 13, each having the above-described functions, may be realized by a dedicated integrated circuit capable of executing the above-described computer program, such as a microcontroller, an ASIC, or an FPGA. In this case, the computer program is pre-installed in a memory element included in the dedicated integrated circuit. The memory element is an example of a computer-readable medium storing a computer program. The processor 333 of the teacher data generation device 33, the processor 342 of the model generation device 34, and the processor 132 of the inference device 13, each having the above-described functions, may also be realized by a combination of a general-purpose microprocessor and a dedicated integrated circuit.

[0082] When the teacher data generation device 33 and the model generation device 34 are provided as devices independent of each other, the output interface 334 of the teacher data generation device 33 and the input interface 341 of the model generation device 34 may be connected to allow wired or wireless communication. That is, the output interface 334 and the input interface 341 may be physical communication interfaces.

[0083] The teacher data generation device 33 and the model generation device 34 may be functional entities realized within the same device. In this case, at least a part of the functions of the processor 333 of the teacher data generation device 33 may be realized by the processor 342 of the model generation device 34. Furthermore, the output interface 334 and the input interface 341 may be logical interfaces.

[0084] When the inference device 13 and the notification device 14 are provided as independent devices, the output interface 133 may be a physical interface that mediates data communication between them. The processor 132 and the notification device 14 may be functional entities realized within the same control device. In this case, the output interface 133 may be a logical interface.

[0085] The above-described embodiments are merely examples for facilitating understanding of the present invention, and the configurations according to the above-described embodiments may be appropriately modified or improved without departing from the spirit and scope of the present invention.

[0086] The image data IM, behavior data BH, supplementary data SP, and background data BG handled in the inference system 10 illustrated in FIG. 1 may each be stored in a storage device (not shown). The storage device may be realized by a semiconductor memory device, a hard disk drive device, a magnetic tape device, or the like. The storage device may be provided within the inference device 13, or may be provided within a cloud server device with which the inference device 13 can communicate data via a communication network. When the storage device is provided within the cloud server device, data may be uploaded and edited from each of the multiple imaging devices 11, multiple image processing devices 12, multiple sensors 15, and multiple background information input devices 16 that are connectable to the communication network.

[0087] The first data D1, second data D2, third data D3, fourth data D4, and fifth data D5 handled in the model generation system 30 illustrated in FIG. 2 may be stored in a storage device (not shown). The storage device may be realized by a semiconductor memory device, a hard disk drive device, a magnetic tape device, or the like. The storage device may be provided within the inference device 13, or within a cloud server device with which the model generation device 34 can communicate data via a communication network. When the storage device is provided within the cloud server device, data may be uploaded and edited from each of the multiple imaging devices 31, multiple image processing devices 32, multiple teacher data generation devices 33, multiple sensors 35, and multiple background information input devices 36 that are connectable to the communication network. [Explanation of symbols]

[0088] 10: inference system, 12: image processing device, 13: inference device, 131: input interface, 132: processor, 133: output interface, 30: model generation system, 32: image processing device, 33: training data generation device, 34: model generation device, 41 to 4n: subject, D1: first data, D2: second data, D3: third data, D4: fourth data, D5: fifth data, BG: background data, BH: behavior data, EM: facial expression detection model, IF: inference data, IM: image data, M: trained model, SM: skeletal model, SP: supplementary data

Claims

1. A method for generating a trained model to be applied to an inference device that infers the probability that a subject has developed delirium based on a video image in which the subject is captured, comprising: acquiring first data corresponding to a moving image in which the subject is captured; generating second data corresponding to a change over time in relative positions of a plurality of feature points on the body of the subject captured in the moving image based on the first data; generating third data indicating a determination result of whether the subject has developed delirium based on the moving image; generating a trained model by training a neural network using the second data and the third data; It contains the second data is generated by applying to the first data a skeletal model in which the plurality of feature points set for at least the four limbs of a human body are connected based on a predetermined relationship; How to generate a trained model.

2. The second data is generated by applying a facial expression detection model, in which the plurality of feature points are set on a face, to the first data. The method for generating a trained model according to claim 1.

3. acquiring fourth data corresponding to at least one of biometric information acquired from the subject, body movement information of the subject, and voice information of the subject; The trained model is generated by training the neural network using the fourth data. A method for generating a trained model according to claim 1 or 2.

4. acquiring fifth data corresponding to at least one of background information of the subject; The neural network is trained using the fifth data to generate the trained model. A method for generating a trained model according to any one of claims 1 to 3.

5. A system for generating a trained model to be applied to an inference device that infers the probability that a subject has developed delirium based on a video image in which the subject is captured, an image processing device that generates, based on first data corresponding to a moving image in which the subject is captured, second data corresponding to a change over time in relative positions of a plurality of feature points on the body of the subject captured in the moving image; a model generation device that generates a trained model by training a neural network using the second data and third data indicating a determination result of whether the subject has developed delirium, the determination result being made based on the video; and It is equipped with the second data is generated by applying to the first data a skeletal model in which the plurality of feature points set for at least the four limbs of a human body are connected based on a predetermined relationship; A system for generating trained models.

6. A computer program executable in a system for generating a trained model to be applied to an inference device that infers the probability that a subject has developed delirium based on a video image in which the subject is captured, By being executed, an image processing device included in the system generates, based on first data corresponding to a moving image in which the subject is captured, second data corresponding to a change over time in relative positions of a plurality of feature points on the body of the subject captured in the moving image; a model generation device included in the system generates a trained model by training a neural network using the second data and third data indicating a determination result of whether the subject has developed delirium, the determination result being made based on the video image; and the second data is generated by applying to the first data a skeletal model in which the plurality of feature points set for at least the four limbs of a human body are connected based on a predetermined relationship; Computer program.

7. an input interface that receives behavior data corresponding to changes over time in relative positions of a plurality of feature points on the body of a subject captured in a moving image that is generated based on image data corresponding to the moving image in which the subject is captured; a processor that inputs the behavior data into a trained model generated by the trained model generation method according to claim 1 or 2, thereby acquiring inference data corresponding to the probability that the subject has developed delirium; an output interface for outputting the inference data; Equipped with Reasoning device.

8. An input interface that accepts supplementary data including behavior data corresponding to changes over time in the relative positions of multiple feature points on the body of the subject captured in a video image generated based on image data corresponding to the video image in which the subject is captured, as well as at least one of biometric information obtained from the subject, body movement information of the subject, and audio information of the subject; a processor that inputs the supplementary data in addition to the behavioral data into a trained model generated by the trained model generation method of claim 3, thereby obtaining inference data corresponding to the probability that the subject has developed delirium; Equipped with Reasoning device.

9. An input interface that receives behavior data corresponding to changes over time in the relative positions of multiple feature points on the body of the subject captured in a moving image generated based on image data corresponding to the moving image in which the subject is captured, and background data including background information of the subject; a processor that inputs the background data in addition to the behavior data into a trained model generated by the trained model generation method of claim 4, thereby obtaining inference data corresponding to the probability that the subject has developed delirium; Equipped with Reasoning device.

10. 1. A computer program executable by a processor of a reasoning device, comprising: When executed, the inference device receiving behavior data corresponding to changes over time in relative positions of a plurality of feature points on the body of the subject captured in a moving image generated based on image data corresponding to the moving image in which the subject is captured; The behavioral data is input into a trained model generated by the trained model generation method according to claim 1 or 2, thereby obtaining inference data corresponding to the probability that the subject has developed delirium; outputting the inference data; Computer program.

11. an image processing device that generates behavior data corresponding to changes over time in relative positions of a plurality of feature points on the body of a subject captured in a moving image based on image data corresponding to the moving image in which the subject is captured; an inference device that inputs the behavior data into a trained model generated by the trained model generation method according to claim 1 or 2, thereby acquiring inference data corresponding to the probability that the subject has developed delirium, and outputs the inference data; Equipped with Inference system.

12. a user interface for inputting a determination result as to whether the subject has developed delirium, the determination result being made by a medical professional based on a video corresponding to the image data; a teacher data generation device that generates the third data based on the determination result; Equipped with The inference system of claim 11.

Images