Diagnosis Support System

Abstract

To grasp a symptom of a disease of a patient at an early stage.SOLUTION: A diagnosis support system according to an embodiment comprises an acquisition unit, an expression recognition unit, an index generation unit and a display control unit. The acquisition unit continuously acquires recorded face information regarding the face of a patient. The expression recognition unit recognizes an expression of the patient on the basis of the face information. The index generation unit generates an index showing time-series variation of the expression by using the face information and medical care data of the patient. The display control unit displays the index and the expression on a display unit with time-series variation of the medical data of the patient.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The embodiments disclosed in the present specification and drawings relate to a diagnosis support system. [Background technology]

[0002] In patient care, it is necessary to identify early signs of serious illnesses. For example, it is known that the prognosis of heart failure is greatly influenced by whether treatment can be administered at an early stage. Therefore, early treatment of heart failure is important for reducing the mortality rate from heart failure. Traditionally, in intensive care units (ICUs), neonatal intensive care units (NICUs), and hospital wards, changes in vital signs such as increased respiratory rate, heart rate, and blood pressure have been used as signs of heart failure. However, it is difficult to reliably detect signs of diseases such as heart failure based on changes in vital signs alone. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2018-43003 Summary of the Invention [Problem to be solved by the invention]

[0004] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to detect early signs of a patient's disease. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described below can also be positioned as other problems. [Means for solving the problem]

[0005] A diagnosis support system according to an embodiment includes an acquisition unit, a facial expression recognition unit, an index generation unit, and a display control unit. The acquisition unit acquires continuously recorded facial information about a patient's face. The facial expression recognition unit recognizes the patient's facial expression based on the facial information. The index generation unit generates an index indicating a time-series change in facial expression using the facial information and the patient's medical data. The display control unit displays the index and the facial expression together with the time-series change in the patient's medical data on the display unit. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a diagnosis support system according to the first embodiment. [Figure 2] FIG. 2 is a flowchart illustrating a processing procedure of the diagnosis support processing by the diagnosis support system according to the first embodiment. [Figure 3] FIG. 3 is a diagram showing an example of a diagnosis support screen displayed by the diagnosis support system according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a display screen displayed by the diagnosis support system according to the first modified example of the first embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of the configuration of a diagnosis support system according to the second embodiment. [Figure 6] FIG. 6 is a flowchart illustrating a processing procedure of a diagnosis support process by the diagnosis support system according to the second embodiment. [Figure 7] FIG. 7 is a diagram showing an example of a display screen displayed by the diagnosis support system according to the second embodiment. [Figure 8] FIG. 8 is a diagram illustrating an example of the configuration of a diagnosis support system according to a first modified example of the second embodiment. [Figure 9] FIG. 9 is a diagram illustrating an example of a combination of inputs and outputs of a trained model according to a first modified example of the second embodiment. [Figure 10] FIG. 10 is a flowchart illustrating a processing procedure of a diagnosis support process by a diagnosis support system according to a first modified example of the second embodiment. [Figure 11] FIG. 11 is a diagram showing an example of a display screen displayed by the diagnosis support system according to the first modified example of the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, an embodiment of a medical image processing apparatus will be described in detail with reference to the drawings. In the following description, components having substantially the same functions and configurations are designated by the same reference numerals, and redundant description will be given only when necessary.

[0008] (First embodiment) 1 is a diagram showing the configuration of a diagnosis support system 100. The diagnosis support system 100 is connected to a Hospital Information System (hereinafter referred to as HIS) 300, an examination device 400, and a Radiology Information System (hereinafter referred to as RIS) 500 via a network 200. The diagnosis support system 100 may also be connected to a workstation, a Picture Archiving and Communication System (PACS), etc. via the network 200.

[0009] The network 200 is, for example, a LAN (Local Area Network). Connection to the network 200 may be either a wired connection or a wireless connection. Furthermore, as long as security is ensured by a VPN (Virtual Private Network) or the like, the connection line is not limited to a LAN. Connection to a public communication line such as the Internet may also be possible.

[0010] The HIS 300 manages information related to medical facilities such as hospitals. The HIS 300 stores electronic medical records (EMR), personal health records (PHR), and other information in a storage device. The HIS 300 also stores information about patients (hereinafter referred to as patient information), medical images of patients, and other information. The patient information includes the patient's name, gender, age, nationality, medical history, information about tests, and examination results. The medical images include images taken using a general camera and test images obtained during tests.

[0011] The inspection device 400 is, for example, a medical imaging diagnostic device such as an X-ray computed tomography (CT) device, a magnetic resonance imaging (MRI) device, an ultrasonic diagnostic device, or an X-ray diagnostic device, or a blood test device.

[0012] The RIS 500 manages information related to radiological examination work. Information related to radiological examination work includes patient information, order information, image data, etc. The image data includes information about images (hereinafter referred to as examination images) previously acquired by examinations using the examination device 400. The examination images are associated with imaging conditions, etc.

[0013] The diagnosis support system 100 can transmit and receive various information to and from the HIS 300, the examination device 400, the RIS 500, and the like via the network 200. The diagnosis support system 100 continuously records information relating to the patient's facial expression (hereinafter referred to as facial information) using a means for recognizing the patient's facial expression, and displays information relating to the transition of the patient's facial expression in chronological order along with other medical data. By checking the transition of the displayed facial expression, a doctor can quickly identify signs of a serious disease such as heart failure. Below, an example will be described in which the diagnosis support system 100 is used to identify signs of heart failure.

[0014] The diagnostic support system 100 includes a diagnostic support device 10. The diagnostic support device 10 is a terminal device used by doctors for diagnosis in, for example, an intensive care unit (ICU), a neonatal intensive care unit (NICU), or a hospital ward. The diagnostic support device 10 includes a memory 11, a communication interface 12, a display 13, an input interface 14, and a processing circuit 15. Note that, although the diagnostic support device 10 will be described below as a single device that executes multiple functions, the multiple functions may be executed by separate devices. For example, the functions executed by the diagnostic support device 10 may be distributed and installed on different console devices or workstation devices.

[0015] The memory 11 is a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or an integrated circuit that stores various information. The memory 11 may also be a portable storage medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a flash memory, in addition to an HDD or SSD. The memory 11 may also be a drive device that reads and writes various information from and to semiconductor memory elements such as flash memory and RAM (Random Access Memory). The storage area of the memory 11 may be located within the diagnostic support device 10 or in an external storage device connected via a network.

[0016] The memory 11 stores programs executed by the processing circuit 15, various data used in the processing of the processing circuit 15, etc. As the programs, for example, a program that is installed in advance on a computer from a network or a non-transitory computer-readable storage medium and causes the computer to realize each function of the processing circuit 15 is used. Note that the various data handled in this specification are typically digital data. The memory 11 is an example of a storage unit.

[0017] The communication interface 12 is a network interface that controls transmission of communications with the HIS 300, the inspection device 400, the RIS 500, and other external devices via the network 200.

[0018] The display 13 displays various types of information. For example, the display 13 outputs medical information generated by the processing circuitry 15, a GUI (Graphical User Interface) for receiving various operations from an operator, etc. For example, the display 13 is a liquid crystal display or a CRT (Cathode Ray Tube) display.

[0019] The display 13 also displays information about the transition of the patient's facial expression in chronological order along with other medical data. At this time, the display 13 displays the information about the transition of the patient's facial expression in chronological order on the screen on which the medical data is displayed. The display 13 is an example of a display unit.

[0020] The input interface 14 accepts various input operations from the operator, converts the accepted input operations into electrical signals, and outputs them to the processing circuitry 15. For example, the input interface 14 accepts input of medical information, input of various command signals, etc. from the operator. The input interface 14 is realized by a mouse, keyboard, trackball, switch buttons, a touch screen integrating a display screen and a touchpad, a non-contact input circuit using an optical sensor, a voice input circuit, etc., for performing various processes in the processing circuitry 15. The input interface 14 is connected to the processing circuitry 15 and converts input operations received from the operator into electrical signals and outputs them to the control circuit. Note that, in this specification, the input interface is not limited to those equipped with physical operating components such as a mouse and keyboard. For example, an electrical signal processing circuit that receives electrical signals corresponding to input operations from an external input device provided separately from the device and outputs the electrical signals to the processing circuitry 15 is also an example of an input interface. The input interface 14 is an example of an input unit.

[0021] The processing circuitry 15 controls the overall operation of the diagnosis support device 10. The processing circuitry 15 is a processor that executes an acquisition function 151, a facial expression recognition function 152, and a display control function 153 by calling and executing programs in the memory 11. The processing circuitry 15 that realizes each of the acquisition function 151, the facial expression recognition function 152, and the display control function 153 is an example of an acquisition unit, a facial expression recognition unit, and a display control unit, respectively.

[0022] 1, the acquisition function 151, the facial expression recognition function 152, and the display control function 153 are described as being implemented by a single processing circuit 15. However, the processing circuit may be configured by combining multiple independent processors, and each processor may execute a program to implement each function. Also, the acquisition function 151, the facial expression recognition function 152, and the display control function 153 may be implemented as individual hardware circuits. The above description of each function executed by the processing circuit 15 applies to the following embodiments and modifications.

[0023] Although the diagnosis support device 10 is described as a single console that executes multiple functions, the multiple functions may be executed by separate devices. For example, the functions of the processing circuitry 15 may be distributed and installed in different devices.

[0024] The term "processor" used in the above description refers to a circuit such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC, a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)). The processor realizes its function by reading and executing a program stored in the memory 11. Note that instead of storing the program in the memory 11, the program may be directly embedded in the processor circuit. In this case, the processor realizes its function by reading and executing the program embedded in the circuit. Note that each processor in this embodiment is not limited to being configured as a single circuit, but may be configured as a single processor by combining multiple independent circuits to realize its function. Furthermore, multiple components in FIG. 1 may be integrated into a single processor to realize its function. The above description of "processor" also applies to the following embodiments and modifications.

[0025] The processing circuit 15 acquires medical data of a patient using the acquisition function 151. The medical data is acquired, for example, from the HIS 300, the RIS 500, etc. via the network 200. The medical data includes information about the patient's vital activities (hereinafter referred to as vital information), information about medications administered in the past (hereinafter referred to as medication information), and information about tests (hereinafter referred to as test information). The vital information is information about vital signs such as heart rate, respiratory rate, blood pressure, body temperature, and percutaneous arterial oxygen saturation (SpO2). The medication information includes, for example, the names of medications administered in the past, administration periods, and dosages. The test information includes, for example, the names of tests performed in the past, test conditions, test dates and times, and test results. The processing circuit 15 realizing the acquisition function 151 may be called a vital record acquisition means, a medication record acquisition means, a test record acquisition means, or the like.

[0026] Furthermore, the processing circuit 15 acquires continuously recorded information about the patient's face (hereinafter referred to as face information) using the acquisition function 151. The face information is acquired, for example, from the HIS 300 or the like via the network 200. The face information includes, for example, an image of the patient's face (hereinafter referred to as a face image) and the date and time the face image was captured. The face image may be one whose facial expression, such as facial color, can be identified using known facial expression recognition software. The face image may be, for example, a still image such as a photograph taken using a general-purpose camera, or may be a video. The face image may also be a thermographic image. For example, face images may be taken of hospitalized patients at regular intervals, or of outpatients at each visit to the hospital.

[0027] The processing circuit 15 recognizes the patient's facial expression based on the facial information using the facial expression recognition function 152. Specifically, for each continuously recorded facial image, the processing circuit 15 detects the patient's facial expression and facial color from the facial image and determines whether the facial expression indicates a bad mood based on the detection result. The facial expression indicates, for example, a smiling face or a frowning face. For example, the processing circuit 15 determines whether the facial expression indicates a bad mood by determining the degree of bad mood based on the patient's facial expression and facial color. Being in a bad mood means being in a bad mood, being short, or feeling uncomfortable. The degree of bad mood is an index indicating the degree of bad mood. The determination result of the degree of bad mood may be, for example, two types: bad mood or not. Alternatively, the determination result of the degree of bad mood may be classified into three or more types depending on the degree of bad mood. For example, if a bad face and a frowning face are detected, the processing circuit 15 determines that the patient's facial expression is a bad mood and determines that the patient was in a bad mood when the facial image was captured. The method for determining the degree of bad mood from the facial image may be, for example, a known facial expression detection algorithm for determining facial expressions from the facial image. The determination result regarding the degree of bad mood is stored in the memory 11 in association with the facial image.

[0028] The processing circuitry 15 causes various pieces of information to be displayed on the display 13 by the display control function 153. The processing circuitry 15 also generates a screen to assist a doctor in making a diagnosis (hereinafter referred to as a diagnostic support screen) and displays the generated diagnostic support screen on the display 13. The diagnostic support screen displays information necessary for diagnosing a patient. For example, a diagnostic support screen used in an ICU, NICU, or ward displays vital signs, medication information, test information, and the like as medical data. The diagnostic support screen may also display the patient's chief complaint, progress information, nursing records, the name of the current illness, medical history, family composition, social history, and the like.

[0029] Furthermore, the processing circuitry 15 causes the display control function 153 to display the patient's facial expression in chronological order along with the patient's medical data on the diagnostic support screen of the display 13. For example, the processing circuitry 15 causes the determination results as to whether the facial expression indicates bad mood to be displayed in chronological order on a time axis. Alternatively, the processing circuitry 15 causes the determination results of the patient's degree of bad mood to be displayed in chronological order on a time axis.

[0030] Next, the operation of the diagnostic support processing executed by the diagnostic support system 100 will be described. The diagnostic support processing is processing for recognizing changes in a patient's facial expression based on facial information and displaying the changes in facial expression in chronological order together with the patient's medical data. FIG. 2 is a flowchart showing an example of the procedure of the diagnostic support processing. In FIG. 2, as an example, a case is described in which a patient's facial image is used as facial information to support a doctor in diagnosing signs of heart failure. Note that the processing procedures in each process described below are merely examples, and each process can be modified as appropriate as possible. Furthermore, steps in the processing procedures described below can be omitted, replaced, or added as appropriate depending on the embodiment.

[0031] (Diagnosis support processing) (Step S101) The processing circuit 15 uses the acquisition function 151 to acquire, from the HIS 300, facial images of patients that have been acquired in the past and the dates on which the facial images were taken.

[0032] (Step S102) The processing circuitry 15 acquires vital information, medication information, and test information from the HIS 300 as medical data of the patient using the acquisition function 151.

[0033] (Step S103) The processing circuit 15 detects the patient's facial color and facial expression for each of the acquired facial images using the facial expression recognition function 152, and determines the patient's level of bad mood based on the detection results. The determination results are stored in the memory 11 in association with the date the corresponding facial image was taken.

[0034] (Step S104) The processing circuit 15 generates a diagnostic support screen using the display control function 153 and displays it on the display 13. FIG. 3 is a diagram showing an example of the diagnostic support screen. As shown in FIG. 3, the diagnostic support screen includes a patient information display unit 610, a management screen display unit 620 for managing the flow of medical care, a vital information display unit 630, a drug information display unit 640, and a test result display unit 650. The patient information display unit 610 displays patient information. The displayed patient information includes the patient ID, patient name, age, sex, disease name, etc. The management screen display unit 620 displays, for example, information related to events, tests, document data, and nursing care on a time axis. The vital information display unit 630 displays, for example, the progression of the patient's vital signs. The drug information display unit 640 displays, for example, the name of the administered drug and the administration period. The test result display unit 650 displays, for example, test results of a specimen test or the like.

[0035] The diagnostic support screen also includes a facial expression display section 660. The facial expression display section 660 displays the transition of the patient's facial expression. As the transition of the patient's facial expression, for example, the transition of the determination result of the patient's bad mood level is displayed. As the determination result of the bad mood level, in FIG. 3, an icon indicating the patient's bad mood level is displayed on the time axis. Here, either an icon 661 indicating that the patient is not in a bad mood or an icon 662 indicating that the patient is in a bad mood is displayed at a position corresponding to the date on which the face image was captured. The doctor can understand the transition of the patient's facial expression by checking the icons 661 and 662 indicating the bad mood level displayed on the diagnostic support screen.

[0036] The method of displaying the bad mood level determination result is preferably a highly visible method such as icons 661 and 662, but is not limited to this. For example, instead of icons 661 and 662, the bad mood level determination result may be displayed in text, or a symbol indicating the determination result may be displayed.

[0037] The effects of the diagnosis support system 100 according to this embodiment will be described below.

[0038] In patient care, it is necessary to identify early signs of serious illnesses. For example, it is known that the prognosis of heart failure is greatly influenced by whether treatment can be administered at an early stage. Therefore, early treatment of heart failure is important for reducing the mortality rate from heart failure. Traditionally, in intensive care units (ICUs), neonatal intensive care units (NICUs), and hospital wards, changes in vital signs such as increased respiratory rate, heart rate, and blood pressure have been used as signs of heart failure. However, it is difficult to reliably detect signs of diseases such as heart failure based on changes in vital signs alone.

[0039] For this reason, during medical examinations, changes in the patient's facial expression are important as a primary complaint. In particular, when examining patients who cannot communicate verbally, the primary complaint can only be obtained from facial information, so changes in the patient's facial expression are particularly important. For example, if the patient is an infant who cannot speak, an intellectually disabled patient, a patient with a neurological disease, or a patient wearing an artificial respirator, the doctor cannot communicate with the patient verbally. Furthermore, if the patient is a foreign national, the doctor cannot communicate with the patient verbally. For this reason, it is preferable that the severity of the patient's condition, as estimated from the patient's facial expression and complexion, be utilized as an important finding in medical information.

[0040] However, when a doctor other than the patient's attending physician examines the patient, there is a possibility that the doctor may not be able to notice changes in the patient's facial expression. Therefore, when a doctor other than the patient examines the patient, there is a possibility that the doctor may miss signs of heart failure. For example, even if a doctor other than the patient judges that there is no problem with the patient's facial expression, the patient's facial expression may actually be more displeased than usual.

[0041] The diagnosis support system 100 according to this embodiment can acquire continuously recorded facial information about a patient's face, recognize the patient's facial expression based on the facial information, and display the recognized facial expression in chronological order along with the patient's medical data. The medical data can include, for example, vital signs, medication information, and test information.

[0042] With the above configuration, the diagnosis support system 100 according to this embodiment allows doctors to confirm qualitative changes in the patient's facial expression displayed on the screen, even for patients whose chief complaints are difficult to identify, and obtain important findings. Furthermore, by displaying the facial expression changes along with other medical data in chronological order, the display screen's visibility and readability are improved, allowing doctors to easily understand the progression of the patient's condition. Doctors can then review the displayed medical data with the patient's chief complaint in mind and make comprehensive judgments, allowing them to provide appropriate medical care. In other words, by checking the patient's facial expression changes, doctors can quickly respond to changes in vital signs that occur after the change in facial expression, thereby achieving more rapid medical care.

[0043] Furthermore, with the diagnosis support system 100 according to this embodiment, the transition of a patient's facial expression can be confirmed as data, so even doctors other than the patient's attending physician can notice changes in the patient's facial expression. Then, by using a change in the patient's facial expression as an opportunity to perform tests for serious diseases such as heart failure, the possibility of early detection of the disease increases. In this way, even when the attending physician is absent, the transition of the patient's facial information as well as vital signs can be confirmed over time, allowing a non-attending physician to make a judgment with the same quality as the attending physician. This allows for appropriate medical treatment and increases the rate of disease avoidance.

[0044] In this way, according to the diagnosis support system 100 of this embodiment, by displaying the transition of the patient's facial expression as a confirmation element, it becomes possible to grasp early signs of a patient's disease.

[0045] In recent years, it has become clear that a patient's bad mood can be used as an indicator of the patient's signs of illness. In the diagnosis support system 100 according to this embodiment, a facial image of the patient's face can be used as facial information. The system can also detect the patient's facial expression and complexion from the facial image, determine whether the facial expression indicates bad mood based on the detection results of the facial expression and complexion, and display the determination results in chronological order.

[0046] Therefore, the diagnosis support system 100 according to this embodiment can recognize the patient's facial expression from the patient's facial image and determine whether the patient is in a bad mood. By checking the progress of the determination results regarding whether the patient is in a bad mood, even a doctor other than the patient's attending physician can notice that the patient is in a worse mood than usual. By performing an examination when the patient is in a worse mood than usual, serious diseases such as heart failure can be discovered at an early stage.

[0047] Furthermore, according to the diagnostic support system 100 of this embodiment, when the facial expression indicates a bad mood, the degree of bad mood of the facial expression can be further determined based on the facial image, and the determination results of the degree of bad mood can be displayed in chronological order.

[0048] Therefore, according to the diagnosis support system 100 of this embodiment, by displaying the transition of the patient's level of bad mood, it is possible to grasp the change in the patient's facial expression more accurately. For example, even for a patient whose facial expression is usually judged to be bad-tempered, by observing the transition of the level of bad mood, even a doctor who is not in charge of the patient can notice that the patient is in a worse mood than usual.

[0049] (First Modification of the First Embodiment) A first modified example of the first embodiment will be described. This modified example is obtained by modifying the configuration of the first embodiment as follows. Descriptions of the configuration, operation, and effects that are the same as those of the embodiment will be omitted. In the diagnostic support system 100 according to this modified example, the processing circuitry 15 uses the display control function 153 to display only the determination results when the facial expression indicates a displeased mood.

[0050] 4 is a diagram showing an example of a diagnostic support screen displayed on the display 13 in the diagnostic support processing executed by the diagnostic support system 100 of this modified example. As shown in Fig. 4, in the facial expression display section 660 of the diagnostic support screen, only an icon 662 indicating that the patient is in a bad mood is displayed in a position corresponding to the date the face image was taken.

[0051] In this modification, the doctor can also grasp the transition of the patient's facial expression by checking the icon 662 indicating the degree of bad mood displayed on the diagnostic support screen. Moreover, in this modification, the determination result as to whether or not the patient is in a bad mood is displayed only when the patient is in a bad mood, so the doctor can easily notice that the patient's facial expression has changed to a bad mood, and can efficiently check information useful for diagnosis.

[0052] (Second embodiment) A second embodiment will be described. This embodiment is a modification of the configuration of the first embodiment as follows. Descriptions of configurations, operations, and effects that are the same as those of the first embodiment will be omitted. The diagnosis support system 100 according to this embodiment detects signs of serious diseases such as heart failure based on changes in the patient's facial expression and medical data, and issues a warning when such signs are detected.

[0053] 5 is a diagram showing the configuration of a diagnosis support system 100 of this embodiment. In addition to the functions described in the first embodiment, the processing circuitry 15 executes an index generation function 154. The processing circuitry 15 that realizes the index generation function 154 is an example of an index generation unit.

[0054] Processing circuitry 15 uses the facial information and the patient's medical data to generate an index indicating a time-series change in the patient's facial expression using index generation function 154. Specifically, processing circuitry 15 predicts symptoms that may occur based on medical data such as the expected disease name and the name of the disease at the time of hospitalization, and generates information about the predicted symptoms as an index indicating a time-series change in the patient's facial expression. For example, processing circuitry 15 predicts symptoms that may occur by reading from memory 11 a table indicating combinations of the expected disease name and the name of the disease at the time of hospitalization with symptoms that may occur. The index indicating a time-series change in the patient's facial expression is, for example, a warning indicating that a sign of a symptom has been detected.

[0055] The predicted symptom is a symptom that is likely to occur. The likely symptom is, for example, a disease such as heart failure. Alternatively, the likely symptom is a symptom of a specific disease such as vomiting, vomiting blood, fainting, or convulsions. Hereinafter, as an example, a case will be described in which heart failure is predicted as a likely symptom and a warning indicating that a symptom of heart failure has been detected is generated using an indicator indicating a time-series change in the patient's facial expression. In this case, the processing circuit 15 first detects the symptom of heart failure using facial information and medical data. The processing circuit 15 first detects the symptom of heart failure using, for example, a change in facial expression, a drop in vital signs, administration of a specific drug, a change in test values, and a change in test images as detection elements. Specifically, the processing circuit 15 detects the symptom of heart failure using the medical data when the patient's facial expression in the facial image is determined to be in a bad mood. A known heart failure symptom detection algorithm is used to detect the symptom of heart failure by inputting vital sign information, drug information, and test information. For example, an increase in body fluid volume is a symptom of heart failure. In this case, even if there is no cause affecting metabolism, such as a change in medication, a symptom of continuous weight gain appears over a relatively short period of time. Another symptom of heart failure is an increase in resting respiratory volume, which is a compensatory symptom for oxygen saturation deficiency associated with a decrease in cardiac output. In this case, an increase in resting respiratory volume appears as a symptom.

[0056] Thereafter, the processing circuitry 15 issues a warning when a sign of heart failure is detected. For example, the processing circuitry 15 displays a warning indicating that a sign of heart failure has been detected on a diagnostic support screen displayed on the display 13. The warning may be issued by displaying a warning message on the display 13, by displaying an icon indicating that a sign of heart failure has been detected on the display 13, or by audio notification. Another method of issuing the warning may be by superimposing a warning bar indicating that a sign of heart failure has been detected on a time axis displayed on the display 13 at a position corresponding to the date and time of detection. The processing circuitry 15 causes the display control function 153 to display the generated index and the patient's facial expression together with time-series changes in the patient's medical data on the diagnostic support screen of the display 13. For example, the processing circuitry 15 displays, on the time axis, the determination result as to whether the facial expression indicates a bad mood or not, and a warning indicating that a sign of heart failure has been detected, along with the time-series changes in the medical data.

[0057] (Diagnosis support processing) Next, a description will be given of the operation of the diagnosis support processing executed by the diagnosis support system 100 of this embodiment. Fig. 6 is a flowchart showing an example of the procedure of the diagnosis support processing according to this embodiment.

[0058] The processing in steps S201 to S204 is similar to the processing in steps S101 to S104 in FIG. 2, respectively, and therefore a description thereof will be omitted.

[0059] (Step S205) The processing circuitry 15 determines whether or not there is a facial image in which the patient is determined to be in a bad mood in the processing of step S203 using the index generation function 154. If there is no facial image in which the patient is determined to be in a bad mood (step S205-No), the processing circuitry 15 ends the series of processes of the diagnostic support processing.

[0060] (Step S206) If there is a face image determined to be in a bad mood (step S205-Yes), the processing circuitry 15 acquires vital information, medication information, and test information using the index generation function 154, and executes a process for detecting signs of heart failure.

[0061] (Step S207) The processing circuitry 15 determines whether or not a sign of heart failure has been detected by the index generation function 154. If a sign of heart failure has not been detected (step S207-No), the processing circuitry 15 ends the series of steps of the diagnostic support process.

[0062] (Step S208) If a symptom of heart failure is detected (step S207-Yes), the processing circuitry 15 causes the index generating function 154 to display on the diagnostic support screen that a symptom of heart failure has been detected. FIG. 7 is a diagram showing an example of the diagnostic support screen displayed in this embodiment. In FIG. 7, a warning bar 670 indicating that a symptom of heart failure has been detected is displayed in each display section of the diagnostic support screen at a position corresponding to the date and time when the symptom of heart failure was detected. By checking the display of the warning bar 670, a doctor can easily understand that a symptom of heart failure has been detected.

[0063] The following describes the effects of the diagnosis support system 100 according to this embodiment. The diagnosis support system 100 according to this embodiment has the following effects in addition to the effects described in the first embodiment.

[0064] The diagnosis support system 100 according to this embodiment can predict a patient's symptoms using facial information and the medical data continuously acquired from the facial information and the medical data, and generate information about the predicted symptoms as an index showing time-series changes in the patient's facial expression. Specifically, it can detect signs of symptoms and display a warning indicating the detection of the signs on a diagnosis screen. For example, if the facial expression is determined to be unhappy, it can detect signs of heart failure using the medical data.

[0065] With the above configuration, the diagnosis support system 100 according to this embodiment can utilize information on facial expression changes in addition to related medical data as an important time-series index for diagnosis. For example, by detecting signs of heart failure when a patient is determined to be in a bad mood, it is possible to reliably capture changes in vital signs that occur after a change in facial expression, enabling faster medical care.

[0066] Furthermore, the diagnosis support system 100 according to this embodiment can issue a warning when a sign of heart failure is detected.

[0067] (First modified example of the second embodiment) A first modified example of the second embodiment will be described. This modified example is obtained by modifying the configuration of the second embodiment as follows. Descriptions of the same configuration, operation, and effects as those of the embodiment will be omitted. The diagnosis support system 100 according to this modified example calculates the probability of occurrence of a symptom of concern as an index based on the transition of the patient's facial expression and medical data, and displays the calculation result together with the medical data.

[0068] FIG. 8 is a diagram showing the configuration of a diagnosis support system 100 of this embodiment. In this modification, the memory 11 stores a plurality of trained models 111. Each of the trained models 111 is a trained machine learning model. Each of the trained models 111 is a parameterized composite function obtained by combining a plurality of functions. The parameterized composite function is defined by a combination of a plurality of adjustable functions and parameters. Each of the trained models 111 may be any parameterized composite function that meets the above requirements, but is assumed to be a multi-layer network model.

[0069] A trained model 111 is prepared for each symptom that is likely to occur. The trained model 111 calculates the probability of a symptom occurring (hereinafter referred to as occurrence probability) for a symptom inferred from a disease name assumed from medical data or a disease name at the time of hospitalization. Specifically, the trained model 111 receives input of a facial image, patient information, and medical data, and outputs the occurrence probability of the corresponding symptom. The medical data input to the trained model 111 includes, for example, the disease name at the time of hospitalization, vital signs information, nursing records, etc. FIG. 9 is a diagram showing combinations of inputs and outputs of the trained model 111 corresponding to heart failure. For example, the trained model 111 corresponding to heart failure receives input of a facial image, patient information, and medical data, and outputs the occurrence probability of heart failure. Similarly, the trained model 111 corresponding to vomiting receives input of a facial image, patient information, and medical data, and outputs the occurrence probability of vomiting.

[0070] When training the trained model 111, reinforcement learning is performed using, for example, facial images, patient information, and medical data of past patients, and the duration of symptoms that the patients developed, as input data.

[0071] The processing circuitry 15 uses the index generation function 154 to predict symptoms that may occur based on medical data such as the assumed disease name and the disease name at the time of hospitalization, and reads out a trained model corresponding to the predicted symptoms from among multiple trained models 111 stored in the memory 11. Next, the processing circuitry 15 inputs a facial image, patient information, and medical data into the read trained model 111, causing the trained model 111 to output the probability of occurrence of the symptoms. The processing circuitry 15 then generates the occurrence probability output from the trained model 111 as an index.

[0072] The processing circuit 15 causes the display control function 153 to display the calculated occurrence probability together with the patient's medical data on the diagnostic support screen of the display 13.

[0073] (Diagnosis support processing) Next, the operation of the diagnostic support processing executed by the diagnostic support system 100 of this modified example will be described. Fig. 10 is a flowchart showing an example of the procedure of the diagnostic support processing according to this modified example. The processing of steps S301-S303 is the same as the processing of steps S101-S103 in Fig. 2, respectively, and therefore description thereof will be omitted.

[0074] (Step S304) The processing circuitry 15 uses the index generation function 154 to select symptoms that may occur based on the patient's medical data, and reads out a trained model corresponding to the predicted symptoms from among multiple trained models 111 stored in the memory 11. Here, we will explain the case where the trained model 111 corresponding to heart failure is selected and read out from the memory 11. The processing circuitry 15 inputs a face image, patient information, and medical data into the trained model 111 corresponding to heart failure, causing the trained model 111 corresponding to heart failure to output the probability of heart failure occurring.

[0075] (Step S305) The processing circuitry 15 uses the display control function 153 to display the probability of occurrence of heart failure and the patient's facial expression on a diagnostic support screen that displays time-series changes in the patient's medical data. FIG. 11 is a diagram showing an example of the diagnostic support screen displayed in this modification. As shown in FIG. 11, in this modification, the diagnostic support screen further includes a probability display section 680. The probability display section 680 displays the calculated probability of occurrence of heart failure on a time axis.

[0076] If multiple symptoms are predicted in the process of step S304, multiple trained models 111 corresponding to the multiple symptoms are read out, and the occurrence probability of the corresponding symptom is calculated in each of the multiple trained models 111. In this case, the probability display unit 680 displays the multiple symptoms. The probability of occurrence is displayed on the time axis.

[0077] The following describes the effects of the diagnosis support system 100 according to this modification. The diagnosis support system 100 according to this modification has the following effects in addition to the effects described in the second embodiment.

[0078] The diagnosis support system 100 according to this modification can generate an index showing time-series changes in a patient's facial expression using face information and the patient's medical data. Specifically, the system can predict a patient's symptoms using the face information and the patient's medical data, calculate the probability of occurrence of the predicted symptoms, and generate the calculation result as an index showing time-series changes in the patient's facial expression. The predicted probability of occurrence of the symptoms and the face information can then be displayed together with the patient's medical data.

[0079] With the above configuration, the diagnosis support system 100 according to this modification can analyze facial information, patient information, and medical data, calculate the likelihood of a symptom predicted from the assumed disease name or the disease name at the time of admission, and display the calculation result together with the facial information and medical data. By checking the probability of the predicted symptom occurring together with the patient's facial information, a doctor can obtain information about symptoms that may occur before the patient's condition suddenly worsens. This allows for early response to symptoms that may occur, enabling more rapid medical care.

[0080] With the above-described configuration, the diagnosis support system 100 according to this embodiment allows a doctor to easily confirm that a sign of heart failure has been detected by checking the warning, thereby preventing the sign of heart failure from being overlooked.

[0081] According to at least one of the embodiments described above, symptoms of a patient's disease can be detected early.

[0082] Although several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0083] 100...Diagnosis support system 200…Network 300...Hospital Information System (HIS) 400...Inspection equipment 500...Radiology Information System (RIS) 10...Diagnosis support device 11...Memory 111...Trained model 12...Communication interface 13...Display 14...Input interface 15...Processing circuit 151...Acquisition function 152...Facial expression recognition function 153...Display control function 154…Indicator generation function 610...Patient information display section 620…Management screen display section 630...Vital information display section 640...Drug information display section 650...Test result display section 660...Facial expression display section 661, 662...Icons 670...Warning bar 680…Probability display section

Claims

1. an acquisition unit for acquiring continuously recorded facial information relating to the patient's face; an expression recognition unit that recognizes a facial expression of a patient based on the facial information and determines whether the facial expression indicates displeasure; an index generating unit that, when there is face information determined to be in a bad mood, predicts a symptom of the patient using the patient's medical data, and generates a warning indicating that a sign of the symptom has been detected or an index indicating the probability of the symptom occurring, and, when there is no face information determined to be in a bad mood, does not predict the symptom; a display control unit that displays the index and the facial expression on a display unit together with time-series changes in the medical data of the patient; A diagnostic support system comprising:

2. The medical data includes vital information, medication information, and test information. The diagnostic support system according to claim 1 .

3. The facial information includes a facial image of the patient's face. The diagnosis support system according to claim 1 or 2.

4. the facial expression recognition unit detects the facial expression and complexion of the patient from the facial image, and determines whether the facial expression indicates a bad mood based on the detection result; The index indicates a time series change in the bad mood judgment result. The diagnostic support system according to claim 3 .

5. If the facial expression indicates displeasure, the facial expression recognition unit further determines a degree of displeasure of the facial expression based on the face image; The index indicates a time series change in the degree of bad mood. The diagnostic support system according to claim 4.

6. the display control unit causes the display unit to display only the facial expression indicating the displeased mood. The diagnosis support system according to claim 4 or 5.

7. the face information includes a facial image of the patient's face; the facial expression recognition unit detects the facial expression and complexion of the patient from the facial image, and determines whether the facial expression indicates displeasure based on the detection result; The diagnostic support system according to claim 1 .

8. An acquisition unit that acquires continuously recorded facial information about a patient's face; an expression recognition unit that recognizes a facial expression of a patient based on the facial information and determines whether the facial expression indicates displeasure; a display control unit that displays, on a display unit, a time series change in the result of the bad mood determination for the face information together with a time series change in the medical data of the patient; A diagnostic support system comprising:

Images