Medical image diagnostic system, operation method of medical image diagnostic system, and information processing system
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- FUJIFILM CORP
- Filing Date
- 2023-05-16
- Publication Date
- 2026-06-10
AI Technical Summary
Existing medical imaging technologies, such as MRI and X-ray CT, cause anxiety in patients due to long examination times and confined spaces, with communication challenges during the execution of pulse sequence commands, making it difficult for patients to confirm if their concerns are being heard by operators.
A medical image diagnostic system that uses video and audio detectors to capture patient responses, processes them through machine learning models to recognize statements, and displays the recognized content to the patient and operator, ensuring communication and reducing anxiety.
Patients can confirm that their statements are being conveyed to operators, enhancing a sense of security and alleviating anxiety during medical imaging procedures.
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a medical image diagnostic system, an operating method thereof, and an information processing system, and more particularly to a technique for reducing anxiety of a subject undergoing an examination using an image diagnostic apparatus. [Background technology]
[0002] The time it takes to examine a subject using imaging diagnostic equipment such as a magnetic resonance imaging (MRI) device or an X-ray computed tomography (CT) device is relatively long, with an MRI device taking around 20 to 30 minutes, an X-ray CT device taking around 5 to 10 minutes for a simple test, and around 5 to 20 minutes for a contrast test.
[0003] In addition, imaging using MRI devices and other equipment is performed in a cylindrical imaging space called a "bore," and bodily movement during the examination can cause image artifacts, so bodily movement, including breathing, must be suppressed as much as possible. Undergoing an examination for a long period of time in a closed space called a "bore" while suppressing bodily movement is stressful for the subject.
[0004] During the examination, subjects may feel anxious and become unwell as they are not allowed to move in a confined space. Even if subjects try to communicate their anxiety or unwellness to the technician, the MRI machine makes loud noises during the examination, and in some cases the technician in the control room cannot hear them.
[0005] Patent Document 1 describes a technique for achieving communication between an operator of a magnetic resonance imaging system and a subject being imaged by the magnetic resonance imaging system.
[0006] The magnetic resonance imaging system described in Patent Document 1 acquires moving images of the subject's facial area while acquiring magnetic resonance imaging data, and determines the voice activity state (whether the subject is speaking or not speaking) from the acquired moving images.If the voice activity state indicates a speaking state, a dialog box opens on the user interface displaying the message "Subject's speech has been detected" and an indicator for opening a communication channel.When the communication channel is opened, the subject can speak to the operator using the subject's microphone (the operator can hear the subject's voice).
[0007] Patent document 1 also describes that by detecting mouth movement (facial movement) when speaking from moving images of the subject's facial area, it is possible to trigger an event that is normally triggered by the patient pressing a balloon, thereby making it possible to eliminate the need for a balloon. [Prior art documents] [Patent documents]
[0008] [Patent Document 1] Special Publication No. 2021-526048 Summary of the Invention [Problem to be solved by the invention]
[0009] Patent Document 1 describes that the operator and the subject can talk during the preparation stage of the first scan (silent period of the preparation phase) and between subsequent scans, but does not describe that they can talk while a pulse sequence command is being executed. In particular, it is considered that the operator cannot be heard due to the noise generated by the MRI device while a pulse sequence command is being executed.
[0010] On the other hand, even when a pulse sequence command is being executed, if the voice activity state indicates that the subject is speaking, the operator can hear the subject's voice using noise cancellation, etc., but the subject cannot confirm whether or not the operator has heard what he or she said.
[0011] In other words, although the magnetic resonance imaging system described in Patent Document 1 allows the subject to communicate his / her anxiety or feeling unwell to the operator by voice during the examination, it is not possible to confirm whether the contents of the statement have been properly conveyed to the operator (whether the operator is connected to the outside), and it is not possible to alleviate the anxiety of being left alone in the examination room.
[0012] The present invention has been made in consideration of the above circumstances, and has an object to provide a medical image diagnostic system, an operating method for a medical image diagnostic system, and an information processing system that can reduce the anxiety of a subject during examination using an image diagnostic device. [Means for solving the problem]
[0013] The invention of a first aspect is a medical image diagnostic system comprising: an imaging diagnostic device for acquiring medical images; a first detector for detecting utterance-related information related to speech made by a subject during an examination of the subject with the imaging diagnostic device, using at least one of video and audio; a second detector for detecting response information of the subject to questions asked to the subject, using at least one of video and audio, before examination of the subject begins with the imaging diagnostic device; a first display for displaying information in a manner visible to the subject during examination with the imaging diagnostic device; a processor; and a memory for storing a program to be executed by the processor, wherein the processor generates subject characteristic information related to the subject's speech based on the response information detected by the second detector, recognizes the content of the subject's speech based on the utterance-related information and the subject characteristic information detected by the first detector, and displays the recognized content of the speech on the first display.
[0014] According to the first aspect of the invention, response information of the subject to questions is detected in advance by at least one of video and audio, and subject characteristic information related to the subject's speech is generated based on the response information. Then, when utterance-related information related to the subject's speech is detected by at least one of video and audio during the examination of the subject by the imaging diagnostic device, the content of the subject's speech is recognized based on the utterance-related information and the subject characteristic information, and the recognized content of the speech is displayed on the first display. This allows the subject to visually check the content of his / her speech on the first display, and to confirm that the system understands the content of his / her speech, providing a sense of security.
[0015] In the medical image diagnostic system according to the second aspect of the present invention, in the first aspect, it is preferable that the first detector is arranged on an imaging diagnostic device or in the vicinity of the imaging diagnostic device and includes at least one of a first camera that captures an image of the facial area of the subject including at least the lips during examination by the imaging diagnostic device and a first microphone that detects sound emitted by the subject.
[0016] In the medical image diagnostic system according to the third aspect of the present invention, in the second aspect, the utterance related information is preferably lip movement of the subject acquired from the video captured by the first camera, in order to "lip read" the contents of the subject's utterance by analyzing the lip movement of the subject.
[0017] In the medical image diagnostic system according to the fourth aspect of the present invention, in any one of the first to third aspects, it is preferable that the second detector is provided at a position farther away from the image diagnostic device than the first detector, and includes at least one of a second camera that captures an image including the subject's facial area and a second microphone that detects sound emitted by the subject.
[0018] In the medical image diagnostic system according to the fifth aspect of the present invention, in the fourth aspect, the response information is preferably a lip movement of the subject acquired from an image captured by the second camera, in order to acquire the characteristics of the subject when speaking (lip movement of the subject) in advance.
[0019] The medical image diagnostic system according to the sixth aspect of the present invention is any one of the first to fifth aspects, and includes a questioning device for asking questions to the subject, and the questioning device preferably asks questions in a predetermined format by at least one of voice and characters on a monitor screen. Note that this does not prevent an operator who operates the image diagnostic device from asking questions to the subject.
[0020] In the medical image diagnostic system according to the seventh aspect of the present invention, in the sixth aspect, the questions in the predetermined format preferably include questions that induce the subject to answer or read aloud a statement that the subject may make during the examination, in order to improve the recognition accuracy when automatically recognizing the contents of the subject's statements.
[0021] In the medical image diagnostic system according to an eighth aspect of the present invention, in any one of the first to seventh aspects, the processor preferably trains a first machine learning model dedicated to the subject based on the response information detected by the second detector, inputs the utterance-related information detected by the first detector to the trained first machine learning model, and acquires the utterance content recognized by the first machine learning model. By using the first machine learning model customized for each subject, the recognition accuracy of the utterance content of the subject is improved.
[0022] A ninth aspect of the present invention relates to a medical image diagnostic system according to the eighth aspect, wherein the subject characteristic information includes parameters optimized during the learning process of the first machine learning model based on the response information of the subject.
[0023] A medical image diagnostic system according to a tenth aspect of the present invention, in the eighth or ninth aspect, has a second machine learning model that has been machine-learned in advance based on a learning dataset consisting of utterance-related information related to utterances of multiple people, and the processor preferably inputs the utterance-related information detected by the second detector into the second machine learning model and acquires the utterance content recognized by the second machine learning model when the utterance content recognized by the first machine learning model is not meaningful or when the confidence level of the utterance content is less than a threshold. This is because there are cases where the number of questions to the subject (response information of the subject to the questions to the subject) is small and the learning of the first machine learning model is insufficient, or where the second machine learning model trained with a learning dataset consisting of utterance-related information related to utterances of a large number of subjects can provide higher recognition accuracy than the first machine learning model.
[0024] A medical image diagnostic system according to an eleventh aspect of the present invention, in any one of the first to tenth aspects, preferably includes an alarm that notifies an operator in an operation room of the image diagnostic device of the contents of the remarks, and the processor outputs the contents of the remarks to the alarm, so that the operator can also check the contents of the remarks of the subject.
[0025] A twelfth aspect of the present invention relates to a medical image diagnostic system of the eleventh aspect, wherein the alarm is at least one of a second display device that displays characters indicating the content of the statement and a speaker that generates sound indicating the content of the statement.
[0026] In the medical image diagnostic system according to the thirteenth aspect of the present invention, in the third aspect, the processor preferably determines whether or not the subject's lip movement is interfering with the examination of the subject by the image diagnostic device when the first detector does not detect the next utterance-related information for a certain period of time after detecting the utterance-related information, and when it determines that the examination of the subject is not interfering, causes the first display to display text encouraging the subject to speak. When the examination site is other than the head, it can be determined that the body movement (head movement) caused by the subject's speech does not interfere with the examination of sites other than the head. In this case, text encouraging the subject to speak is displayed on the first display, which allows communication between the subject and the operator and relieves the subject's anxiety.
[0027] A medical image diagnostic system according to a fourteenth aspect of the present invention, in any one of the first to thirteenth aspects, preferably includes a vital information measuring device that measures vital information of the subject during an examination of the subject by the image diagnostic device, and the processor determines whether or not a reply to the remark is necessary based on the remark and the measured vital information, and when it is determined that a reply is necessary, creates a reply corresponding to the remark and displays the reply on the first display. By acquiring the subject's vital information in addition to the remark of the subject, it is possible to more appropriately judge the subject's anxiety, physical condition, etc., and provide the subject with necessary information.
[0028] A medical image diagnostic system according to a fifteenth aspect of the present invention is the medical image diagnostic system of the fourteenth aspect, wherein the vital information measuring device measures one or more of the subject's heart rate, blood pressure, respiratory rate, body temperature, electrocardiogram, and blood oxygen saturation level.
[0029] A medical image diagnostic system according to a 16th aspect of the present invention is any one of the 1st to 15th aspects, wherein the first display is a projector that projects onto a screen visible to the subject, a head-up display, a head-mounted display, a liquid crystal display, or an organic EL display.
[0030] A medical image diagnostic system according to a seventeenth aspect of the present invention is any one of the first to sixteenth aspects, in which the image diagnostic device includes a magnetic resonance imaging device, an X-ray CT device, a PET device, a radiation therapy device, or a particle beam therapy device. The present invention is suitable for an image diagnostic device in which the examination time and treatment time are relatively long.
[0031] An 18th aspect of the invention is a method for operating a medical image diagnostic system comprising: a first detector that detects utterance-related information related to a subject's utterances during an examination of the subject using an imaging diagnostic device, using at least one of video and audio; a second detector that detects response information of the subject to questions asked to the subject, using at least one of video and audio, before examination of the subject begins using the imaging diagnostic device; a first display that displays information in a manner visible to the subject during examination using the imaging diagnostic device; a processor; and a memory that stores a program to be executed by the processor, wherein the processor executes each of the following steps: generating subject characteristic information related to the subject's speech generation based on the response information detected by the second detector; recognizing the content of the subject's utterance based on the utterance-related information and the subject characteristic information detected by the first detector; and displaying the recognized content of the utterance on the first display.
[0032] In a method for operating a medical image diagnostic system according to a 19th aspect of the present invention, in the 18th aspect, it is preferable that a processor executes the steps of training a first machine learning model dedicated to the subject based on the response information detected by the second detector, and inputting the utterance-related information detected by the first detector into the trained first machine learning model to obtain the utterance content recognized by the first machine learning model.
[0033] The invention of the twentieth aspect is an information processing system comprising a first detector that detects utterance related information related to speech made by a subject during an examination of the subject using an imaging diagnostic device, using at least one of video and audio; a second detector that detects response information of the subject to questions asked to the subject, using at least one of video and audio, before examination of the subject using the imaging diagnostic device begins; a first display that displays information in a manner visible to the subject during examination using the imaging diagnostic device; a processor; and a memory that stores a program to be executed by the processor, wherein the processor generates subject characteristic information related to the subject's speech generation based on the response information detected by the second detector, recognizes the content of the subject's speech based on the utterance related information and the subject characteristic information detected by the first detector, and displays the recognized content of the speech on the first display. Effect of the Invention
[0034] According to the present invention, a subject undergoing an examination using an image diagnostic apparatus can confirm that the contents of what he or she says are being conveyed to an external operator or the like, which gives the subject a sense of security. [Brief description of the drawings]
[0035] [Figure 1] FIG. 1 is a diagram showing an outline of a floor plan of an image diagnostic facility to which a medical image diagnostic system according to the present invention is applied. [Diagram 2] FIG. 2 is a perspective view showing an example of an imaging diagnostic apparatus. [Diagram 3] FIG. 3 is a schematic cross-sectional view of the MRI apparatus shown in FIG. [Figure 4] FIG. 4 is a diagram showing how the subject's body is turned relative to the MRI apparatus. [Diagram 5] FIG. 5 is a diagram showing the inside of the operation room. [Figure 6] FIG. 6 is a diagram showing an example of equipment installed in the interview room. [Figure 7] FIG. 7 is a configuration diagram of a medical image diagnostic system according to the present invention. [Figure 8]FIG. 8 is a block diagram showing an embodiment of hardware of a medical image diagnostic apparatus which is a main device constituting a medical image diagnostic system. [Figure 9] FIG. 9 is a conceptual diagram showing an embodiment of the utterance recognition unit. [Figure 10] FIG. 10 is a diagram showing another embodiment of the first display device that displays information in a manner that is visible to the subject during an examination using an MRI apparatus. [Figure 11] FIG. 11 is a flow chart illustrating an embodiment of a method of operation of a medical image diagnostic system that operates when a subject is in an interview room. [Figure 12] FIG. 12 is a flow chart illustrating a first embodiment of a method for operating a medical imaging diagnostic system during an MRI examination. [Figure 13] FIG. 13 is a flow chart illustrating a second embodiment of a method for operating a medical imaging system during an MRI examination. [Figure 14] FIG. 14 is a flow chart illustrating a third embodiment of a method for operating a medical imaging system during an MRI examination. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0036] Hereinafter, preferred embodiments of a medical image diagnostic system, an operating method of the medical image diagnostic system, and an information processing system according to the present invention will be described with reference to the accompanying drawings.
[0037] [Outline of medical imaging diagnostic systems] FIG. 1 is a diagram showing an outline of a floor plan of an image diagnostic facility to which a medical image diagnostic system according to the present invention is applied.
[0038] The imaging diagnostic facility shown in FIG. 1 is part of a hospital or health checkup facility, and has an examination room 1 in which an imaging diagnostic device is installed, an operation room 2 in which an operator operates the imaging diagnostic device, an interview room 3 in which interviews of subjects are conducted, and a waiting room 4 in which subjects and their family members or other attendants wait.
[0039] FIG. 2 is a perspective view showing an example of an imaging diagnostic apparatus.
[0040] The imaging diagnostic device shown in Figure 2 is a magnetic resonance imaging device (MRI device) 10, and includes a gantry 11 and a bed 13 with a table top 13A arranged on the front side of a bore 12, which is a cylindrical imaging space provided in the gantry 11.
[0041] The image diagnostic device in this example is an MRI device 10, but the present invention is not limited to this and includes an X-ray CT device, a PET (Positron Emission Tomography) device, a radiation therapy device, or a particle beam therapy device. The present invention is suitable for devices with relatively long examination and treatment times.
[0042] FIG. 3 is a schematic cross-sectional view of the MRI apparatus shown in FIG.
[0043] 3, an MRI magnet 10A and various other coils are arranged on a gantry 11, and a subject 5 lying on a top board 13A is moved to a bore 12 so that an examination site of the subject 5 is located at the center of the static magnetic field in the bore 12 by moving the top board 13A to a bore 12. An RF (Radio Frequency) coil 10B for receiving FID (Free Induction Decay) from the human body is arranged at the examination site of the subject 5. In this example, the abdomen of the subject 5 is the examination site, and the RF coil 10B is arranged on the abdomen.
[0044] A first camera 15 (15A, 15B) and a first microphone 16 (16A, 16B) which are first detectors are disposed at the upper portions of both ends of the bore 12. In addition, in FIG. 3, the first camera 15A and the first microphone 16A are disposed at the upper portion of the left end of the bore 12, and the first camera 15B and the first microphone 16B are disposed at the upper portion of the right end of the bore 12.
[0045] The two first cameras 15A, 15B each capture an image of the subject 5 and output a video (image), and at least one of the two first cameras 15A, 15B captures an image including the face area of the subject 5.
[0046] The two first microphones 16A and 16B each detect the voice emitted by the subject 5. The first microphones 16A and 16B are preferably non-magnetic microphones that use piezoelectric ceramics, which are non-magnetic materials, in their sound receiving portions.
[0047] A projector 17 is disposed at one end (the left end in FIG. 3) of the top board 13A, and the projector 17 projects an image onto a screen 17A disposed on the ceiling inside the bore 12. This projector 17 functions as a display (first display) that displays information in a manner that is visible to the subject 5 during an examination using the MRI apparatus 10, and the subject 5 can view the image projected onto the screen 17A during the examination. Note that the screen 17A may not be provided, and the ceiling inside the bore 12 may be used as the screen.
[0048] 3, 18 is a vital information measuring device that measures one or more of the heart rate, blood pressure, respiratory rate, body temperature, electrocardiogram, and blood oxygen saturation concentration of the subject 5. The vital information measuring device 18 is not limited to being placed on the back of the subject 5.
[0049] Specific vital information measuring devices 18 may include an ECG (electrocardiogram) sensor that measures electrocardiogram and heart rate, an SpO2 sensor that measures blood oxygen saturation, a bellows that detects respiratory rate, an optical camera capable of detecting the facial expression and body temperature of the subject 5, an infrared camera, a thermograph that measures body temperature, an NMR (Nuclear Magnetic Resonance) signal (the signal itself obtained by an MRI examination), an MRI image (an image of the NMR signal), other sensors (pressure sensors, optical sensors, ultrasonic sensors, etc.), and combinations of these.
[0050] FIG. 4 is a diagram showing how the subject's body is turned relative to the MRI apparatus.
[0051] FIG. 4(A) shows the case where the subject 5 enters the bore 12 head first (head first), and FIG. 4(B) shows the case where the subject 5 enters the bore 12 feet first (foot first).
[0052] Head-first or feet-first is selected depending on the examination site of the subject 5. In this example, first cameras 15A and 15B and first microphones 16A and 16B are disposed at both ends of the bore 12 so that the facial region of the subject 5 can be photographed and the voice can be detected regardless of whether head-first or feet-first is selected.
[0053] In this example, the number of first cameras 15 (15A, 15B) and first microphones 16 (16A, 16B) is two, but the number is not limited to two, and may be one, or three or more. Also, the first camera 15 and the first microphone 16 are not limited to being placed in the MRI device 10, and may be placed near the MRI device 10.
[0054] FIG. 5 is a diagram showing the inside of the operation room.
[0055] As shown in FIG. 5, an information processing device 20 is installed in the operation room 2 to control the MRI apparatus 10 installed in the examination room 1, and to process the NMR signals output from the MRI apparatus 10 and generate MRI images.
[0056] The information processing device 20 can be configured using a computer. The computer applied to the information processing device 20 may be a personal computer or a workstation. An operator 28 can control the operation of the MRI apparatus 10 through the information processing device 20 using an operation unit such as a keyboard and a mouse.
[0057] Further, the information processing device 20 of this embodiment has some of the functions of the medical image diagnostic system and the information processing system according to the present invention, the details of which will be described later.
[0058] The information processing device 20 is equipped with a display (second display) 25 that displays text, MRI images, vital information, and other information indicating the content of statements made by the subject during the examination, and a speaker 26 that generates sounds, etc. indicating the content of statements made by the subject during the examination.
[0059] 5, 2A denotes a window through which an operator 28 observes the state of the examination room 1. 29 denotes a display that displays images captured by the first cameras 15A and 15B.
[0060] FIG. 6 is a diagram showing an example of equipment installed in the interview room.
[0061] As shown in FIG. 6, the medical examination room 3 is provided with a second camera 31, a second microphone 32, a questioning device 33, and a display .
[0062] The operator 28 has the subject 5 enter the interview room 3 and explains the MRI examination, conducts an interview, etc., and in this example, guides the subject 5 in front of the display 34 and activates the questioning device 33.
[0063] The question device 33 asks questions in a predetermined format by displaying characters on the display 34 and by voice from a speaker (not shown).
[0064] The questions in the predetermined format preferably include questions that induce the subject to respond with a statement that the subject may make during an examination using the MRI apparatus 10, or questions that prompt the subject to read aloud.
[0065] Examples of statements that subjects may make during the test include: I feel unwell My ears hurt ·it's stuffy How long until the test is finished? Possible reasons include:
[0066] Other questions may be asked to verify your identity, such as your name and date of birth.
[0067] Furthermore, the question device 33 may provide an explanation of the MRI examination using images displayed on the display device 34 and audio.
[0068] The second camera 31 and the second microphone 32 each function as a detector (second detector) that detects the response information of the subject 5 to questions asked to the subject 5 using video and audio before the MRI apparatus 10 begins examining the subject 5.
[0069] The second camera 31 can be configured with a video camera that captures a moving image (image) including the face area of the subject 5. Also, the second microphone 32 can use a microphone provided in the video camera that is the second camera 31. In this way, the second camera 31 records and outputs a moving image (image) with audio.
[0070] In this example, the second camera 31, the second microphone 32, the questioning device 33, and the display 34 are installed in the interview room 3, but this is not limited thereto. They may be installed in a room other than the examination room 1, or in a case where the examination room 1 is sufficiently large, or conversely, in a small hospital where only a small changing room is available in addition to the examination room and operation room, they may be installed in the same room as the examination room 1 but at a position away from the MRI device 10 (at a position farther away than the first camera 15, which is the first detector, and the first microphone 16 for the MRI device 10).
[0071] [Medical image diagnostic system configuration] FIG. 7 is a configuration diagram of a medical image diagnostic system according to the present invention.
[0072] 7 is configured by connecting the various devices described above, which are installed in an examination room 1, an operation room 2, and an interview room 3, to each other via a network 50 such as a LAN (Local Area Network). In addition, a terminal 40 in a waiting room 4 is also capable of communicating through the network 50.
[0073] FIG. 8 is a block diagram showing an embodiment of the hardware of an information processing device which is a main device constituting a medical image diagnostic system.
[0074] The information processing device 20 shown in FIG. 8 is configured by a computer or the like as described above, and includes a processor 21, a memory 22, a database 23, an input / output interface 24, a display (second display) 25, a speaker 26, an operation unit 27, and the like.
[0075] The processor 21 is composed of a CPU (Central Processing Unit) etc., and controls each part of the information processing device 20 and various devices (see Figure 7) installed in the examination room 1, operation room 2, and interview room 3, and realizes various functions described later by executing an information processing program stored in the memory 22.
[0076] The memory 22 is one or more memories including a flash memory, a read-only memory (ROM), a random access memory (RAM), a hard disk drive, etc. The flash memory, the ROM, and the hard disk drive are non-volatile memories that store an operation system, an information processing program that causes the processor 21 to function as the information processing device 20, various programs including machine learning models (first machine learning model, second machine learning model), and the like.
[0077] The RAM functions as a working area for processing by the processor 21, and also temporarily stores information processing programs and the like stored in the non-volatile memory. Note that the processor 21 may incorporate a part of the memory 22 (RAM).
[0078] The database 23 is a part that stores and manages the video and audio response information detected by the second camera 31 and the second microphone 32 in the interview room 3, and stores it as learning data for training the first machine learning model described below.
[0079] The input / output interface 24 includes a communication unit connectable to the network 50, and a connection unit connectable to an external device. As the connection unit connectable to an external device, a Universal Serial Bus (USB), a High-Definition Multimedia Interface (HDMI) (HDMI is a registered trademark), etc. can be applied.
[0080] The processor 21 communicates with various devices in the examination room 1 and interview room 3 via the input / output interface 24 and the network 50, and also communicates with a terminal 40 in the waiting room 4, thereby transmitting and receiving necessary information.
[0081] The display 25 is used as part of a GUI (Graphical User Interface) for accepting text indicating what the subject 5 says during the examination, MRI images, vital sign information, personal information of the subject 5, and input from the operation unit 27.
[0082] The speaker 26 generates sounds or the like indicating what the subject 5 will say during the examination.
[0083] The operation unit 27 includes a mouse, a keyboard, etc., and functions as a part of a GUI that uses the display screen of the display device 25 and receives input from the operator 28.
[0084] <Utterance Recognition Unit> The utterance recognition unit in this example is a function provided in the processor 21. That is, the processor 21 acquires from the database 23 the video and audio response information detected by the second camera 31 and the second microphone 32 in the interview room 3, and generates subject characteristic information related to the speech production of the subject 5 based on the acquired response information. The subject characteristic information is characteristic information corresponding to the subject's unique habits of lip movement (including the overall facial expression) when the subject 5 produces speech.
[0085] When the subject 5 speaks during examination of the subject 5 by the MRI apparatus 10, the processor 21 acquires at least one of an image including the facial area of the subject 5 (lip movement of the subject 5) captured by the first camera 15 and the voice of the subject 5 detected by the first microphone 16 as utterance-related information, and recognizes the content of the utterance made by the subject 5 based on this utterance-related information and subject characteristic information generated in advance.
[0086] FIG. 9 is a conceptual diagram showing an embodiment of the utterance recognition unit.
[0087] As shown in FIG. 9, the processor 21 functions as a utterance recognition section 21A.
[0088] The utterance recognition unit 21A is a lip reading part that reads characters from the lip movements of the subject 5, and in this example, a trained machine learning model (first machine learning model) is used, and an image including the facial area of the subject 5 captured by the first camera 15 is input.
[0089] The processor 21 reads out an untrained first machine learning model stored in the memory 22 or the database 23, and trains the first machine learning model.
[0090] The first machine learning model can be various learning models such as an end-to-end deep learning network, a convolutional neural network, or a support vector machine.
[0091] The learning dataset used for training the first machine learning model is a plurality of pairs of videos including the facial region of the subject 5 captured by the second camera 31 when the subject 5 answers each of the plurality of questions in the interview room 3, and text information indicating each answer. The plurality of videos are input data when training the first machine learning model, and the plurality of pieces of text information are correct answer data for the corresponding videos.
[0092] Text information indicating each answer can be obtained by a voice recognition AI (Artificial Intelligence) that recognizes characters from the voice detected by the second microphone 32. In addition, it is preferable that the characters recognized by the voice recognition AI are displayed on the monitor screen of the display 34 of the interview room 3, and if the displayed characters differ from the answer, the subject 5 is asked to point out the error and correct it. This is to obtain correct answer data.
[0093] When the subject is asked to read aloud predetermined words or sentences, characters indicating the predetermined words or sentences can be used as the correct answer data. In this case, it is preferable that the predetermined words or sentences are utterances that the subject may utter during the examination by the MRI apparatus 10.
[0094] By training the first machine learning model using the training data set, a trained first machine learning model dedicated to the subject can be generated. The subject characteristic information includes parameters optimized in the training process of the first machine learning model based on the response information of the subject 5, and can be stored in the memory 22 or the database 23 in association with the subject.
[0095] In this example, the untrained first machine learning model is trained using a training data set obtained from the subject's own response information, but the present invention is not limited to this. An existing trained first machine learning model may be used, and the existing first machine learning model may be transfer trained or fine-tuned using a training data set obtained from new response information of the subject himself / herself. This is effective when the number of questions is small and the training data set is small.
[0096] The utterance recognition unit 21A, which uses a trained first machine learning model (a learning model in which parameters optimized during the learning process of the first machine learning model are set), inputs an image including the facial area of the subject 5 captured by the first camera 15 during an examination of the subject 5 using the MRI device 10, and in particular, when an image including the facial area of the subject 5 when the subject 5 makes a speech is input, the utterance recognition unit 21A recognizes and outputs the content of the speech character by character.
[0097] In this example, the utterance recognition unit 21A outputs a plurality of characters (character strings) indicating the content of the utterance made by the subject 5 from an input video showing the lip movement of the subject 5, but it is preferable to further provide a function of performing natural language processing on the character strings output from the utterance recognition unit 21A to determine whether the character strings are meaningful utterances or not, and correcting the character strings to make them correct if there is an error as a result of the natural language processing. Such a function can be achieved by adding an existing natural language processing AI.
[0098] The utterance recognition unit may also be configured to simultaneously input the video including the facial region of the subject 5 captured by the first camera 15 and the voice of the subject 5 detected by the first microphone 16, and recognize the content of the subject's utterance. In this case, the first machine learning model applied to the utterance recognition unit preferably has a learning model that recognizes the content of the utterance from the video including the facial region of the subject 5 (lip movements of the subject 5) and a learning model that recognizes the content of the utterance from the voice of the subject 5, and is trained so that both show the same utterance content. An end-to-end deep learning network is suitable as this type of first machine learning model.
[0099] Moreover, it is possible to recognize the utterance contents with higher accuracy by recognizing the utterance contents from the lip movement and voice of the subject 5. When the voice detected by the first microphone 16 is input to the first learning model, it is preferable to reduce the noise generated by the MRI apparatus 10 in advance by a filter or the like.
[0100] When the subject 5 speaks during an examination using the MRI device 10 as described above and the processor 21 recognizes the content of the subject 5's speech, the processor 21 projects an image (text) indicating the recognized content of the speech from the projector 17 onto the screen 17A, and displays it on the screen 17A (see Figure 3).
[0101] The subject 5 can visually confirm the content of his / her own speech projected on the screen 17A. This allows the subject 5 to confirm that the content of his / her own speech is being conveyed to an external operator or the like, which gives the subject a sense of security.
[0102] The processor 21 may be configured to project an image that relaxes the subject 5 (e.g., an image of nature) or an image of an attendant from the projector 17 onto the screen 17A.In this case, when the subject 5 speaks, the content of the statement is projected onto the screen 17A instead of the image, or the content of the statement is superimposed on the image and projected onto the screen 17A.
[0103] Furthermore, when the processor 21 recognizes the content of the statement made by the subject 5, it displays characters indicating the recognized content of the statement on the display 25 in the operation room 2, and similarly generates a sound indicating the content of the statement from the speaker 26 (see FIG. 5). This allows the operator 28 to check the content of the statement made by the subject 5 during the examination.
[0104] Furthermore, when the processor 21 recognizes the content of the statement made by the subject 5, it can display characters indicating the recognized content of the statement on the terminal 40 in the waiting room 4. This allows the attendant of the subject 5 to check the content of the statement made by the subject 5 during the examination on the terminal 40 in the waiting room 4.
[0105] <Another embodiment of the first display unit> FIG. 10 is a diagram showing another embodiment of the first display device that displays information in a manner that is visible to the subject during an examination using an MRI apparatus.
[0106] The first display shown in Figure 10 is composed of a projector (not shown), a rear projection screen 60 onto which characters indicating what the subject 5 is saying are projected from the projector, and a reflective mirror 62 that allows the subject 5, who is lying on his back on the tabletop 13A, to view the rear projection screen 60.
[0107] The rear projection screen 60 and the reflecting mirror 62 are disposed on a support member 63 attached to the top plate 13A.
[0108] When the projector projects characters indicating what the subject 5 said onto the rear projection screen 60, the subject 5 can visually recognize the characters indicating what the subject 5 said projected onto the rear projection screen 60 via the reflecting mirror 62. In addition, a first camera for photographing the face area of the subject 5 can be attached to the outer periphery of the reflecting mirror 62.
[0109] Other embodiments of the first display that displays information in a manner that can be seen by the subject during examination using an MRI device are not limited to the one shown in FIG. 10, and may include a head-up display that displays a virtual image (aerial image), a head-mounted display that is worn on the subject's head, a liquid crystal display, or an organic EL (electro-luminescence) display.
[0110] [Operation method of medical image diagnostic system] FIG. 11 is a flow chart illustrating an embodiment of a method of operation of a medical image diagnostic system that operates when a subject is in an interview room.
[0111] The processing of each step of the operating method of the medical image diagnostic system shown in FIG. 11 corresponds to the processing performed by the processor 21 of the information processing device 20 shown in FIG.
[0112] 11, the processor 21 displays an explanation about the MRI examination on the display 34 installed in the interview room 3 (step S10). Here, the procedure of the MRI examination, points to note during the MRI examination, etc. are displayed.
[0113] Next, the processor 21 operates the second camera 31 and the second microphone 32 to start detecting, by video and audio, response information of the subject 5 to questions to the subject (step S20).
[0114] The processor 21 causes a question in a predetermined format (eg, a question about information required for an MRI examination) to be displayed in text on the display 34 and as audio from a speaker (not shown) (step S30).
[0115] The subject orally answers the questions asked in step S30 (step S40).
[0116] The processor 21 converts the voice of the subject 5 who answers orally and detected by the second microphone 32 into text information (step S50). The conversion of the voice into text information can be performed using a voice recognition AI.
[0117] The processor 21 also calculates features including the lip movement habits of each subject 5 captured by the second camera 31, which is an image including the facial area of the subject 5 who answered verbally (step S60), and calculates the correspondence between the text information converted in step S50 and the features of the lip movement (step S70).
[0118] Next, processor 21 determines whether the questions have been completed (step S80), and if it determines that the questions have not been completed (if "No"), it transitions to step S30 and repeats the processes from step S30 to step S80 for the next question.
[0119] When the processor 21 determines that all questions have been completed (if "Yes"), it stores the character information and the feature amount of the lip movement calculated in step S70 in the database 23 (step S90).
[0120] The character information and lip movement features stored in the database 23 are used during lip reading during the test.
[0121] In this example, the processing of steps S30 to S80 is repeated to train the first machine learning model, and the optimized parameters of the trained first machine learning model are stored in database 23 as subject characteristic information unique to subject 5.
[0122] In addition, processor 21 displays the character information converted in step S50 on display 34, and asks subject 5 to confirm whether the converted character information matches the verbal response, and if they do not match, it is preferable to have them verbally point out the difference and correct the character information.
[0123] FIG. 12 is a flow chart illustrating a first embodiment of a method for operating a medical imaging diagnostic system during an MRI examination.
[0124] The processing of each step of the operating method of the medical image diagnostic system shown in FIG. 12 corresponds to the processing performed by the processor 21 of the information processing device 20 shown in FIG.
[0125] 12, the processor 21 causes the projector 17 to display information related to the MRI examination (step S100). The subject 5 undergoing the examination can visually confirm the information displayed by the projector 17. Note that in step S100, information other than information related to the MRI examination can also be displayed. For example, an image of an attendant or an image of the natural world can be displayed, which can relax the subject 5.
[0126] Next, the processor 21 operates the first camera 15 and the first microphone 16 to start capturing an image including the lip movement of the subject 5 during the test and measuring the sound emitted by the subject 5 (step S110).
[0127] The processor 21 determines whether the subject 5 has spoken (step S120). Whether the subject 5 has spoken can be determined by detecting the presence or absence of lip movement of the subject 5 from the image captured by the first camera 15 and / or by detecting the presence or absence of sound emitted by the subject 5 with the first microphone 16.
[0128] When the processor 21 determines that the subject 5 has not spoken (in the case of "No"), the processor 21 transitions to step S100, and repeats the processes from step S100 to step S120.
[0129] On the other hand, if it is determined in step S21 that the subject 5 is speaking (if "Yes"), the process proceeds to step S130.
[0130] In step S130, lip movements of the subject 5 included in the video captured by the first camera 15 are analyzed (lip-read) and converted into text information. In this example, parameters of the trained first machine learning model acquired in advance for the subject 5 and stored in the database 23 are read out and set in the first machine learning model, and the video including the lip movements of the subject 5 is input to an utterance recognition unit using this first machine learning model, thereby recognizing text information corresponding to the lip movements. Note that the text information may be recognized not only from lip movements but also from information of the entire face.
[0131] Furthermore, the processor 21 converts the voice uttered by the subject 5 detected by the first microphone 16 into text information (step S140). The processor 21 can convert the voice into text information using a voice recognition AI.
[0132] The processor 21 adds voice-recognized character information to the character information recognized in response to lip movements, and recognizes character information using both, thereby improving the accuracy of the converted character information.
[0133] Next, the processor 21 determines whether the converted character information (character string) is a meaningful comment (step S150). Whether the comment is meaningful or not can be determined by natural language processing of the character string. If it is determined that the comment is not meaningful (if "No"), the process proceeds to step S100.
[0134] On the other hand, if it is determined that the comment is meaningful (if "Yes"), the processor 21 causes the projector 17 to display characters indicating the content of the comment (step S160).
[0135] This allows the subject 5 to visually confirm the text indicating the content of his / her own statement during the examination. This allows the subject 5 to confirm that the content of his / her own statement has been conveyed to an external operator, etc., and gives a sense of security.
[0136] Furthermore, when the processor 21 determines that the comment is meaningful (if "Yes"), it also displays characters indicating the comment content on the display 25 (second display) in the operation room 2 (step S170). This allows the operator 28 in the operation room 2 to check the comment content uttered by the subject 5 during the examination. The annunciator that notifies the operator 28 in the operation room 2 of the comment content can be configured with at least one of the display 25 that displays characters indicating the comment content and the speaker 26 that generates a sound indicating the comment content.
[0137] Next, the processor 21 determines whether the MRI examination has been completed (step S180), and if it is determined that the MRI examination has not been completed (if "No"), the processor 21 transitions to step S100 and repeats the processes from step S100 to step S180.
[0138] On the other hand, if the processor 21 determines that the MRI examination has ended (if "Yes"), it ends this process.
[0139] The processor 21 may also display characters indicating the content of the statement made by the subject 5 on the terminal 40 in the waiting room 4.
[0140] In addition, in this example, only meaningful comments are displayed, but meaningless comments may also be displayed. This is because it is thought that a patient may feel uneasy if nothing is displayed even when they make a sound during the examination.
[0141] FIG. 13 is a flow chart illustrating a second embodiment of a method for operating a medical imaging system during an MRI examination.
[0142] In FIG. 13, the same steps as those in the flowchart shown in FIG. 12 are denoted by the same step numbers, and detailed descriptions thereof will be omitted.
[0143] The operating method of the medical image diagnostic system of the second embodiment shown in FIG. 13 differs from the first embodiment in that the processes of steps S190, S192, and S194 are added.
[0144] 13, on the flow line returning to step S100, processor 21 determines whether or not a certain time has passed since the previous message (step S190). The certain time can be an appropriate time that can be determined as a period of silence after a message.
[0145] When the processor 21 determines that a certain period of time has elapsed (if "Yes"), it further determines whether or not the subject's 5 speech interferes with MRI imaging (MRI examination) (step S192). Whether or not the subject's 5 speech interferes with MRI imaging can be determined based on the examination area set in advance. In other words, it can be determined that the body movement (head movement) caused by the subject's 5 speech does not interfere with the examination of areas other than the head. Therefore, the processor 21 can determine that there is no interference with MRI imaging when examining an area other than the head.
[0146] When the processor 21 determines that a certain time has passed since the previous utterance and that there is no problem with the MRI imaging, it causes the projector 17 to display text prompting the subject 5 to speak (step S194).
[0147] This allows communication between the subject 5 and the operator 28, and can alleviate the subject's 5 anxiety.
[0148] FIG. 14 is a flow chart illustrating a third embodiment of a method for operating a medical imaging system during an MRI examination.
[0149] In FIG. 14, the same steps as those in the flowchart shown in FIG. 12 are denoted by the same step numbers, and detailed descriptions thereof will be omitted.
[0150] The operating method of the medical image diagnostic system of the third embodiment shown in FIG. 14 differs from the first embodiment in that the processes of steps S112, S152, and S154 are added.
[0151] 14, in step S112, the vital information measuring device 18 (see FIG. 3) starts measuring the vital information of the subject 5. Here, the vital information measuring device 18 measures, as the vital information of the subject 5, one or more of the subject's heart rate, blood pressure, respiratory rate, body temperature, electrocardiogram, and blood oxygen saturation concentration.
[0152] In step S152, based on the content of the statement made by the subject 5 and the vital information of the subject 5 measured in step S112, it is determined whether or not it is necessary to provide the subject 5 with necessary information (a response to the statement), and if it is determined that a response is necessary, a response sentence to the statement is further prepared.
[0153] For example, if a significant increase in heart rate or respiratory rate is detected, it is believed that the subject 5 is anxious or tense due to being unable to endure the long examination or the confined space. On the other hand, if the subject 5 complains of anxiety, it is preferable to present the subject 5 with information that will ease his / her anxiety as the information required for the subject 5 to respond. Responses such as "Are you feeling unwell?" and "The examination will be over soon, are you OK?" can be created. Other necessary information can include images of the operator 28's face or the face of an attendant.
[0154] Also, if the body temperature at the examination site rises and subject 5 complains that he or she feels hot, a response such as "The temperature is rising due to the examination. Can you bear it?" is prepared.
[0155] When the processor 21 determines that it is necessary to provide the necessary information to the subject 5 (if "Yes"), it causes the projector 17 to display text indicating the content of the subject 5's statement, as well as the necessary information (response text) that has been created (step S154).
[0156] On the other hand, if the processor 21 determines that it is not necessary to provide the necessary information to the subject 5 (in the case of "No"), the process proceeds to step S160, and causes the projector 17 to display only the text indicating the content of the subject 5's statement.
[0157] [others] In this embodiment, a first machine learning model dedicated to the subject is generated, and an image including the subject's facial area during an MRI examination is input into a speech recognition unit using the first machine learning model, so that lip movement when the subject speaks is analyzed (lip reading) and text information indicating the content of the speech is obtained as the recognition result. However, for example, if the content of the speech recognized by the first machine learning model is not meaningful or the confidence level of the content of the speech is below a threshold, another machine learning model (second machine learning model) may be used or used in combination.
[0158] In this case, the second machine learning model is a learning data set consisting of utterance-related information related to utterances by multiple people, and is a learning model for which machine learning has been performed in advance based on a large number of learning data sets, and one that has been stored in advance in memory 22 or database 23 can be used.
[0159] In this embodiment, the hardware structure of a processing unit that executes various processes, such as a CPU, is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electric circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically for executing specific processes.
[0160] One processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with one processor. As an example of configuring multiple processing units with one processor, first, as represented by a computer such as a client or server, there is a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units. Second, as represented by a system on chip (SoC), there is a form in which a processor is used that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.
[0161] Moreover, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.
[0162] Furthermore, the present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the spirit of the present invention. [Explanation of symbols]
[0163] 1. Examination room 2...Control room 3…Question room 4. Waiting room 10...MRI device 11. Gantry 12…Boa 13. Bed 13A…Tabletop
Claims
1. An image diagnostic device for acquiring medical images; a first detector that detects utterance-related information related to a utterance of a subject by at least one of an image and an audio during an examination of the subject by the image diagnostic apparatus; a second detector that detects, before an examination of the subject is started by the image diagnostic apparatus, response information of the subject to a question for the subject by at least one of an image and an audio; a first display that displays information in a manner that is visible to the subject during an examination by the image diagnostic apparatus; A processor; A memory that stores a program to be executed by the processor; The processor, generating subject characteristic information related to the subject's speech generation based on the response information detected by the second detector; recognize the content of the utterance of the subject based on the utterance related information and the subject characteristic information detected by the first detector, and display the recognized content of the utterance on the first display. Medical imaging diagnostic system.
2. The first detector is disposed on or near the image diagnostic device, and includes at least one of a first camera that captures an image of a face region of the subject including at least the lips during an examination by the image diagnostic device, and a first microphone that detects a voice emitted by the subject. The medical image diagnostic system according to claim 1 .
3. the utterance-related information is a lip movement of the subject acquired from an image captured by the first camera; The medical image diagnostic system according to claim 2 .
4. The second detector is provided at a position farther away from the image diagnostic apparatus than the first detector, and includes at least one of a second camera that captures an image including a face region of the subject and a second microphone that detects a voice emitted by the subject. The medical image diagnostic system according to claim 1 .
5. The response information is a lip movement of the subject acquired from an image captured by the second camera. The medical image diagnostic system according to claim 4 .
6. A questioning device for asking questions to the subject, the question device asks a question in a predetermined format by at least one of voice and text on a monitor screen; The medical image diagnostic system according to claim 1 .
7. The predetermined type of question includes a question content that induces or causes the subject to read aloud a statement that the subject may make during the test as an answer. The medical image diagnostic system according to claim 6.
8. The processor, training a first machine learning model specific to the subject based on the response information detected by the second detector; inputting the utterance-related information detected by the first detector into the trained first machine learning model, and acquiring the utterance content recognized by the first machine learning model; The medical image diagnostic system according to claim 1 .
9. The subject characteristic information includes parameters optimized during a learning process of the first machine learning model based on the response information of the subject. The medical image diagnostic system according to claim 8.
10. a second machine learning model that has been subjected to machine learning in advance based on a learning dataset that includes utterance-related information related to utterances by a plurality of people; When the utterance content recognized by the first machine learning model is not meaningful or when the certainty of the utterance content is less than a threshold, the processor inputs the utterance related information detected by the second detector into the second machine learning model and acquires the utterance content recognized by the second machine learning model.
10. The medical image diagnostic system according to claim 8 or 9.
11. an alarm for notifying an operator in an operation room of the image diagnostic apparatus of the content of the statement; The processor outputs the content of the speech to the annunciator. The medical image diagnostic system according to claim 1 .
12. The annunciator is at least one of a second display that displays characters indicating the content of the speech and a speaker that generates a sound indicating the content of the speech. The medical image diagnostic system according to claim 11.
13. The processor, when the first detector does not detect the next utterance-related information for a certain period of time or more after detecting the utterance-related information, it is determined whether or not the lip movement of the subject is interfering with the examination of the subject by the image diagnostic device, and when it is determined that the lip movement of the subject is not interfering with the examination of the subject, it causes the first display to display text encouraging the subject to speak. The medical image diagnostic system according to claim 3 .
14. a vital information measuring device for measuring vital information of the subject during an examination of the subject by the image diagnostic apparatus, the processor determines whether or not a reply to the comment content is necessary based on the comment content and the measured vital sign information, and when it determines that a reply is necessary, creates a reply sentence corresponding to the comment content and causes the first display to display the reply sentence. The medical image diagnostic system according to claim 1 .
15. The vital information measuring device measures one or more of the subject's heart rate, blood pressure, respiratory rate, body temperature, electrocardiogram, and blood oxygen saturation concentration. The medical image diagnostic system according to claim 14.
16. The first display is a projector that projects onto a screen visible to the subject, a head-up display, a head-mounted display, a liquid crystal display, or an organic electroluminescence display. The medical image diagnostic system according to claim 1 .
17. The imaging diagnostic device includes a magnetic resonance imaging device, an X-ray CT device, a PET device, a radiation therapy device, or a particle beam therapy device. The medical image diagnostic system according to any one of claims 1 to 9.
18. A method for operating a medical image diagnostic system comprising: a first detector that detects, by at least one of video and audio, utterance-related information related to utterances made by a subject during an examination of the subject using an image diagnostic device; a second detector that detects, by at least one of video and audio, response information of the subject to a question for the subject before an examination of the subject using the image diagnostic device begins; a first display that displays information in a manner visible to the subject during an examination using the image diagnostic device; a processor; and a memory that stores a program to be executed by the processor, The processor, generating subject characteristic information related to the subject's speech generation based on the response information detected by the second detector; recognizing content of a statement made by the subject based on the statement related information detected by the first detector and the subject characteristic information; displaying the recognized utterance content on the first display; The above-mentioned steps are executed by the processor in accordance with the present invention.
19. training a first machine learning model specific to the subject based on the response information detected by the second detector; inputting the utterance-related information detected by the first detector into the trained first machine learning model, and acquiring the utterance content recognized by the first machine learning model; The method of claim 18, wherein the processor executes the steps of:
20. a first detector that detects utterance-related information related to an utterance of a subject by at least one of an image and an audio during an examination of the subject by an image diagnostic device; a second detector that detects, before an examination of the subject is started by the imaging diagnostic apparatus, response information of the subject to a question for the subject by at least one of an image and an audio; a first display that displays information in a manner that is visible to the subject during an examination by the image diagnostic apparatus; A processor; A memory that stores a program to be executed by the processor; The processor, generating subject characteristic information related to the speech generation of the subject based on the response information detected by the second detector; recognize the content of the utterance of the subject based on the utterance related information and the subject characteristic information detected by the first detector, and display the recognized content of the utterance on the first display. Information processing system.