Medical image diagnosis system, method for operating medical image diagnosis system, and information processing system
By integrating the camera and microphone on the image diagnosis device and combining the machine learning model to identify the subject's speech content, the problem of difficulty in conveying uneasiness during the examination is solved, and effective communication and sense of peace of mind are achieved.
Patent Information
- Application Number
- CN202410605568.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-16
- Filing Date
- 2024-05-15
- Publication Date
- 2024-11-26
AI Technical Summary
When using image diagnostic devices such as magnetic resonance imaging for inspection, the subject is difficult to effectively convey uneasy or physical discomfort due to the limitations of the closed space and noise interference, which makes it impossible to confirm whether the speech content is received by the operator, causing uneasiness.
The camera and microphone of the image diagnosis device detect the lip movement and sound of the subject, generate characteristic information with the subject, use machine learning models to identify the speech content, and display it on the display so that the subject can be visually confirmed to ensure the operator Received the speech.
It is realized that the subject can confirm that the content of his speech has been conveyed to the operator, which reduces the sense of uneasiness during the inspection and improves communication efficiency.
Smart Images

Figure CN119014849A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a medical image diagnosis system, a working method of the medical image diagnosis system and an information processing system, and in particular to a technology for reducing the anxiety of a subject being examined using an image diagnosis device. Background Art
[0002] The examination time of the subject using imaging diagnostic equipment such as magnetic resonance imaging equipment (MRI) and X-ray CT (Computed Tomography) equipment is relatively long. In the case of MRI equipment, it takes about 20 minutes to 30 minutes. In the case of X-ray CT equipment, a simple examination takes about 5 to 10 minutes, and a contrast examination takes about 5 to 20 minutes.
[0003] Furthermore, imaging using an MRI device or the like is performed in a cylindrical imaging space called a "bore", and body movement during the examination may become an image artifact, so body movement including breathing must be suppressed as much as possible. It is stressful for the examinee to suppress body movement for a long time while being examined in a closed space called a "bore".
[0004] During the examination, the patient is not allowed to move in a closed space, and may feel uneasy and unwell. Even if the patient wants to tell the technician about the uneasy or unwell condition, the examination sound of the MRI device is so loud that the technician in the operating room may not be able to tell the technician.
[0005] Patent Document 1 describes a technique for achieving communication between an operator of a magnetic resonance imaging system and a subject imaged by the magnetic resonance imaging system.
[0006] In the magnetic resonance imaging system described in Patent Document 1, a dynamic image of the facial area of the subject is acquired during the acquisition of magnetic resonance imaging data, and the sound activity state (whether the subject is in a speaking state or a non-speaking state) is determined based on the acquired dynamic image. When the sound activity state indicates a speaking state, "the subject's voice has been detected" is displayed in a dialog box opened on the user interface, or an indicator for opening a communication channel is displayed. When the communication channel is opened, the subject can speak to the operator using the subject's microphone (the operator can hear the subject's voice).
[0007] Patent Document 1 describes that by detecting the mouth movement (face movement) when speaking from a dynamic image of the subject's face area, an event that is usually triggered by the patient pressing the balloon can be triggered, thereby omitting the balloon.
[0008] Patent Document 1: Japanese Patent Application No. 2021-526048
[0009] Patent Document 1 states that the operator and the subject can have a conversation during the initial scan preparation phase (quiet period of the preparation phase) and between subsequent scans, but does not state that the operator can have a conversation during the execution of a pulse sequence command. In particular, during the execution of a pulse sequence command, it is believed that the operator's words cannot be heard due to the noise generated by the MRI device.
[0010] On the other hand, even during the execution of the pulse sequence command, when the voice activity state indicates the utterance state, the operator can hear the subject's voice by noise cancellation or the like, but the subject cannot confirm whether the speech content is conveyed to the operator.
[0011] That is, the magnetic resonance imaging system described in Patent Document 1 can convey the subject's anxiety or physical discomfort to the operator through sound during the examination, but cannot confirm whether the speech content is reliably conveyed to the operator (whether it is connected to the outside), and cannot eliminate the anxiety of being left alone in the examination room. Summary of the invention
[0012] The present invention has been made in view of such circumstances, and an object of the present invention is to provide a medical image diagnosis system, an operating method of the medical image diagnosis system, and an information processing system that can reduce the anxiety of a subject being examined using an image diagnosis apparatus.
[0013] The invention involved in the first embodiment is a medical image diagnosis system, which comprises: an image diagnosis device that acquires medical images; a first detector that detects speech-related information related to the speech of the subject through at least one of images and sounds during the examination of the subject using the image diagnosis device; a second detector that detects the response information of the subject to questions asked to the subject through at least one of images and sounds before the examination of the subject using the image diagnosis device begins; a first display that displays information in a manner that is visually recognizable to the subject during the examination using the image diagnosis device; a processor; and a memory that stores a program that causes the processor to execute the following processing: based on the response information detected by the second detector, generating subject characteristic information related to the generation of the subject's sound; and based on the speech-related information detected by the first detector and the subject characteristic information, identifying the content of the subject's speech, and displaying the identified speech content on the first display.
[0014] According to the invention involved in the first embodiment, the subject's response information to the question asked to the subject is detected in advance by at least one of the image and the sound, and the subject's characteristic information related to the subject's sound generation is generated based on the response information. And when the subject is examined by the image diagnostic device, the speech-related information related to the subject's speech is detected by at least one of the image and the sound, the subject's speech content is identified based on the speech-related information and the subject's characteristic information, and the identified speech content is displayed on the first display. As a result, the subject can visually recognize the content of his or her speech through the first display, can confirm that the system understands the content of his or her speech, and can get a sense of security.
[0015] The medical image diagnosis system involved in the second embodiment of the present invention is preferably as follows in the first embodiment: the first detector includes at least one of a first camera and a first microphone, the first camera is arranged at or near the image diagnosis device, and captures an image of at least the lips of a facial area of a subject being examined using the image diagnosis device, and the first microphone detects sounds uttered by the subject.
[0016] The medical image diagnosis system according to the third aspect of the present invention is preferably configured as follows in the second aspect: the speech-related information is obtained from the lip movements of the subject captured by the first camera, so as to "lip-read" the speech content of the subject by analyzing the lip movements of the subject.
[0017] The medical image diagnosis system involved in the fourth embodiment of the present invention is preferably as follows in any one of the first to third embodiments: the second detector includes at least one of a second camera and a second microphone, the second camera is arranged at a position farther away from the image diagnosis device than the first detector and captures an image including the facial area of the subject, and the second microphone detects the sound emitted by the subject.
[0018] The medical image diagnosis system according to the fifth aspect of the present invention is preferably configured as follows: the response information is the subject's lip movement obtained from the image captured by the second camera. This is to obtain the subject's utterance characteristics (the subject's lip movement) in advance.
[0019] The medical image diagnosis system according to the sixth aspect of the present invention is preferably as follows in any of the first to fifth aspects: a questioning device for asking questions to the subject is provided, the questioning device uses at least one of sound and characters on a monitor screen to ask questions in a predetermined format. In addition, an operator who operates the image diagnosis device is not prevented from asking questions to the subject.
[0020] The medical image diagnosis system according to the seventh aspect of the present invention is preferably configured as follows: the predetermined question format includes a question content that guides or reads out possible remarks made by the examinee during the examination as answers, in order to improve the recognition accuracy when automatically recognizing the remarks made by the examinee.
[0021] The medical image diagnosis system involved in the eighth aspect of the present invention is preferably as follows in any of the first to seventh aspects: the processor performs the following processing: training a first machine learning model dedicated to the subject based on the response information detected by the second detector; and inputting speech-related information detected by the first detector into the learned first machine learning model to obtain speech content recognized by the first machine learning model. By using the first machine learning model customized for each subject, the recognition accuracy of the subject's speech content is improved.
[0022] In the medical image diagnosis system according to a ninth aspect of the present invention, in the eighth aspect, the subject feature information includes parameters optimized in a learning process of the first machine learning model based on the subject's response information.
[0023] The medical image diagnosis system involved in the tenth embodiment of the present invention is preferably as follows in the eighth embodiment or the ninth embodiment: it has: a second machine learning model that has previously performed machine learning based on a learning data set consisting of speech-related information related to speeches of multiple people, and when the speech content recognized by the first machine learning model does not become meaningful content, or when the certainty of the speech content is less than a threshold, the processor inputs the speech-related information detected by the second detector into the second machine learning model to obtain the speech content recognized by the second machine learning model. This is because, sometimes, compared with the case of using the first machine learning model, the number of questions asked to the subject (the subject's response information to the questions asked to the subject) is small and the learning of the first machine learning model is insufficient, or the recognition accuracy is high when using the second machine learning model trained with a large amount of speech-related information related to the subject's speech.
[0024] The medical image diagnosis system according to the eleventh aspect of the present invention is preferably as follows in any one of the first to tenth aspects: it comprises: a notifier for notifying an operator in an operation room of the image diagnosis apparatus of the speech content, and the processor outputs the speech content to the notifier, so that the operator can also confirm the speech content of the subject.
[0025] In the medical image diagnosis system according to the twelfth aspect of the present invention, in the eleventh aspect, the notification device is at least one of a second display that displays characters indicating the content of the speech and a speaker that generates a sound indicating the content of the speech.
[0026] The medical image diagnosis system involved in the 13th mode of the present invention is preferably as follows in the 3rd mode: when the first detector detects speech-related information and does not detect the next speech-related information for more than a certain period of time, the processor determines whether the lip movement of the subject hinders the examination of the subject by the image diagnosis device, and when it is determined that there is no obstacle to the examination of the subject, a character urging the subject to speak is displayed on the first display. In the case where the examination part is a part other than the head, it can be determined that the body movement (movement of the head) caused by the subject's utterance does not hinder the examination of the part other than the head. In this case, the character urging the subject to speak can be displayed on the first display, communication can be achieved between the subject and the operator, and the subject's anxiety can be eliminated.
[0027] The medical image diagnosis system according to the 14th aspect of the present invention is preferably as follows in any one of the 1st to 13th aspects: it comprises: a vital sign information measuring device that measures the vital sign information of the subject during the examination of the subject by the image diagnosis device, and the processor determines whether a reply to the speech content is required based on the speech content and the measured vital sign information, and when it is determined that a reply is required, creates a reply corresponding to the speech content and displays the reply on the first display. By acquiring the vital sign information of the subject in addition to the speech content of the subject, it is possible to more appropriately judge the subject's anxiety, physical condition, etc., and provide the subject with necessary information.
[0028] The medical image diagnosis system according to the fifteenth aspect of the present invention is preferably as follows in the fourteenth aspect: the vital sign information measuring device measures one or more of the subject's heart rate, blood pressure, respiratory rate, body temperature, electrocardiogram, and blood oxygen saturation concentration.
[0029] In the medical image diagnosis system according to the 16th embodiment of the present invention, in any one of the 1st to 15th embodiments, the first display is a projector, a head-up display, a head-mounted display, a liquid crystal display or an organic EL display that projects onto a screen that is visually recognizable by the subject.
[0030] According to a 17th aspect of the present invention, in any one of the first to 16th aspects, the medical image diagnosis system comprises an image diagnosis apparatus including a magnetic resonance imaging apparatus, an X-ray CT apparatus, a PET apparatus, a radiotherapy apparatus, or a particle beam therapy apparatus. The present invention is applicable to an image diagnosis apparatus having a relatively long examination time or treatment time.
[0031] The invention involved in the 18th embodiment is a working method of a medical image diagnosis system, which comprises: a first detector, which detects speech-related information related to the speech of the subject through at least one of images and sounds during the examination of the subject using the image diagnosis device; a second detector, which detects the response information of the subject to the questions asked to the subject through at least one of images and sounds before the examination of the subject using the image diagnosis device begins; a first display, which displays information in a manner that is visually recognizable to the subject during the examination using the image diagnosis device; a processor; and a memory, which stores a program for the processor to execute, in the working method of the medical image diagnosis system, the processor respectively executes the following steps: based on the response information detected by the second detector, generating subject characteristic information related to the sound generation of the subject; based on the speech-related information and the subject characteristic information detected by the first detector, identifying the speech content of the subject; and displaying the identified speech content on the first display.
[0032] The working method of the medical image diagnosis system involved in the 19th mode of the present invention is preferably as follows in the 18th mode: the processor executes the following steps: training the first machine learning model dedicated to the subject based on the response information detected by the second detector; and inputting the speech-related information detected by the first detector into the learned first machine learning model to obtain the speech content recognized by the first machine learning model.
[0033] The invention involved in the 20th embodiment is an information processing system, which comprises: a first detector, which detects speech-related information related to the speech of the subject through at least one of images and sounds during the examination of the subject using an image diagnostic device; a second detector, which detects the response information of the subject to questions asked to the subject through at least one of images and sounds before the examination of the subject using the image diagnostic device begins; a first display, which displays information in a manner that is visually recognizable to the subject during the examination using the image diagnostic device; a processor; and a memory, which stores a program executed by the processor, and the processor performs the following processing: based on the response information detected by the second detector, generates subject characteristic information related to the sound generation of the subject; and based on the speech-related information detected by the first detector and the subject characteristic information, identifies the content of the subject's speech, and displays the identified speech content on the first display.
[0034] Effects of the Invention
[0035] According to the present invention, a subject who is undergoing an examination using an image diagnostic apparatus can confirm that the contents of his or her speech have been communicated to an external operator or the like, and can feel relieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1This is a diagram schematically showing a plan view of an image diagnosis facility to which the medical image diagnosis system according to the present invention is applied.
[0037] Figure 2 It is a perspective view showing an example of an image diagnosis apparatus.
[0038] Figure 3 yes Figure 2 A schematic cross-sectional view of an MRI apparatus is shown.
[0039] Figure 4 The diagram shows the change in orientation of the subject's body relative to the MRI apparatus.
[0040] Figure 5 It is a figure which shows the internal situation of an operation room.
[0041] Figure 6 This is a diagram showing an example of equipment installed in a consultation room.
[0042] Figure 7 It is a structural diagram of the medical image diagnosis system involved in the present invention.
[0043] Figure 8 This is a block diagram showing an embodiment of hardware of a medical image diagnosis apparatus which is a main device constituting a medical image diagnosis system.
[0044] Fig. 9 This is a conceptual diagram showing an embodiment of a speech recognition unit.
[0045] Fig.10 This is a diagram showing another embodiment of the first display that displays information in a manner that is visually recognizable by a subject during an examination using an MRI apparatus.
[0046] Fig.11 This is a flowchart showing an embodiment of an operating method of the medical image diagnosis system that operates when a subject is in a consultation room.
[0047] Fig.12 This is a flowchart showing a first embodiment of an operating method of a medical image diagnosis system in an MRI examination.
[0048] Fig.13 This is a flowchart showing a second embodiment of an operating method of the medical image diagnosis system during MRI examination.
[0049] Fig.14 This is a flowchart showing a third embodiment of the operating method of the medical image diagnosis system during MRI examination.
[0050] Explanation of symbols
[0051] 1-examination room, 2-operation room, 3-inquiry room, 4-waiting room, 10-MRI device, 11-rack, 12-well, 13-bed, 13A-top board. DETAILED DESCRIPTION
[0052] Hereinafter, preferred embodiments of the medical image diagnosis system, the operating method of the medical image diagnosis system, and the information processing system according to the present invention will be described with reference to the accompanying drawings.
[0053] [Overview of Medical Image Diagnosis System]
[0054] Figure 1 This is a diagram schematically showing a plan view of an image diagnosis facility to which the medical image diagnosis system according to the present invention is applied.
[0055] Figure 1 The image diagnosis facility shown is a part of a hospital or health diagnosis facility, and has an examination room 1 equipped with an image diagnosis device, an operation room 2 where an operator operates the image diagnosis device, a consultation room 3 where the subject is examined, and a waiting room 4 where the subject and his / her family members and other accompanying persons wait.
[0056] Figure 2 It is a perspective view showing an example of an image diagnosis apparatus.
[0057] Figure 2 The image diagnosis apparatus shown is a magnetic resonance imaging apparatus (MRI apparatus) 10 , and includes a gantry 11 and a bed 13 including a top plate 13A disposed in front of a hole 12 which is a cylindrical imaging space provided in the gantry 11 .
[0058] The image diagnosis device in this example is an MRI device 10, but the present invention is not limited thereto, and includes an X-ray CT device, a PET (Positron Emission Tomography) device, a radiotherapy device, or a particle beam therapy device. The present invention is suitable for devices with relatively long inspection and treatment times.
[0059] Figure 3 yes Figure 2 A schematic cross-sectional view of an MRI apparatus is shown.
[0060] exist Figure 3In the embodiment, an MRI magnet 10A and various other coils are arranged on a gantry 11, and a subject 5 lying on a top plate 13A is moved so that the examination part of the subject 5 is located at the center of the static magnetic field in the hole 12 by moving the top plate 13A to the hole 12. An RF (Radio Frequency) coil 10B for receiving FID (Free Induction Decay) from the human body is arranged at the examination part of the subject 5. In this example, the abdomen of the subject 5 is the examination part, and the RF coil 10B is arranged on the abdomen.
[0061] A first camera 15 (15A, 15B) and a first microphone 16 (16A, 16B) as a first detector are arranged at the upper part of both ends of the hole 12. Figure 3 In the embodiment, a first camera 15A and a first microphone 16A are arranged above the left end of the hole 12, and a first camera 15B and a first microphone 16B are arranged above the right end of the hole 12.
[0062] The two first cameras 15A and 15B are used to capture images of the subject 5 and output dynamic images (videos), respectively. At least one of the two first cameras 15A and 15B captures an image including a facial region of the subject 5 .
[0063] The two first microphones 16A and 16B respectively detect sounds uttered by the subject 5. The first microphones 16A and 16B are preferably non-magnetic microphones using piezoelectric ceramics which are non-magnetic bodies for the sound receiving body.
[0064] At one end of the top plate 13A ( Figure 3 A projector 17 is disposed at the left end of the hole 12, and the projector 17 projects an image onto a screen 17A disposed on the ceiling portion in the hole 12. The projector 17 functions as a display (first display) that displays information in a manner that can be visually recognized by the subject 5 during the examination using the MRI apparatus 10, and the subject 5 can visually recognize the image projected on the screen 17A during the examination. In addition, the ceiling portion in the hole 12 may be used as a screen without providing the screen 17A.
[0065] And, in Figure 3 In the figure, 18 is a vital sign information measuring device, which measures one or more of the heart rate, blood pressure, respiratory rate, body temperature, electrocardiogram and blood oxygen saturation concentration of the subject 5. The vital sign information measuring device 18 is not limited to being arranged on the back of the subject 5.
[0066] As specific vital sign information measuring devices 18, ECG (electrocardiogram) sensors for measuring electrocardiogram and heart rate, SpO2 sensors for measuring blood oxygen saturation concentration, bellows for detecting respiratory rate, optical cameras capable of detecting the expression and body temperature of the subject 5, infrared cameras, thermal imagers for measuring body temperature, devices for measuring vital sign information based on NMR (Nuclear Magnetic Resonance) signals (the signals themselves obtained through MRI examinations) and MRI images (NMR signals are visualized), other sensors (pressure sensors, optical sensors, ultrasonic sensors, etc.) and combinations thereof may be considered.
[0067] Figure 4 The diagram shows the change in orientation of the subject's body relative to the MRI apparatus.
[0068] Figure 4 (A) shows the situation where the subject 5 enters the hole 12 from the head (head priority), Figure 4 (B) shows a case where the subject 5 enters the hole 12 from the feet (feet first).
[0069] The head priority or the foot priority is selected according to the inspection part of the subject 5. In this example, the first cameras 15A, 15B and the first microphones 16A, 16B are respectively arranged at the two ends of the hole 12, so that the facial area of the subject 5 can be photographed and the sound can be detected regardless of whether the head priority or the foot priority is selected.
[0070] In this example, there are two first cameras 15 (15A, 15B) and two first microphones 16 (16A, 16B), but the present invention is not limited thereto and the number may be one or more than three. Furthermore, the first camera 15 and the first microphone 16 are not limited to being disposed in the MRI apparatus 10 and may be disposed near the MRI apparatus 10.
[0071] Figure 5 It is a figure which shows the internal situation of an operation room.
[0072] like Figure 5 As shown, an information processing device 20 is installed in the operation room 2 , and the information processing device 20 controls the MRI device 10 installed in the inspection room 1 , processes NMR signals output from the MRI device 10 , and generates an MRI image.
[0073] The information processing device 20 can be configured using a computer. The computer used in the information processing device 20 may be a personal computer or a workstation. The operator 28 can control the operation of the MRI apparatus 10 through the information processing device 20 using an operation unit such as a keyboard and a mouse.
[0074] Furthermore, the information processing device 20 of this example has a part of the functions of the medical image diagnosis system and the information processing system according to the present invention, but the details thereof will be described later.
[0075] The information processing device 20 includes: a display (second display) 25 that displays characters representing the content of speech uttered by the subject during the examination, MRI images, vital sign information, and other information; and a speaker 26 that generates sound representing the content of speech uttered by the subject during the examination.
[0076] And, in Figure 5 In the figure, 2A is a window for an operator 28 to observe the situation in the examination room 1. 29 is a display for displaying images taken by the first cameras 15A and 15B.
[0077] Figure 6 This is a diagram showing an example of equipment installed in a consultation room.
[0078] like Figure 6 As shown, a second camera 31 , a second microphone 32 , a questioning device 33 , and a display 34 are installed in the consultation room 3 .
[0079] The operator 28 allows the subject 5 to enter the examination room 3 and explains and asks questions about the MRI examination. In this example, the operator guides the subject 5 to the display 34 and activates the questioning device 33.
[0080] The questioning device 33 asks questions in a predetermined format through character display on the display 34 and sound from a speaker (not shown).
[0081] The questions in the predetermined format preferably include question contents that guide or read out as answers speech that the subject may utter during the examination using the MRI apparatus 10 .
[0082] As examples of statements that might be made by the examinee during an examination, consider the following:
[0083] Physical discomfort
[0084] Ear pain
[0085] Difficulty breathing
[0086] How long will it take for the inspection to be completed?
[0087] wait.
[0088] In addition, you can ask questions such as your name and date of birth to verify your identity.
[0089] Furthermore, the questioning device 33 may explain the MRI examination through images and sounds displayed on the display 34 .
[0090] The second camera 31 and the second microphone 32 each function as a detector (second detector) that detects response information of the subject 5 to questions asked to the subject 5 by means of images and sounds before the examination of the subject 5 starts using the MRI apparatus 10 .
[0091] The second camera 31 can be composed of a video camera that captures a moving image (video) including the facial area of the subject 5. Furthermore, the second microphone 32 can use a microphone provided in the video camera serving as the second camera 31. Thus, the second camera 31 records and outputs a moving image (video) with sound.
[0092] In addition, in this example, the second camera 31, the second microphone 32, the questioning device 33 and the display 34 are arranged in the consultation room 3, but are not limited to this. They can also be arranged in other rooms other than the examination room 1, or when the examination room 1 is sufficiently spacious or conversely in a small hospital where there is only a small changing room other than the examination room / operation room, they can be arranged in a position away from the MRI device 10 (a position farther away from the MRI device 10 than the first camera 15 and the first microphone 16 which serve as the first detector) in the same room as the examination room 1.
[0093] [Structural diagram of medical image diagnosis system]
[0094] Figure 7 It is a structural diagram of the medical image diagnosis system involved in the present invention.
[0095] Figure 7 The medical image diagnosis system shown is configured such that the above-mentioned various devices installed in the examination room 1, the operation room 2 and the consultation room 3 are mutually connected via a network 50 such as a LAN (Local Area Network). Furthermore, the terminal 40 in the waiting room 4 can also communicate via the network 50.
[0096] Figure 8 This is a block diagram showing an embodiment of hardware of an information processing device which is a main device constituting a medical image diagnosis system.
[0097] Figure 8 The information processing device 20 shown is constituted by a computer or the like as described above, and includes a processor 21 , a memory 22 , a database 23 , an input / output interface 24 , a display (second display) 25 , a speaker 26 , an operation unit 27 , and the like.
[0098] The processor 21 is composed of a CPU (Central Processing Unit) and the like, and controls various components of the information processing device 20 and various devices installed in the examination room 1, the operation room 2, and the consultation room 3 (see Figure 7 ) performs centralized control, and executes the information processing program stored in the memory 22, thereby realizing the various functions described later.
[0099] The memory 22 is one or more memories including a flash memory, a ROM (Read-only Memory), a RAM (Random Access Memory), a hard disk device, etc. The flash memory, the ROM, and the hard disk device are non-volatile memories that store an operating system, an information processing program that enables the processor 21 to function as the information processing device 20, and various programs including a machine learning model (a first machine learning model, a second machine learning model).
[0100] The RAM functions as a work area for processing by the processor 21 and temporarily stores information processing programs stored in the nonvolatile memory, etc. The processor 21 may also incorporate a part of the memory 22 (RAM).
[0101] The database 23 is a part for storing and managing response information of images and sounds detected by the second camera 31 and the second microphone 32 of the consultation room 3, and stores the information as learning data for training a first machine learning model described later.
[0102] The input / output interface 24 includes a communication unit that can be connected to the network 50 and a connection unit that can be connected to an external device. As the connection unit that can be connected to the external device, USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface) (HDMI is a registered trademark), etc. can be used.
[0103] The processor 21 communicates with various devices in the examination room 1 and the consultation room 3 via the input / output interface 24 and the network 50 , and communicates with the terminal 40 in the waiting room 4 , thereby transmitting and receiving necessary information.
[0104] The display 25 is used as a part of a GUI (Graphical User Interface) for receiving characters representing the content of speech uttered by the subject 5 during the examination, MRI images, vital sign information, personal information of the subject 5 , and input from the operation unit 27 .
[0105] The speaker 26 generates sounds and the like indicating the content of speech uttered by the subject 5 during the examination.
[0106] The operation unit 27 includes a mouse, a keyboard, and the like, and functions as a part of a GUI that receives input from the operator 28 using the display screen of the display 25 .
[0107] <Speech Recognition Unit>
[0108] The speech recognition unit in this example is a function possessed by the processor 21. That is, the processor 21 obtains the response information of the image and sound detected by the second camera 31 and the second microphone 32 of the consultation room 3 from the database 23, and generates the subject characteristic information related to the sound generation of the subject 5 based on the obtained response information. The subject characteristic information is characteristic information corresponding to the subject's inherent habits of the lip movement (including the expression of the entire face) when the subject 5 generates sound.
[0109] When the subject 5 speaks during the examination using the MRI device 10, the processor 21 obtains at least one of the image of the facial area of the subject 5 (the lip movement of the subject 5) taken by the first camera 15 and the voice of the subject 5 detected by the first microphone 16 as speech-related information, and identifies the speech content of the subject 5 based on the speech-related information and the pre-generated subject feature information.
[0110] Fig. 9 This is a conceptual diagram showing an embodiment of a speech recognition unit.
[0111] like Fig. 9 As shown, the processor 21 functions as a speech recognition unit 21A.
[0112] The speech recognition unit 21A is the lip reading part that reads characters from the lip movements of the subject 5. In this example, a learned machine learning model (first machine learning model) is used to input an image containing the facial area of the subject 5 captured by the first camera 15.
[0113] The processor 21 reads out the unlearned first machine learning model stored in the memory 22 or the database 23, and trains the first machine learning model.
[0114] As the first machine learning model, various learning models such as end-to-end deep learning networks, convolutional neural networks, support vector machines, etc. can be used.
[0115] The learning data set used in learning the first machine learning model is a plurality of pairs of images including the facial area of the subject 5 captured by the second camera 31 when the subject 5 answers a plurality of questions in the consultation room 3, and character information representing each answer. The plurality of images are input data for learning the first machine learning model, and the plurality of character information are correct answer data for the respective corresponding images.
[0116] The character information representing each answer can be obtained by voice recognition AI (Artificial Intelligence) that recognizes characters based on the voice detected by the second microphone 32. In addition, it is preferable to display the characters recognized by the voice recognition AI on the monitor screen of the display 34 in the consultation room 3, and when the displayed characters are different from the answers, the examinee 5 is asked to point out the error and make corrections. This is to obtain correct correct answer data.
[0117] When the subject is asked to read a predetermined sentence or text, the characters representing the predetermined sentence or text can be used as the correct answer data. In this case, the predetermined sentence or text is preferably set as the content of speech that the subject may utter during the examination using the MRI apparatus 10.
[0118] By training the first machine learning model with the above-mentioned learning data set, a first machine learning model that has been learned specifically for the subject can be generated. Furthermore, the subject characteristic information includes parameters optimized during the learning process of the first machine learning model based on the response information of the subject 5, and can be stored in the memory 22 or the database 23 in association with the subject.
[0119] In addition, in this example, the first machine learning model that has not been learned is trained using the learning data set obtained from the subject's own response information, but this is not limited to this. The existing first machine learning model that has been learned can also be used to transfer training or fine-tune the existing first machine learning model using the learning data set obtained from the new subject's own response information. This is effective when the number of questions is small and the learning data set is small.
[0120] The speech recognition unit 21A, which uses the learned first machine learning model (a learning model in which parameters optimized during the learning process of the first machine learning model are set), inputs an image of the facial area of the subject 5 captured by the first camera 15 when the subject 5 is examined using the MRI device 10, but particularly when the image of the facial area of the subject 5 when the subject 5 is speaking is input, recognizes and outputs the speech content character by character.
[0121] In this example, the speech recognition unit 21A outputs a plurality of characters (character strings) representing the speech content uttered by the examinee 5 from the image representing the input lip movement of the examinee 5, but preferably also has the following function: natural language processing is performed on the character string output from the speech recognition unit 21A to determine whether the character string is a meaningful speech, or if the result of the natural language processing of the character string is wrong, it is corrected to become the correct speech content. Such a function can be realized by adding an existing natural language processing AI.
[0122] Furthermore, the speech recognition unit may also simultaneously input an image of the facial region of the subject 5 captured by the first camera 15 and the voice of the subject 5 detected by the first microphone 16 to recognize the speech content of the subject. In this case, the first machine learning model applied to the speech recognition unit includes a learning model for recognizing speech content based on an image of the facial region of the subject 5 (lip movements of the subject 5) and a learning model for recognizing speech content based on the voice of the subject 5, and preferably a model trained so that both represent the same speech content is used. An end-to-end deep learning network is suitable as such a first machine learning model.
[0123] Furthermore, the speech content can be recognized more accurately by recognizing the speech content based on the lip movements and sounds of the subject 5. When the sound detected by the first microphone 16 is input to the first learning model, it is preferable to reduce the noise generated by the MRI apparatus 10 by a filter or the like in advance.
[0124] As described above, when the subject 5 makes a sound during the examination by the MRI apparatus 10 and the speech content of the subject 5 is recognized, the processor 21 projects an image (character) representing the recognized speech content from the projector 17 onto the screen 17A and displays it on the screen 17A (see FIG. Figure 3 ).
[0125] The subject 5 can visually recognize the content of his / her speech projected on the screen 17A. Thus, the subject 5 can confirm that the content of his / her speech has been communicated to an external operator, etc., and can feel relieved.
[0126] The processor 21 may also project an image that relaxes the subject 5 (for example, an image of nature) or an image of an accompanying person from the projector 17 onto the screen 17A. In this case, when the subject 5 makes a sound, the speech content is projected onto the screen 17A instead of the image, or the speech content is superimposed on the image and projected onto the screen 17A.
[0127] When the speech content of the subject 5 is recognized, the processor 21 displays characters representing the recognized speech content on the display 25 of the operation room 2, and similarly generates a sound representing the speech content from the speaker 26 (refer to Figure 5 ). Thus, the operator 28 can confirm the content of the speech uttered by the subject 5 during the examination.
[0128] Furthermore, when the speech content of the examinee 5 is recognized, the processor 21 can display characters indicating the recognized speech content on the terminal 40 in the waiting room 4. Thus, the person accompanying the examinee 5 can confirm the speech content uttered by the examinee 5 during the examination through the terminal 40 in the waiting room 4.
[0129] <Another Embodiment of the First Display>
[0130] Fig.10 This is a diagram showing another embodiment of the first display that displays information in a manner that is visually recognizable by a subject during an examination using an MRI apparatus.
[0131] Fig.10 The first display shown is composed of a projector (not shown), a rear projection screen 60 on which characters representing the speech content of the subject 5 are projected from the projector, and a reflection mirror 62 for the subject 5 lying on his back on the top plate 13A to visually recognize the rear projection screen 60.
[0132] The rear projection screen 60 and the reflector 62 are disposed on a support member 63 attached to the top plate 13A.
[0133] When characters representing the speech content of the subject 5 are projected from the projector onto the rear projection screen 60, the subject 5 can visually recognize the characters representing the speech content of the subject 5 projected onto the rear projection screen 60 via the reflector 62. In addition, the first camera that captures the facial area of the subject 5 can be installed on the outer periphery of the reflector 62.
[0134] Another embodiment of the first display that displays information in a manner that can be visually recognized by the subject during the examination using the MRI apparatus is not limited to Fig.10 As the display shown, a head-up display that displays a virtual image (an aerial image), a head-mounted display that is worn on the head of the subject, a liquid crystal display, or an organic EL (electro-luminescence) display can be used.
[0135] [Working method of medical image diagnosis system]
[0136] Fig.11 This is a flowchart showing an embodiment of an operating method of the medical image diagnosis system that operates when a subject is in a consultation room.
[0137] Fig.11 The processing of each step of the working method of the medical image diagnosis system shown corresponds to the processing of each step of the working method of the medical image diagnosis system shown in FIG. Figure 8 The processing is performed by the processor 21 of the information processing device 20 shown.
[0138] exist Fig.11 In step S10, the processor 21 displays the explanation about the MRI examination on the display 34 provided in the consultation room 3. Here, the order of the MRI examination, the precautions in the MRI examination, etc. are displayed.
[0139] Next, the processor 21 operates the second camera 31 and the second microphone 32 to start detecting the response information of the subject 5 to the question to the subject through images and sounds (step S20).
[0140] The processor 21 asks a question in a predetermined format (for example, a question about information necessary for an MRI examination) through character display on the display 34 and sound from a speaker (not shown) (step S30).
[0141] The subject verbally answers the question in step S30 (step S40).
[0142] Processor 21 converts the voice of subject 5 who answered orally, that is, the voice detected by second microphone 32, into character information (step S50). Voice recognition AI can be used to convert the voice into character information.
[0143] Furthermore, the processor 21 calculates the characteristic quantity of each subject's habit including the image of the facial area of the subject 5 who gives an oral answer, that is, including the lip movement of the subject 5 captured by the second camera 31 (step S60), and calculates the correspondence between the character information converted in step S50 and the characteristic quantity of the lip movement (step S70).
[0144] Next, the processor 21 determines whether the questioning is completed (step S80 ). If it is determined that the questioning is not completed (“No”), the process moves to step S30 , and the processes from step S30 to step S80 are repeated for the next question.
[0145] When the processor 21 determines that all questions have been completed (in the case of YES), the character information and the feature amount of the lip movement calculated in step S70 are stored in the database 23 (step S90).
[0146] The character information and the feature amount of the lip movement stored in the database 23 are used in lip reading during the inspection.
[0147] In this example, the first machine learning model is learned by repeating the processing of the above steps S30 to S80, and the optimized parameters of the learned first machine learning model are stored in the database 23 as the inherent subject characteristic information of the subject 5.
[0148] In addition, the processor 21 preferably displays the character information converted in step S50 on the display 34, allowing the subject 5 to confirm whether the converted character information is consistent with the content of the verbal answer. If inconsistent, the subject 5 verbally points out the difference and corrects the character information.
[0149] Fig.12 This is a flowchart showing a first embodiment of an operating method of a medical image diagnosis system in an MRI examination.
[0150] Fig.12 The processing of each step of the working method of the medical image diagnosis system shown corresponds to the processing of each step of the working method of the medical image diagnosis system shown in FIG. Figure 8 The processing is performed by the processor 21 of the information processing device 20 shown.
[0151] exist Fig.12 In step S100, the processor 21 displays information related to the MRI examination through the projector 17 (step S100). The subject 5 under examination can visually recognize the information displayed by the projector 17. In addition, in step S100, information other than the information related to the MRI examination can also be displayed. For example, the image of an accompanying person or the image of nature can be displayed, thereby relaxing the subject 5.
[0152] Next, the processor 21 activates the first camera 15 and the first microphone 16 to start capturing images including lip movements of the subject 5 under examination and measuring sounds uttered by the subject 5 (step S110).
[0153] Processor 21 determines whether subject 5 speaks (step S120). Whether subject 5 speaks can be determined by detecting the presence of subject 5 lip movements in the image captured by first camera 15 and / or by detecting the presence of subject 5 sound through first microphone 16.
[0154] If processor 21 determines that subject 5 has not spoken (in the case of "No"), the process moves to step S100 and repeats the processes from step S100 to step S120.
[0155] On the other hand, when processor 21 determines that subject 5 is speaking (in the case of YES), the process moves to step S130.
[0156] In step S130, the lip movements of the subject 5 contained in the image captured by the first camera 15 are analyzed (lip reading) and converted into character information. In this example, the parameters of the first machine learning model that has been learned and stored in the database 23 for the subject 5 are read out in advance and set in the first machine learning model, and the image containing the lip movements of the subject 5 is input to the speech recognition unit using the first machine learning model, thereby recognizing the character information corresponding to the lip movements. In addition, it is not limited to lip movements, and character information can also be recognized based on information of the entire face.
[0157] Then, the processor 21 converts the sound uttered by the subject 5 detected by the first microphone 16 into character information (step S140). The processor 21 can convert the sound into character information using the sound recognition AI.
[0158] The processor 21 adds the character information recognized by the voice to the character information recognized in accordance with the lip movement, and recognizes the character information by both, thereby improving the accuracy of the converted character information.
[0159] Next, the processor 21 determines whether the converted character information (character string) is a meaningful speech (step S150). By natural language processing on the character string, it is possible to determine whether it is a meaningful speech. If it is determined that it is not a meaningful speech ("No"), the process moves to step S100.
[0160] On the other hand, if it is determined that the speech is meaningful (in the case of YES), processor 21 displays characters indicating the content of the speech via projector 17 (step S160).
[0161] Thus, the examinee 5 who is being examined can visually recognize the characters indicating the content of his / her speech. Thus, the examinee 5 can confirm that the content of his / her speech has been communicated to an external operator, etc., and can feel relieved.
[0162] Furthermore, if the processor 21 determines that the speech is meaningful ("Yes"), characters indicating the speech content are also displayed on the display 25 (second display) of the operation room 2 (step S170). Thus, the operator 28 in the operation room 2 can confirm the speech content uttered by the subject 5 during the examination. In addition, the notification device for notifying the operator 28 in the operation room 2 of the speech content can be composed of at least one of the display 25 that displays the characters indicating the speech content and the speaker 26 that generates the sound indicating the speech content.
[0163] Next, the processor 21 determines whether the MRI examination is finished (step S180 ). If it is determined that the examination is not finished (“No”), the process moves to step S100 and the processes from step S100 to step S180 are repeated.
[0164] On the other hand, when the processor 21 determines that the MRI examination has ended (in the case of YES), the present process ends.
[0165] In addition, the processor 21 may also display characters indicating the content of the speech uttered by the subject 5 on the terminal 40 in the waiting room 4 .
[0166] In this example, only meaningful speech is displayed, but meaningless speech may also be displayed. This is because even if a sound is made during the inspection, it will make people feel uneasy if nothing is displayed.
[0167] Fig.13 This is a flowchart showing a second embodiment of an operating method of the medical image diagnosis system during MRI examination.
[0168] In addition, Fig.13 In, with Fig.12 The common parts in the flowcharts shown are marked with the same step numbers, and their detailed descriptions are omitted.
[0169] Fig.13 The operating method of the medical image diagnosis system according to the second embodiment shown is different from that of the first embodiment in that the processing of steps S190 , S192 , and S194 is added.
[0170] exist Fig.13 In the process line returning to step S100, processor 21 determines whether a certain time has passed since the last speech (step S190). The certain time may be an appropriate time that can be determined as silence after speech.
[0171] If it is determined that a certain period of time has passed ("Yes"), the processor 21 further determines whether the vocalization of the subject 5 hinders the MRI imaging (MRI examination) (step S192). Through the pre-set examination area, it is possible to determine whether the vocalization of the subject 5 hinders the MRI imaging. In other words, it can be determined that the body movement (movement of the head) caused by the vocalization of the subject 5 does not hinder the examination of parts other than the head. Therefore, when examining parts other than the head, the processor 21 can determine that there is no obstruction to MRI imaging.
[0172] If it is determined that a certain time has passed since the last speech and there is no obstacle to MRI imaging, processor 21 displays characters urging subject 5 to speak via projector 17 (step S194).
[0173] This enables communication between the subject 5 and the operator 28 , and the anxiety of the subject 5 can be eliminated.
[0174] Fig.14This is a flowchart showing a third embodiment of the operating method of the medical image diagnosis system during MRI examination.
[0175] In addition, Fig.14 In, with Fig.12 The common parts in the flowcharts shown are marked with the same step numbers, and their detailed descriptions are omitted.
[0176] Fig.14 The operating method of the medical image diagnosis system of the third embodiment shown is different from that of the first embodiment in that the processing of steps S112, S152, and S154 is added.
[0177] exist Fig.14 In step S112, the vital sign information measuring device 18 (reference Figure 3 ) Measurement of vital sign information of the subject 5. Here, the vital sign information measuring device 18 measures one or more of the subject's heart rate, blood pressure, respiratory rate, body temperature, electrocardiogram, and blood oxygen saturation concentration as the subject's vital sign information.
[0178] In step S152, based on the content of the speech made by subject 5 and the vital sign information of subject 5 measured in step S112, it is determined whether it is necessary to prompt subject 5 with necessary information (an answer to the speech content). If it is determined that an answer is necessary, a reply to the speech content is further created.
[0179] For example, when a significant increase in heart rate or respiratory rate is detected, it is considered that the subject 5 is feeling uneasy or nervous because he cannot endure the long-term examination or the closed space. On the other hand, when the subject 5 complains of being uneasy, as information that needs to be answered by the subject 5, it is preferred to prompt information that relieves the anxiety of the subject 5. Create responses such as "Do you feel okay?" and "The examination is about to end, are you okay?" In addition, the image of the face of the operator 28 or the face of the accompanying person can be used as necessary information.
[0180] Furthermore, when the body temperature of the examination site rises and the subject 5 complains of "heat", a response such as "The temperature is rising during the examination, is it tolerable?" is created.
[0181] If processor 21 determines that necessary information needs to be provided to subject 5 (in the case of YES), projector 17 displays characters indicating the content of subject 5's speech and displays the created necessary information (reply) (step S154).
[0182] On the other hand, if the processor 21 determines that it is not necessary to provide the subject 5 with the necessary information (in the case of "No"), the process proceeds to step S160, and the projector 17 displays only characters indicating the content of the subject 5's speech.
[0183] [other]
[0184] In this embodiment, a first machine learning model dedicated to the subject is generated, and an image of the facial area of the subject under MRI examination is input into a speech recognition unit using the first machine learning model, thereby analyzing (lip reading) the lip movements of the subject when making sounds, and obtaining character information representing the speech content as a recognition result. However, for example, in the case where the speech content recognized by the first machine learning model does not become meaningful content, or in the case where the certainty of the speech content is less than a threshold, other machine learning models (second machine learning model) may also be used or used simultaneously.
[0185] At this time, the second machine learning model is a learning data set composed of speech-related information related to speeches of multiple people, and is a learning model that has been pre-machined based on a large number of learning data sets. It is possible to use a learning model pre-stored in the memory 22 or the database 23.
[0186] Furthermore, in the present embodiment, for example, the hardware structure of the processing unit that executes various processes such as a CPU is various processors as shown below. The various processors include a general-purpose processor that executes software (programs) and functions as various processing units, namely a CPU (Central Processing Unit), a processor whose circuit structure can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array), a processor having a circuit structure specially designed for executing specific processes, such as a programmable logic device (PLD), and an ASIC (Application Specific Integrated Circuit), namely a dedicated circuit.
[0187] One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, multiple processing units may be composed of one processor. As an example of multiple processing units composed of one processor, first, there is the following method: as represented by computers such as clients or servers, one processor is composed of a combination of one or more CPUs and software, and the processor functions as multiple processing units. Second, there is the following method: as represented by a system on chip (SoC), a processor that implements the functions of the entire system including multiple processing units using one IC (Integrated Circuit) chip is used. In this way, as a hardware structure, various processing units are composed of one or more of the above-mentioned various processors.
[0188] Furthermore, more specifically, the hardware structure of these various processors is a circuit (circuitry) formed by combining circuit elements such as semiconductor elements.
[0189] Furthermore, the present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the spirit of the present invention.
Claims
1. A medical image diagnosis system comprising: Image diagnostic device, to acquire medical images; a first detector for detecting speech-related information related to speech of the subject by at least one of images and sounds during examination of the subject by the image diagnostic apparatus; a second detector for detecting, by at least one of images and sounds, response information of the subject to a question asked to the subject before the subject is examined by the image diagnostic apparatus; a first display for displaying information in a manner that is visually recognizable by the subject during an examination using the image diagnostic apparatus; processor; and a memory storing a program for the processor to execute, The processor performs the following processing: generating subject characteristic information related to the sound production of the subject based on the response information detected by the second detector; and The speech content of the subject is recognized based on the speech-related information detected by the first detector and the subject characteristic information, and the recognized speech content is displayed on the first display.
2. The medical image diagnosis system according to claim 1, wherein: The first detector includes at least one of a first camera and a first microphone, wherein the first camera is arranged at or near the image diagnostic device and captures an image of at least the lips of the facial area of the subject being examined using the image diagnostic device, and the first microphone detects sounds uttered by the subject.
3. The medical image diagnosis system according to claim 2, wherein: The speech-related information is the lip movement of the subject acquired from the image captured by the first camera.
4. The medical image diagnosis system according to claim 1, wherein: The second detector includes at least one of a second camera and a second microphone, wherein the second camera is arranged at a position farther away from the image diagnosis device than the first detector and captures an image including a facial area of the subject, and the second microphone detects sounds emitted by the subject.
5. The medical image diagnosis system according to claim 4, wherein: The response information is the lip movement of the subject acquired from the image captured by the second camera.
6. The medical image diagnosis system according to claim 1, comprising a questioning device for asking questions to the subject. The questioning device asks questions in a predetermined format using at least one of sound and characters on a monitor screen.
7. The medical image diagnosis system according to claim 6, wherein: The questions in the predetermined format include question contents that guide or read out as answers speech that the subject may utter during the examination.
8. The medical image diagnosis system according to claim 1, wherein: The processor performs the following processing: training a first machine learning model specific to the subject based on the response information detected by the second detector; and The speech related information detected by the first detector is input into the first machine learning model that has been learned, and the speech content recognized by the first machine learning model is obtained.
9. The medical image diagnosis system according to claim 8, wherein: The subject characteristic information includes parameters optimized in a learning process of the first machine learning model based on the response information of the subject.
10. The medical image diagnosis system according to claim 8 or 9, comprising: The second machine learning model performs machine learning in advance based on a learning data set consisting of speech-related information related to speeches of multiple people. When the speech content recognized by the first machine learning model does not become meaningful content, or when the certainty of the speech content is less than a threshold, the processor inputs the speech-related information detected by the second detector into the second machine learning model to obtain the speech content recognized by the second machine learning model.
11. The medical image diagnosis system according to claim 1, comprising: a notifier for notifying an operator in an operation room of the image diagnosis apparatus of the content of the speech, The processor outputs the speech content to the notifier.
12. The medical image diagnosis system according to claim 11, wherein: The notification device is at least one of a second display that displays characters indicating the speech content and a speaker that generates a sound indicating the speech content.
13. The medical image diagnosis system according to claim 3, wherein: When the first detector does not detect the next speech-related information for more than a certain period of time after detecting the speech-related information, the processor determines whether the lip movement of the subject hinders the examination of the subject using the image diagnostic device, and when it is determined that there is no obstacle to the examination of the subject, characters urging the subject to speak are displayed on the first display.
14. The medical image diagnosis system according to claim 1, comprising: a vital sign information measuring device for measuring vital sign information of the subject during examination of the subject by the image diagnostic device, The processor determines whether a response to the speech content is required based on the speech content and the measured vital sign information, and when it is determined that the response is required, creates a reply corresponding to the speech content and displays the reply on the first display.
15. The medical image diagnosis system according to claim 14, wherein: The vital sign information measuring device measures at least one of the subject's heart rate, blood pressure, respiratory rate, body temperature, electrocardiogram, and blood oxygen saturation concentration.
16. The medical image diagnosis system according to claim 1, wherein: The first display is a projector, a head-up display, a head-mounted display, a liquid crystal display, or an organic EL display that projects onto a screen that can be visually recognized by the subject.
17. The medical image diagnosis system according to any one of claims 1 to 9, wherein: The image diagnosis device includes a magnetic resonance imaging device, an X-ray CT device, a PET device, a radiotherapy device or a particle beam therapy device.
18. An operating method of a medical image diagnosis system, the medical image diagnosis system comprising: a first detector for detecting speech-related information related to speech of a subject by at least one of images and sounds during examination of the subject by an image diagnosis device; a second detector for detecting, by at least one of images and sounds, the subject's response information to questions asked to the subject before the subject is examined by the image diagnostic apparatus; a first display for displaying information in a manner visually recognizable to the subject during the examination by the image diagnostic apparatus; a processor; and a memory for storing a program for the processor to execute. In the working method of the medical image diagnostic system, the processor respectively executes the following steps: generating subject characteristic information related to the sound production of the subject based on the response information detected by the second detector; identifying the speech content of the subject based on the speech-related information detected by the first detector and the subject feature information; and The recognized speech content is displayed on the first display.
19. The working method of the medical image diagnosis system according to claim 18, wherein: The processor performs the following steps: training a first machine learning model specific to the subject based on the response information detected by the second detector; and The speech related information detected by the first detector is input into the first machine learning model that has been learned, and the speech content recognized by the first machine learning model is obtained.
20. An information processing system comprising: a first detector for detecting speech-related information related to speech of a subject by at least one of images and sounds during examination of the subject by the image diagnostic device; a second detector for detecting, by at least one of images and sounds, response information of the subject to a question asked to the subject before the subject is examined by the image diagnostic apparatus; a first display for displaying information in a manner that is visually recognizable by the subject during an examination using the image diagnostic apparatus; processor; and a memory storing a program for the processor to execute, The processor performs the following processing: generating subject characteristic information related to the sound production of the subject based on the response information detected by the second detector; and The speech content of the subject is recognized based on the speech-related information detected by the first detector and the subject characteristic information, and the recognized speech content is displayed on the first display.
