Method and system for generating sensor signal-dependent dialogue during medical imaging processes

Abstract

The present invention relates to a system for sensor signal dependent interaction generation during a medical imaging process, the system comprising: a sensor module configured to measure patient status data; a processor module configured to analyze the patient status data to determine biometric and health status data of the patient; and an interaction data generation module configured to generate questionnaire data for obtaining real-time feedback from the patient during the medical imaging process, the questionnaire data being based on parameters of the medical imaging process and the determined biometric and health status data of the patient.

Description

Technical Field

[0001] The present invention relates to a system and method for a sensor signal-dependent dialogue manager for capturing a patient's condition and related motion generation. In particular, the present invention relates to a system and method for sensor signal-dependent dialogue generation during a medical imaging process, as well as a system, a method for training a machine learning model, a computer program element, and a computer-readable medium.

Background Art

[0002] EP 3 473 181 A1 describes a method for operating a medical imaging device, an imaging device, a computer program, and an electronically readable data carrier. The device described therein comprises an audio dialogue system having at least a noise of a patient receiving microphone and at least one speaker to the patient, and the voice emitted from the patient is recorded by means of the microphone and evaluated by the voice recognition unit of the audio dialogue system to determine patient information describing the state of the patient and / or the content of the uttered words.

[0003] During self-imaging, there are several situations when the patient is alone during the preparation stage, the scan stage, and after the scan. During magnetic resonance imaging, MRI, or computed tomography, CT, procedures, the patient is isolated in the modality room and given several instructions to follow.

[0004] These instructions may relate to staying still, holding one's breath at intervals, and not moving for a certain period of time. The patient is given a rough idea of the time required for the scan. The patient also has no sense of the passage of time when undergoing the treatment.

[0005] In some scenarios, the patient may feel anxious and uncomfortable during the scan or medical imaging process. This may be due to a scan duration longer than predicted, changes in other biometric conditions, or similar and corresponding aspects.

[0006] However, in autonomous imaging scenarios, patients may not be able to clearly express their feedback or their anxieties during the scan or medical imaging process. [Overview of the project] [Problems that the invention aims to solve]

[0007] Therefore, improved medical imaging systems may be needed. The object of the present invention is achieved by the subject matter of the independent claims, and further embodiments are incorporated into the dependent claims. It should be noted that the embodiments of the present invention described below apply equally to systems and methods, methods for training machine learning models, computer program elements, and computer-readable media. [Means for solving the problem]

[0008] According to a first aspect of the present invention, a system for generating sensor signal dependency dialogue during a medical imaging process, wherein the system is A sensor module configured to measure patient status data, A processor module configured to analyze the patient's condition data and determine the patient's biometric and health status data, A dialogue data generation module configured to generate questionnaire data for obtaining real-time feedback from the patient during the medical imaging process, wherein the questionnaire data is based on the parameters of the medical imaging process and the patient's determined biometric and health status data. A system having the following characteristics is provided.

[0009] In contrast to the prior art, the present invention enables the automation of a method for obtaining real-time feedback from a patient during a medical imaging process or during a single scan of a medical imaging process.

[0010] The present invention advantageously uses the automatic generation of relevant questionnaires based on the patient's psychological and health status to obtain desired feedback.

[0011] The present invention advantageously uses an automated system to continuously monitor a patient's biological, physiological, or health status and obtain relevant feedback to take appropriate subsequent actions.

[0012] The present invention makes it possible to determine whether a patient has understood a question from the facial and bodily expressions presented by the patient. The present invention advantageously makes the patient more comfortable by making the patient more flexible in providing feedback using simple gestures, voice commands, and facial expressions, including lip-reading, using sensory input and / or feedback data.

[0013] Advantageously, this invention enables an improved workflow through better patient cooperation and reduced patient anxiety.

[0014] According to exemplary embodiments of the present invention, the sensor module is further configured to measure audio data from a patient, and the processor module is further configured to analyze the measured audio data from the patient to perform emotion recognition and determine the patient's level of comfort.

[0015] According to exemplary embodiments of the present invention, the patient comfort level may refer to, for example, a level describing physical pain, acoustic noise, visual stimuli or immobilization or restriction of movement, or sleep disturbances affecting the patient.

[0016] According to exemplary embodiments of the present invention, the sensor module is further configured to measure video data of a patient, and the processor module is further configured to analyze the measured video data of the patient to perform emotion recognition and determine the patient's level of comfort.

[0017] According to exemplary embodiments of the present invention, emotion recognition includes at least one of face recognition, voice recognition, and gesture recognition.

[0018] According to exemplary embodiments of the present invention, the sensor module is further configured to measure patient feedback data, and the processor module is further configured to analyze the measured patient feedback in order to control the medical imaging process.

[0019] According to exemplary embodiments of the present invention, the parameters of the medical imaging process include at least one of the following: the scan type of the medical imaging process, the scan sequence of the medical imaging process, the duration of the scan of the medical imaging process, and the target portion of the patent of the medical imaging process.

[0020] According to an exemplary embodiment of the present invention, the dialogue data generation module is configured to generate questionnaire data in order to minimize the amount of feedback data required from the patient.

[0021] According to an exemplary embodiment of the present invention, the processor module comprises a neural network configured to optimize the generation of questionnaire data based on parameters of the medical imaging process and determined biometric and health status data of the patient.

[0022] In yet another embodiment, a computer program element is provided which, when executed by at least one processing unit, is adapted to cause the method to perform the method according to a previous embodiment.

[0023] In yet another embodiment, at least one computer-readable medium is provided that stores program elements or machine learning modules.

[0024] In general, "machine learning" can include classifications that are programmed based on algorithms that build models based on sample data known as "training data" in order to make predictions or decisions without being explicitly programmed.

[0025] Some machine learning algorithms are model-based. Model-based ML algorithms operate to adjust the parameters of a machine learning model. This adjustment procedure is called "training".

[0026] Thus, the model is configured by training to perform a task. ML algorithms also include instance-based learning. The task performance by the ML algorithm is measurably improved, and newer training data is used for training. The performance can be objectively measured by supplying test data to the system. The performance can be defined with respect to a specific error rate to be achieved for a given test data.

[0027] Here, exemplary embodiments of the present invention will be described with reference to the following drawings, which are not drawn to scale unless otherwise specified.

Brief Description of the Drawings

[0028] [Figure 1] A flowchart diagram of functional modules according to an exemplary embodiment of the present invention is shown. [Figure 2] Data representing the psychological and health states of a patient according to an exemplary embodiment of the present invention is shown. [Figure 3] A block diagram of a computer-implemented system for sensor signal-dependent dialogue generation during a medical imaging process according to an exemplary embodiment of the present invention is shown. [Figure 4] A flowchart of a method for sensor signal-dependent dialogue generation during a medical imaging process according to an exemplary embodiment of the present invention is shown.

Modes for Carrying Out the Invention

[0029] Figure 1 shows a flowchart of the functional module. According to an exemplary embodiment of the present invention, the block diagram shown in Figure 1 illustrates how all the different modules described above are connected in realizing the proposed system.

[0030] According to exemplary embodiments of the present invention, the process can begin with continuously monitoring various sensor data, including the following: i) Sensors that monitor the patient's psychological state (ECG, EMG, temperature, blood pressure, SpO2, or corresponding parameters or conditions). ii) A camera for facial recognition and patient movement detection (which can also be used for heart rate and respiration detection). iii) A microphone for voice and voice emotion recognition. iv) An integrated pain sensor in the mattress based on an electromagnetic sensor. v) Skin conductance sensors such as GSR. vi) The output of each sensor module is analyzed by the AI ​​module to determine the patient's psychological and physical condition. vii) The patient's psychological / biometric state continuously estimates fluctuations in the patient's vital parameters (oxygen saturation, heart rate, blood pressure, etc.). viii) Stress levels are determined by analyzing facial expressions using a series of images obtained from one or more cameras, employing a pre-trained AI model.

[0031] According to an exemplary embodiment of the present invention, the stress level identification module can also use the output of a biometric AI model as one of its input features. First, a supervised AI model is realized using known ground truth samples that can be obtained from patient feedback as labels.

[0032] According to an exemplary embodiment of the present invention, the algorithm may be a combination of a machine learning approach for estimating the current stress level (e.g., SVM, CNN) and a machine learning approach for predicting the development of the stress level (e.g., RNN or LSTM) over the next few minutes.

[0033] According to an exemplary embodiment of the present invention, the stress level is assessed by focusing on the output of two specific vital parameter sensors i) a GSR sensor, where the stress level is registered as the peak intensity (more precisely, the sum of the rising edges of peaks above the background level) taken as a moving average over a period of several minutes and / or ii) the sum of the periodicity of the patient's heart rate variability, thereby associating stronger periodicity with a more relaxed patient.

[0034] According to exemplary embodiments of the present invention, optionally, in ii), if the heart rate variability is periodic and further synchronized with the patient's respiratory rate (measured by a third vital parameter sensor), the patient may be considered to be even more relaxed.

[0035] According to exemplary embodiments of the present invention, the patient's pain level may be obtained from an integrated pain sensor in the mattress, or from the skin based on an electromagnetic sensor, or from the skin such as GSR, galvanic skin response, attached to any convenient peripheral point on the patient where many sweat glands are present, such as on the feet, for example, two adjacent fingers on one side, or two electrodes on the inside of a wristband.

[0036] According to an exemplary embodiment of the present invention, the patient's exercise level or exercise grade is identified, for example, between none, minimum, moderate, and severe exercise levels by analyzing a series of images obtained from a camera or the output of a motion sensor, or a combination thereof.

[0037] According to exemplary embodiments of the present invention, the patient's level of comfort is estimated by detecting the patient's emotional level using facial expressions, other physical movements, and the vital parameter measurements described above.

[0038] According to an exemplary embodiment of the present invention, specifically in stressful situations such as undergoing an autonomous scan, the emotional level is often related to the total amount of stress encountered, and the emotional valence becomes more negative as the total amount of stress increases.

[0039] According to exemplary embodiments of the present invention, this is further evaluated, for example, by measuring the integral sum of peak intensity over the entire measurement period in the GSR measurement (in contrast to the running average described above).

[0040] The block diagram shown in Figure 2 illustrates how the outputs of different sensors can be used to determine a patient's psychological and physical state.

[0041] According to exemplary embodiments of the present invention, in the next step, the outputs of these AI models are used to generate relevant questionnaires in order to more easily and comfortably obtain the necessary feedback.

[0042] According to exemplary embodiments of the present invention, problems can be predefined and evaluated by simplicity and answer range (only "yes" and "no" as answers), semi-open problems ("cold", "warm", "good", "bad", "good"), and open problems.

[0043] According to exemplary embodiments of the present invention, the output of the patient status module is used to select a question from these groups, and as a result, the questions are intentionally kept simpler in the case of a worse patient status. During the actual MR scan, preferably only closed or semi-open questions are issued. This keeps the answers short and thus minimizes the movements induced by the answers. Secondly, lip reading can be applied to analyze answers in noisy environments because the answer range is small and largely unknown.

[0044] According to an exemplary embodiment of the present invention, a list of questions can be generated as a lookup table based on the sequence / stage of each scan in an anatomical scan. This list can be entered by analyzing the questions that are asked by a human technician during a normal scan of a selected patient profile.

[0045] According to exemplary embodiments of the present invention, if the patient profile is related to OLD AGE, the system needs to ask more questions at each phase of the scan. This list is generated by analyzing population-level data and generating contextual questions that depend on the patient profile, the stage of the scan, and the anatomical structures scanned, etc.

[0046] According to an exemplary embodiment of the present invention, the questionnaire data may also include instructions.

[0047] According to exemplary embodiments of the present invention, the problems with the generated questionnaire data may include, but are not limited to, the following:

[0048] Are you feeling pain?

[0049] Are you feeling anxious or uncomfortable?

[0050] Should I pause the scan for a while?

[0051] Should I permanently stop the scan?

[0052] Do you want to move your body a little to relax?

[0053] Do you want to know the remaining time for the scan?

[0054] Are you having trouble breathing?

[0055] Is it very cold?

[0056] Shall we go to the break room?

[0057] Does it feel warm?

[0058] According to an exemplary embodiment of the present invention, the generated questionnaire is communicated to the patient by visual text audio, tactile, bone conduction, etc.

[0059] According to exemplary embodiments of the present invention, in most scenarios, patients may not be able to clearly express their feedback or their anxiety during the scanning or medical imaging process. Therefore, based on more relevant issues, patients during the examination can use a microphone to provide feedback / responses in the form of simple gestures, such as hand movements, eye movements, or simple voice commands such as "yes" or "no."

[0060] According to exemplary embodiments of the present invention, biometric parameter sensors may also be used to evaluate a patient's response, as is known from recumbent detector technology.

[0061] According to exemplary embodiments of the present invention, related or more related problems can be generated based on the scan type, the sequence of scans, the anatomical structures targeted, and the patient's physiological and psychological state.

[0062] According to an exemplary embodiment of the present invention, for example, once the neck and upper back are beyond the problem, a problem may be generated such as "Relax / move your neck a little" because the patient may need to remain still and relax slightly during the first part of the scan for a complete spinal scan. If the patient's psychological state is indicated as "stressed," the dialogue generator may generate a stress-related problem, such as "You are under stress," and if the problem is positive, it may also generate an action, such as playing music to reduce stress.

[0063] Similarly, if a patient's physiological state, such as an increase in HR / RR, is identified, questions about anxiety can be generated, and upon confirmation, haptic solutions can be deployed to reduce HR. If negative feedback is received more than expected positive feedback, the scanning procedure may be adapted for patient safety or even stopped. This may be another action performed by the machine.

[0064] In another example, a patient's panic response as they approach sleep can be analyzed in additional ways using combined images received from cameras and other sensors. For instance, in a respiratory control scan, the system must wake the patient if they fall asleep and ensure they begin to follow instructions. Another example is after administering a contrast agent to check whether the patient feels comfortable.

[0065] According to exemplary embodiments of the present invention, CT contrast can induce a sensation of heat flushing through the body. According to exemplary embodiments of the present invention, the question can be “You feel hot / warm,” and based on the answer, the patient can be reassured that this is normal.

[0066] According to exemplary embodiments of the present invention, the NLP module can be implemented as a neural network classification model trained using a scan sequence, patient status, time instance in the scan, target anatomical structure, and a set of ground truth questions / questions. The trained model can then retrieve questions / questions based on the input.

[0067] According to exemplary embodiments of the present invention, during a scan or tumor treatment session, drowsiness detection should be used to prevent the patient from falling asleep or losing consciousness during a series of different scans. Patients often fall asleep briefly and exhibit panic reactions upon waking. In addition, communication between the operator and the patient is impossible.

[0068] According to an exemplary embodiment of the present invention, it is used to prevent an actuator from falling during sleep or to prevent nudges combined with an audio signal.

[0069] According to exemplary embodiments of the present invention, detection is achieved using 3D detection, such as radar or lidar, optical detection and ranging, in combination with the patient's NLP response time.

[0070] According to exemplary embodiments of the present invention, in the event of falling asleep, special communication protocols such as "Hello do not sleep" and "stay wake" are applied, and additional entertainment is modified and adapted to prevent falling asleep.

[0071] According to an exemplary embodiment of the present invention, when a series of different scans are initiated, each scan has a short autonomous audio tag to inform or communicate with the patient about the type of scan, which helps reduce anxiety. The decision to apply the audio tag is controlled by an AI algorithm that directly classifies the patient's emotional state. The tag informs the patient about timing, robustness against movement, etc. The tag can also be applied to scans in combination with therapy.

[0072] According to an exemplary embodiment of the present invention, microphone-based communication during an MRI scan is impossible in 90% of normal scans. Technically, microphone signals, including noise level cancellation, can be supported by non-contact reading of gestures and lips using a 3D radar or LiDAR-based approach. All signals from different sensors are fed into an AI-based SW for NLP.

[0073] According to exemplary embodiments of the present invention, communication is performed in an environment / scan with a high noise level, and for simple communication, it is proposed to allow the patient to answer "yes" or "no" or "no," and furthermore, to use a respiratory response detector. The detector has a threshold and is activated when the patient takes a short breath. Different detection techniques can be applied to the patient using voice byte communication. A wireless byte sensor is placed in the mouth and is capable of transmitting signals to a communication interface.

[0074] According to an exemplary embodiment of the present invention, another example also allows for the analysis of the “trust level” developed by the patient using an interactive dialogue manager. When the patient feels that the dialogue manager is responding to the patient’s concerns, problems, and requests in an appropriate manner, the relaxed mood and acceptance are analyzed via sensors, and through such confirmation, the system can continue as suggested by the algorithm.

[0075] If the patient's response indicates a low level of confidence and the patient's level of anxiety is high, another predetermined approach can be selected as the first escalation level, and if that does not work, staff can also be involved to improve the situation.

[0076] According to exemplary embodiments of the present invention, AI “learning” from managing some of these situations helps optimize the system and build the best suggestion strategy based on the patient profile for the most successful dialogue processing. At the same time, continuous control of acceptance and “trust levels” ensures a safe environment for patients using the autonomous system.

[0077] Another way to do this is to assign a confidence level to the output of the reaction analyzer module. If the confidence level falls below a certain threshold, which serves as feedback from the user, the analysis or detection is deemed insufficient. In this scenario, the system generates a second level of procedure that helps connect remote staff automatically for further steps and execution of the system.

[0078] According to exemplary embodiments of the present invention, in an autonomous imaging workflow, these feedbacks are processed automatically by processing images received from the camera (hand / eye gestures, lip reading), or by using voice to a text recognition module when the feedback is communicated in the form of voice commands.

[0079] According to an exemplary embodiment of the present invention, based on the feedback, the control center of the scanner unit is notified to take appropriate action.

[0080] One or more features described herein may be configured or implemented as circuits encoded in a computer-readable medium, or using circuits and / or in combination thereof. Circuits may include discrete and / or integrated circuits, systems on a chip (SOC), and combinations thereof, machines, computer systems, processors and memory, and computer programs.

[0081] In another exemplary embodiment of the present invention, a computer program or computer program element is provided, characterized in that it is adapted to perform a method step of a method according to one of the embodiments described above on a suitable system.

[0082] Accordingly, the computer program elements may be stored in a computer unit, which may be part of an embodiment of the present invention. This computing unit may be adapted to perform or trigger the execution of the steps of the method described above. Furthermore, it may be adapted to operate the components of the apparatus described above. The computing unit may be adapted to operate automatically and / or to execute user sequences.

[0083] A computer program can be loaded into the working memory of a data processor. Therefore, the data processor may be equipped to perform the method of the present invention.

[0084] This exemplary embodiment of the present invention encompasses both computer programs that use the present invention from the outset and computer programs that, by means of updating, transform an existing program into a program that uses the present invention.

[0085] Furthermore, the computer program elements can provide all the steps necessary to satisfy the procedures of the exemplary embodiment of the method described above.

[0086] According to a further exemplary embodiment of the present invention, a computer-readable medium such as a CD-ROM is presented, having computer program elements stored thereon, which are described in the previous section.

[0087] Computer programs may be stored and / or distributed on suitable media (in particular, but not necessarily, non-temporary media) such as optical storage media or solid-state media supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

[0088] However, computer programs may also be presented over a network such as the World Wide Web and downloaded from such a network into the working memory of a data processor. According to a further exemplary embodiment of the present invention, a medium is provided for making a computer program element available for download, and this computer program element is configured to perform a method according to one of the aforementioned embodiments of the present invention.

[0089] It should be noted that embodiments of the present invention are described with reference to different subject matter. In particular, some embodiments are described with reference to method-type claims, and other embodiments are described with reference to apparatus-type claims. However, unless otherwise notified, those skilled in the art will find that any combination of features belonging to one type of subject matter, as well as any combination of features relating to different subject matter, are gathered from the above and below descriptions and are deemed to be disclosed in this application. However, all features can be combined to provide a synergistic effect greater than the simple sum of the features.

[0090] Although the present invention is illustrated and described in detail in the drawings and the foregoing description, such illustrations and descriptions should be considered illustrative or exemplary and not limiting. The present invention is not limited to the embodiments disclosed. Other variations of the disclosed embodiments can be understood and practiced by those skilled in the art in carrying out the claimed invention, based on a study of the drawings, disclosure and dependent claims.

[0091] In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude plurals. A single processor or other unit may satisfy the functions of several items enumerated in the claims. The mere fact that certain means are referenced in different dependent claims does not imply that combinations of these means cannot be used advantageously. No reference numeral in the claims should be construed as limiting the scope of protection of the claims. [Explanation of Symbols]

[0092] 10 Sensor Modules 20 Processor Modules 30 Interactive Data Generation Module 100 Systems S1 Measurement S2 analysis S3 generation

Claims

1. A system for generating sensor signal dependency dialogue during a medical imaging process, wherein the system is A sensor module configured to measure patient status data, A processor module configured to analyze the patient's condition data and determine the patient's biometric and health status data, A dialogue data generation module configured to generate questionnaire data for obtaining real-time feedback from the patient during the medical imaging process, wherein the questionnaire data is based on the parameters of the medical imaging process and the patient's determined biometric and health status data. A system that has

2. The system according to claim 1, wherein the sensor module is further configured to measure audio data of a patient, and the processor module is further configured to analyze the measured audio data of the patient to perform emotion recognition and determine the patient's level of comfort.

3. The system according to claim 1 or 2, wherein the sensor module is further configured to measure video data of a patient, and the processor module is further configured to analyze the measured video data of the patient to perform emotion recognition and determine the patient's level of comfort.

4. The system according to claim 2 or 3, wherein the emotion recognition comprises at least one of face recognition, voice recognition, and gesture recognition.

5. The system according to any one of claims 1 to 4, wherein the dialogue data generation module is further configured to obtain real-time feedback from the patient during the medical imaging process.

6. The system according to any one of claims 1 to 4, wherein the sensor module is further configured to measure patient feedback data, and the processor module is further configured to analyze the measured patient feedback in order to control the medical imaging process.

7. The system according to any one of claims 1 to 6, wherein the parameters of the medical imaging process include at least one of the scan type of the medical imaging process, the scan sequence of the medical imaging process, the scan duration of the medical imaging process, and the target portion of the patient in the medical imaging process.

8. The system according to any one of claims 1 to 7, wherein the dialogue data generation module is configured to generate questionnaire data in order to minimize the amount of feedback data required from the patient.

9. The system according to any one of claims 1 to 8, wherein the sensor module is configured to measure patient state data with respect to the detection of drowsiness in the patient, and the dialogue data generation module is configured to generate an audio-based or video-based wake-up signal to stimulate the patient based on the detection of drowsiness.

10. The system according to any one of claims 1 to 9, wherein the processor module has a neural network configured to optimize the generation of the questionnaire data.

11. A medical imaging apparatus having the system described in any one of claims 1 to 8.

12. A method for generating sensor signal dependency dialogue during a medical imaging process, wherein a computer Steps include measuring patient condition data, The steps include: analyzing the patient's condition data to determine the patient's biometric and health status data; A step of generating questionnaire data for obtaining real-time feedback from the patient during the medical imaging process, wherein the questionnaire data is based on the parameters of the medical imaging process and the determined biometric and health status data of the patient. How to do it.

13. A computer program, when executed by at least one processor, adapted to cause the processor to perform the method described in claim 12.

14. At least one computer-readable medium storing the computer program described in claim 13.

Images