Communication and interaction evaluation system based on multi-modal data fusion of medical-patient interaction and method thereof

Abstract

A communication and interaction evaluation system based on multi-modal data fusion of medical-patient interaction and a method thereof are provided. Instant conversation voices and instant conversation videos are obtained by a communication and interaction analysis server from an audio and video capture device. Medical expression, patient expression, medical emotion, patient emotion, and medical conversation analysis result are obtained by using an expression analysis technology, an emotion analysis technology and a natural language processing technology by the communication and interaction analysis server. Instant expression adjustment suggestion information are generated when medical expression and patient expression are mutually exclusive expression. Instant emotion adjustment suggestion information are generated when medical emotion and patient emotion are mutually exclusive emotion. Instant conversation adjustment suggestion information are generated when medical conversation analysis result are inappropriate conversation. Medical adjust their conversation, expression and emotion based on instant expression adjustment suggestion information, instant emotion adjustment suggestion information or instant conversation adjustment suggestion information. Medical communication and interaction scores per unit period is obtained by using communication and interaction analysis model to interaction pattern analysis with all medical expressions, patient expressions, medical emotions, patient emotions and medical conversation analysis results within the unit period. Therefore, the efficiency of providing empathetic assessment of objective care and patient communication may be achieved.

Description

[Technical Field]

[0001] An assessment system and method thereof, particularly a communication interaction assessment system and method based on the fusion of multimodal data of doctor-patient interaction. [Previous Technology]

[0002] Taiwan has entered an aging society, and the average life expectancy of its people continues to increase. Long-term care and mortality issues affect domestic medical care, the economy, caregiver manpower, and other problems. Therefore, respecting patients' medical autonomy and avoiding suffering caused by ineffective medical treatment and waste of medical energy are crucial. Communication between medical staff and patients should be based on respect for autonomy. Maintaining professionalism, positivity, and active communication and interaction to assist patients is essential. It can be seen from this that the evaluation of communication and interaction between medical staff and patients is a subjective evaluation that lacks objectivity.

[0003] Because human emotions and reactions are very complex and changeable, they have hidden meanings in addition to their surface meanings. For example, when different people say "I'm fine," they may express completely different emotions and reactions. They may be truly calm or they may be hiding sadness. Whether human-computer interaction can be used to analyze the communication interactions between medical staff and patients will be a direction for development in the medical field. If human-computer interaction can analyze the communication interactions between medical staff and patients, it can further develop into human-computer interaction that understands emotions and can make the same emotional responses as humans.

[0004] In summary, it can be seen that the prior art has long suffered from the problem of lack of objective assessment of communication interaction between medical staff and patients. Therefore, it is necessary to propose improved technical means to solve this problem. [Summary of the Invention]

[0005] In view of the problem that prior art lacks objectivity in assessing communication interactions between medical staff and patients, the present invention discloses a communication interaction assessment system and method based on multimodal data fusion of doctor-patient interactions, wherein:

[0006] The communication interaction assessment system based on multimodal data fusion of doctor-patient interaction disclosed in this invention includes: an audio-visual capture device, a medical staff prompting device, and a communication interaction analysis server. The communication interaction analysis server further includes: a non-transitory computer-readable storage medium, a transmission processor, a speech-to-text conversion processor, a doctor-patient facial expression analysis processor, a doctor-patient emotion analysis processor, a doctor-patient conversation analysis processor, and a communication interaction analysis processor.

[0007] The audio-visual capture device instantly acquires the voice and video of the conversation between the doctor and the patient, and the voice and video of the conversation have their own timestamps; the medical staff prompting device receives and displays real-time facial expression adjustment suggestions, real-time emotion adjustment suggestions and / or real-time conversation adjustment suggestions to provide medical staff with real-time adjustments to their conversation, facial expressions and emotions, and receives and displays the medical staff communication interaction score within a unit time period.

[0008] The communication and interaction analysis server is connected to the audio-visual capture device and the medical staff prompting device, respectively. A non-transitory computer-readable storage medium stores multiple computer-readable instructions. The transmission processor is electrically connected to the non-transitory computer-readable storage medium to execute the computer-readable instructions, acquire conversational voice and video from the audio-visual capture device, and provide real-time facial expression adjustment suggestions, real-time emotion adjustment suggestions, and / or real-time conversation adjustment suggestions to the medical staff prompting device, and provide a medical staff communication and interaction score for that unit of time to the medical staff prompting device. The speech-to-text conversion processor is electrically connected to the non-transitory computer-readable storage medium to execute computer-readable instructions, using speech-to-text technology and voice recognition technology to convert real-time conversational voice into conversational text information, with the conversational text information set with a corresponding timestamp based on the conversational voice. The doctor-patient expression analysis processor is electrically connected to the non-transitory computer-readable storage medium to execute computer-readable instructions, using expression analysis technology to perform expression analysis on the real-time conversational voice and video to obtain the expressions of medical staff and patients respectively. When the medical staff... When the patient's expression and emotional state are mutually exclusive, real-time expression adjustment suggestions are generated. The doctor-patient emotion analysis processor is electrically connected to a non-transitory computer-readable storage medium to execute computer-readable instructions. It uses emotion analysis technology to analyze the emotions of real-time conversation videos and text information to obtain the emotions of medical staff and patients. When the emotions of medical staff and patients are mutually exclusive, real-time emotion adjustment suggestions are generated. The doctor-patient conversation analysis processor is electrically connected to a non-transitory computer-readable storage medium to execute computer-readable instructions. It uses natural language processing technology to perform semantic and word analysis on the content of the doctor-patient conversation in the text information to obtain the doctor-patient conversation analysis results. When the doctor-patient conversation analysis results are inappropriate conversations, real-time conversation adjustment suggestions are generated. The communication interaction analysis processor is electrically connected to a non-transitory computer-readable storage medium to execute computer-readable instructions. It uses a communication interaction analysis model to analyze the interaction patterns of all medical staff expressions, patient expressions, medical staff emotions, patient emotions, and doctor-patient conversation analysis results within a unit of time to obtain the doctor-patient communication interaction score for that unit of time.

[0009] The communication interaction assessment method based on multimodal data fusion of doctor-patient interaction disclosed in this invention includes the following steps:

[0010] First, the audio-visual capture device instantly acquires the audio and video of the conversation between the doctor and patient, each with its own timestamp. Next, the communication interaction analysis server connects to the audio-visual capture device and acquires the audio and video of the conversation. Then, the communication interaction analysis server uses speech-to-text technology and voice recognition technology to convert the real-time audio into text information, with the text information having a corresponding timestamp based on the audio. Next, the communication interaction analysis server uses facial expression analysis technology to analyze the real-time video to obtain the expressions of the medical staff and the patient. When the expressions of the medical staff and the patient are mutually exclusive, real-time facial expression adjustment suggestions are generated. Finally, the communication interaction analysis server uses emotion analysis technology to analyze the real-time audio, video, and text information to obtain the emotions of the medical staff and the patient. When the emotions of the medical staff and the patient are mutually exclusive... The system generates real-time emotion adjustment suggestions. Then, the communication interaction analysis server uses natural language processing technology to perform semantic and word analysis on the medical staff's conversation text to obtain the medical staff conversation analysis results. When the medical staff conversation analysis result is inappropriate, it generates real-time conversation adjustment suggestions. Next, the communication interaction analysis server connects to the medical staff prompting device, providing real-time facial expression adjustment suggestions, real-time emotion adjustment suggestions, and / or real-time conversation adjustment suggestions to the medical staff prompting device for display, enabling medical staff to adjust their dialogue, facial expressions, and emotions in real time. Next, the communication interaction analysis server uses a communication interaction analysis model to perform interaction pattern analysis on all medical staff facial expressions, patient facial expressions, medical staff emotions, patient emotions, and medical staff conversation analysis results within a unit of time to obtain a medical staff communication interaction score for that unit of time. Finally, the communication interaction analysis server provides the medical staff communication interaction score for that unit of time to the medical staff prompting device for display.

[0011] The system and method disclosed in this invention are as described above. The communication interaction analysis server acquires real-time conversational voice and video from the audio-visual capture device. The communication interaction analysis server uses facial expression analysis technology, emotion analysis technology, and natural language processing technology to obtain medical staff facial expressions, patient facial expressions, medical staff emotions, patient emotions, and medical staff conversation analysis results. Real-time facial expression adjustment suggestions are generated when medical staff facial expressions and patient facial expressions are mutually exclusive; real-time emotion adjustment suggestions are generated when medical staff emotions and patient emotions are mutually exclusive; or real-time conversation adjustment suggestions are generated when the medical staff conversation analysis result is inappropriate conversation. The communication and interaction analysis server provides real-time facial expression adjustment suggestions, real-time emotion adjustment suggestions, and / or real-time conversation adjustment suggestions to the healthcare staff prompting device and displays them, enabling healthcare staff to adjust their conversations, facial expressions, and emotions in real time. The communication and interaction analysis server uses a communication and interaction analysis model to analyze the interaction patterns of all healthcare staff facial expressions, patient facial expressions, healthcare staff emotions, patient emotions, and healthcare staff conversations within a unit of time to obtain a healthcare staff communication and interaction score for that unit of time. The communication and interaction analysis server provides the healthcare staff communication and interaction score for that unit of time to the healthcare staff prompting device and displays it.

[0012] Through the above-mentioned technical means, the present invention can achieve the technical effect of providing objective communication and interaction assessment between medical staff and patients.

Implementation Method

[0013] The embodiments of the present invention will be described in detail below with reference to the drawings and examples, so that the implementation process of how the present invention uses technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly.

[0014] The following will first describe the communication and interaction assessment system based on multimodal data fusion of doctor-patient interaction disclosed in this invention, and please refer to "Figure 1", which is a system block diagram of the communication and interaction assessment system based on multimodal data fusion of doctor-patient interaction of this invention.

[0015] The communication interaction assessment system based on multimodal data fusion of doctor-patient interaction disclosed in this invention includes: an audio-visual capture device 10, a medical staff prompting device 20, and a communication interaction analysis server 30. The communication interaction analysis server 30 further includes: a non-transitory computer-readable storage medium 31, a transmission processor 32, a speech-to-text conversion processor 33, a doctor-patient facial expression analysis processor 34, a doctor-patient emotion analysis processor 35, a doctor-patient conversation analysis processor 36, and a communication interaction analysis processor 37.

[0016] The audio-visual capture device 10 instantly acquires the voice and video of the conversation between the doctor and the patient, and the voice and video of the conversation have their own timestamps; the audio-visual capture device 10 can be a device that integrates a microphone device and a camera device, or the audio-visual capture device 10 can be a computer device with an external microphone device and a camera device. This is only an example and is not intended to limit the scope of application of the present invention. The audio-visual capture device 10 can instantly acquire the voice of the conversation between the doctor and the patient through the microphone device, and the audio-visual capture device 10 can instantly acquire the video of the conversation between the doctor and the patient through the camera device, and the voice and video of the conversation between the doctor and the patient have their own timestamps.

[0017] The medical care reminder device 20 can be a computer, laptop, tablet, smartphone, etc. that includes a display screen. This is only an example and is not intended to limit the scope of application of the present invention. The communication and interaction analysis server 30 establishes a connection with the audio-visual capture device 10 and the medical care reminder device 20 through wired or wireless transmission methods. The aforementioned wired transmission methods are, for example, cable networks, fiber optic networks, etc., and the aforementioned wireless transmission methods are, for example, Wi-Fi, mobile communication networks (e.g., 4G, 5G, etc.). This is only an example and is not intended to limit the scope of application of the present invention.

[0018] The non-transitory computer-readable storage medium 31 stores a plurality of computer-readable instructions. The transmission processor 32, the speech-to-text conversion processor 33, the doctor-patient expression analysis processor 34, the doctor-patient emotion analysis processor 35, the doctor-patient conversation analysis processor 36, and the communication interaction analysis processor 37 are all electrically connected to the non-transitory computer-readable storage medium 31 to execute the computer-readable instructions.

[0019] The transmission processor 32 acquires real-time conversational speech and video from the audio-visual capture device 10. Then, the speech-to-text processor 33 uses speech-to-text technology and voice recognition technology to convert the real-time conversational speech into conversational text information. The speech-to-text technology uses an acoustic model and a language model to convert speech signals into text. The conversational speech is converted into phonemes (the smallest phonemic units in a language) by the acoustic model. Then, the language model uses context and grammatical rules to predict the word order corresponding to the phonemes to improve the accuracy of the converted text. The voice recognition technology analyzes the conversational speech to identify the voiceprint features of medical staff and patients. The aforementioned voiceprint features include, for example, pitch, intensity, speech rate, and formants, etc. These are only examples and are not intended to limit the scope of application of the present invention. By using speech-to-text technology and voice recognition technology, the conversational speech can be converted into conversational text information containing the conversational text content of medical staff and patients. The conversational text information is set with a corresponding timestamp based on the timestamp of the conversational speech.

[0020] Next, the doctor-patient expression analysis processor 34 uses facial expression recognition (FER) technology to analyze the real-time conversation video to obtain the expressions of the medical staff and the patient. When the expressions of the medical staff and the patient are mutually exclusive, real-time expression adjustment suggestion information is generated. Specifically, if the expression of the medical staff is a "happy expression" and the expression of the patient is a "sad expression", then the expressions of the medical staff and the patient are mutually exclusive. The doctor-patient expression analysis processor 34 will then generate real-time expression adjustment suggestion information, such as: "The patient is in a sad state, please treat him with a serious expression." This is only an example and is not intended to limit the scope of application of the present invention.

[0021] When the doctor-patient expression analysis processor 34 generates real-time expression adjustment suggestion information, it can provide the real-time expression adjustment suggestion information to the medical staff prompting device 20 in real time through the transmission processor 32. The medical staff prompting device 20 can display the real-time expression adjustment suggestion information in real time after receiving it from the self-communication interaction analysis server 30, so that medical staff can adjust their expressions in real time.

[0022] Expression analysis technology is a technology based on artificial intelligence (AI) and computer vision. The medical staff and patient expression analysis processor 34 first uses facial detection algorithms (e.g., Haar feature classifier, deep learning model, etc.) to locate the facial regions of medical staff and patients in the real-time conversation video. Then, it marks the key parts of the medical staff and patients' facial regions (e.g., corners of the eyes, corners of the mouth, eyebrows, etc.) and extracts the geometric or appearance features representing expressions from these key parts. Then, based on the facial regions of medical staff and patients, the marked key parts, and the geometric or appearance features of expressions, it uses machine learning models or deep learning models to map them to predefined expression types (e.g., happiness, sadness, surprise, anger, disgust, fear, etc.).

[0023] Next, the doctor-patient emotion analysis processor 35 uses emotion analysis technology (Emotion Recognition or Sentiment Analysis) to perform emotion analysis on real-time conversational voice, video, and text information to obtain the emotions of medical staff and patients. When the emotions of medical staff and patients are mutually exclusive, real-time emotion adjustment suggestions are generated. Specifically, if the emotions of medical staff are "excited" and the emotions of patients are "sad", then the emotions of medical staff and patients are mutually exclusive, and the doctor-patient emotion analysis processor 35 will generate real-time emotion adjustment suggestions. For example, the real-time emotion adjustment suggestions are: "The patient is in a sad mood, and being too excited will not be conducive to doctor-patient communication." This is only an example and is not intended to limit the scope of application of the present invention.

[0024] When the patient-doctor emotion analysis processor 35 generates real-time emotion adjustment suggestion information, it can be provided to the medical staff prompting device 20 in real time through the transmission processor 32. The medical staff prompting device 20 can display the real-time emotion adjustment suggestion information received by the self-communication interaction analysis server 30 in real time, so that medical staff can adjust their emotions in real time.

[0025] Emotion analysis technology is an artificial intelligence technology that provides the ability to identify and analyze an individual's emotions. It infers an individual's emotional response by processing data such as voice, text, and facial expressions. The doctor-patient expression analysis processor 34 uses Natural Language Processing (NLP) technology to extract words and sentences with emotional connotations based on vocabulary and grammar to infer the emotions of the medical staff and the patient. The doctor-patient expression analysis processor 34 infers the emotions of the medical staff and the patient based on the voiceprint features of the conversation. For example, a high-pitched voiceprint feature infers excitement, while a low-pitched voice infers sadness. This is only an example and does not limit the scope of application of the present invention. The doctor-patient expression analysis processor 34 calls the medical staff's and patient's facial expressions obtained by the doctor-patient expression analysis processor 34 and combines them with the inferred emotions of the medical staff and the patient to conduct a comprehensive analysis to obtain the emotions of the medical staff and the patient.

[0026] The doctor-patient conversation analysis processor 36 uses natural language processing technology to perform semantic and word analysis on the doctor-nurse conversation content in the text information to obtain the doctor-nurse conversation analysis result. When the doctor-nurse conversation analysis result is inappropriate conversation, it generates real-time conversation adjustment suggestion information. Specifically, if the doctor-nurse conversation content in the text information is "You are already in stage four of cancer, you don't need treatment anymore.", the doctor-patient conversation analysis processor 36 will inevitably obtain the doctor-nurse conversation analysis result as "inappropriate conversation" after using natural language processing technology to perform semantic and word analysis on the doctor-nurse conversation content in the text information. The doctor-patient conversation analysis processor 36 will then generate real-time conversation adjustment suggestion information. The real-time conversation adjustment suggestion information is, for example, "The conversation content is too cold. It is recommended to supplement the subsequent conversation with palliative care." This is only an example and does not limit the application scope of the present invention.

[0027] When the doctor-patient conversation analysis processor 36 generates real-time conversation adjustment suggestion information, it can provide the real-time conversation adjustment suggestion information to the medical staff prompting device 20 in real time through the transmission processor 32. The medical staff prompting device 20 can display the real-time conversation adjustment suggestion information in real time after receiving it from the self-communication interaction analysis server 30, so that medical staff can adjust the conversation in real time.

[0028] The communication and interaction analysis processor 37 uses a communication and interaction analysis model to perform interaction pattern analysis on all medical staff expressions, patient expressions, medical staff emotions, patient emotions, and medical staff conversation analysis results within a unit time period to obtain a medical staff communication and interaction score for the unit time period. When the communication and interaction analysis processor 37 generates the medical staff communication and interaction score for the unit time period, it can provide the medical staff communication and interaction score for the unit time period to the medical staff prompting device 20 through the transmission processor 32. The medical staff prompting device 20 can display the medical staff communication and interaction score for the unit time period received from the communication and interaction analysis server 30, so that medical staff can know the level of communication and interaction in the doctor-patient conversation.

[0029] The communication interaction analysis model uses natural language processing technology, deep neural networks (ANNs), long short-term memory networks (LSTM), and Transformer models, etc., as examples only, and does not limit the application scope of the present invention. The communication interaction analysis model can perform functions such as emotion analysis, semantic understanding, emotion simulation, and understanding empathetic responses, etc., to analyze the interaction patterns between medical staff and patients, and then score the interaction patterns between medical staff and patients to obtain a medical staff communication interaction score.

[0030] It is worth noting that the doctor-patient expression analysis processor 34 further includes, when the expressions of the medical staff and the patient are mutually exclusive, extracting the frames of the real-time conversation video based on the timestamps corresponding to the expressions of the medical staff and the patient as real-time expression warning images, and providing the real-time expression warning images to the medical staff prompting device 20 through the transmission processor 32. The medical staff prompting device 20 can display the real-time expression warning images received by the self-communication interaction analysis server 30, so that medical staff can refer to the real-time expression warning images to adjust their expressions. The doctor-patient expression analysis processor 34 further marks the positions of the medical staff and the patient in the real-time expression warning images according to the expressions of the medical staff and the patient, respectively.

[0031] It is worth noting that the doctor-patient emotion analysis processor 35 further includes capturing real-time video frames of conversations as real-time emotion warning images based on the timestamps corresponding to the doctor-patient emotions and patient emotions when the doctor-patient emotions and patient emotions are mutually exclusive. The real-time emotion warning images are provided to the doctor-patient prompting device 20 through the transmission processor 32. The doctor-patient prompting device 20 can display the real-time emotion warning images received by the self-communication interaction analysis server 30, so that the doctor-patient staff can refer to the real-time emotion warning images to adjust their emotions. The doctor-patient emotion analysis processor 35 further marks the positions of the doctor-patient staff and the patient in the real-time emotion warning images according to the doctor-patient emotions and patient emotions respectively.

[0032] It is worth noting that the doctor-patient conversation analysis processor 36 further includes the function of extracting real-time conversation voice as real-time conversation warning voice based on the timestamp corresponding to the medical staff conversation analysis result when the medical staff conversation analysis result is inappropriate. The real-time conversation warning voice is provided to the medical staff prompting device 20 through the transmission processor 32. The medical staff prompting device 20 can play the real-time conversation warning voice when it receives the real-time conversation warning voice from the self-communication interaction analysis server 30 and when the real-time conversation warning voice is selected to play, so that medical staff can refer to the real-time conversation warning voice to adjust their conversation.

[0033] When the audio-visual capture device 10 provides a conversation end command to the communication interaction analysis server 30, the communication interaction analysis processor 37 uses the communication interaction analysis model to perform interaction pattern analysis on all medical staff expressions, patient expressions, medical staff emotions, patient emotions, and medical staff conversation analysis results to obtain the total score of medical staff communication interaction, and integrates the medical staff communication interaction scores corresponding to each unit time period according to the time sequence to form a medical staff communication interaction score graph 41. For a schematic diagram of the medical staff communication interaction score graph 41, please refer to "Figure 2A". "Figure 2A" is a schematic diagram of the medical staff communication interaction score graph of the communication interaction evaluation based on multimodal data fusion of medical staff interaction of the present invention. The transmission processor 32 provides the total score of medical staff communication interaction and / or the medical staff communication interaction score graph 41 to the medical staff prompting device 20. The medical staff prompting device 20 receives the total score of medical staff communication interaction and / or the medical staff communication interaction score graph 41 from the communication interaction analysis server 30 and displays it, so that medical staff can know the degree of communication interaction and the changes in communication interaction throughout the entire medical staff conversation process.

[0034] The communication and interaction analysis processor 37 can also use the communication and interaction analysis model to perform interaction pattern analysis on the medical staff's facial expressions, emotions, and conversations within a unit time period to obtain the medical staff score within the unit time period. Then, it integrates the medical staff scores corresponding to each unit time period according to the time sequence to generate a medical staff score graph 42. For a schematic diagram of the medical staff score graph 42, please refer to "Figure 2B". "Figure 2B" is a schematic diagram of the medical staff score graph of the communication and interaction assessment based on multimodal data fusion of doctor-patient interaction of the present invention. The medical staff score graph 42 is provided to the medical staff prompting device 20 through the transmission processor 32. The medical staff prompting device 20 receives the medical staff score graph 42 from the communication and interaction analysis server 30 and displays it.

[0035] The communication and interaction analysis processor 37 can also use the communication and interaction analysis model to analyze the patient's facial expressions and emotions within a unit time period to obtain a patient score. Then, according to the time sequence, the patient scores corresponding to each unit time period are integrated into a patient score graph 43. For a schematic diagram of the patient score graph 43, please refer to "Figure 2C". "Figure 2C" is a schematic diagram of the patient score graph of the communication and interaction assessment based on multimodal data fusion of doctor-patient interaction of the present invention. The patient score graph 43 is provided to the medical care prompting device 20 through the transmission processor 32. The medical care prompting device 20 receives the patient score graph 43 from the communication and interaction analysis server 30 and displays it.

[0036] The communication interaction analysis processor 37 further integrates the doctor-patient communication interaction scores corresponding to each unit time period into a doctor-patient communication interaction score graph according to the time sequence. In the doctor-patient communication interaction score graph 41, the first warning point 411 of the first time stamp t1 and the second warning point 412 of the second time stamp t2 are marked (please refer to "Figure 2A"). This is only an example and does not limit the application scope of the present invention. The first warning point 411 and the second warning point 412 are associated with real-time facial expression warning images, real-time emotion warning images and / or real-time conversation warning voice based on the time stamp.

[0037] The communication and interaction analysis processor 37 further integrates the medical staff scores corresponding to each unit time period into a medical staff score graph according to the time sequence. It also marks the first warning point 421 of the first time stamp t1 and the second warning point 422 of the second time stamp t2 in the medical staff score graph 42 (please refer to "Figure 2B"). This is only an example and does not limit the application scope of the present invention. The first warning point 421 and the second warning point 422 are associated with real-time facial expression warning images, real-time emotion warning images and / or real-time conversation warning voice based on the timestamp.

[0038] When the communication and interaction analysis processor 37 further integrates the patient scores corresponding to each unit time period into a patient score graph according to the time sequence, it also marks the first warning point 431 of the first timestamp t1 and the second warning point 432 of the second timestamp t2 in the patient score graph (please refer to "Figure 2C"). This is only an example and does not limit the application scope of the present invention. The first warning point 431 and the second warning point 432 are associated with real-time facial expression warning images, real-time emotion warning images and / or real-time conversation warning voice based on the timestamp.

[0039] When one of the first warning point 411 in the doctor-patient communication interaction scoring chart 41, the first warning point 421 in the doctor-patient scoring chart 42, or the first warning point 431 in the patient scoring chart 43 displayed on the medical staff prompting device 10 is selected, the medical staff prompting device 10 provides the timestamp corresponding to the first warning point 411, the first warning point 421, or the first warning point 431 to the communication interaction analysis server 30. The communication interaction analysis server 30 finds the corresponding real-time facial expression warning image, real-time emotion warning image, and / or real-time conversation warning voice based on the timestamp, and then provides the found real-time facial expression warning image, real-time emotion warning image, and / or real-time conversation warning voice to the medical staff prompting device 10 for display and / or playback.

[0040] In addition, when the first warning point 411 in the doctor-patient communication interaction rating chart 41 displayed by the medical staff prompt device 10 is selected, the timestamp corresponding to the first warning point 411 is t1. The first warning point 421 in the medical staff rating chart 42 and the first warning point 431 in the patient rating chart 43, which are based on the timestamp t1, will also be highlighted, so as to provide medical staff with the opportunity to refer to the doctor-patient communication interaction rating chart 41, the medical staff rating chart 42 and the patient rating chart 43 to know the time points of the problems that medical staff should improve.

[0041] The communication and interaction analysis processor 37 further calculates the difference between the medical staff communication and interaction score of the current unit time period and the medical staff communication and interaction score of the previous unit time period. When the absolute value of the difference is greater than or equal to the adaptive threshold, an adaptive adjustment parameter is generated based on the difference. The medical staff expression analysis processor 34 will select different facial detection algorithms and / or increase / decrease key parts based on the adaptive adjustment parameter to achieve the adaptive adjustment of the medical staff expression analysis processor 34.

[0042] The communication and interaction analysis processor 37 further calculates the difference between the medical staff communication and interaction score of the current unit time period and the medical staff communication and interaction score of the previous unit time period. When the absolute value of the difference is greater than or equal to the adaptive threshold, an adaptive adjustment parameter is generated based on the difference. The medical staff and patient emotion analysis processor 35 will increase / decrease the usage data of voice, text, facial expressions, etc. based on the adaptive adjustment parameter to realize the adaptive adjustment of the medical staff and patient emotion analysis processor 35.

[0043] The communication and interaction analysis processor 37 further calculates the difference between the medical staff communication and interaction score of the current unit time period and the medical staff communication and interaction score of the previous unit time period. When the absolute value of the difference is greater than or equal to the adaptive threshold, an adaptive adjustment parameter is generated based on the difference. The medical staff conversation analysis processor 36 will increase / decrease the vocabulary, grammar, etc. used by the natural language processing technology according to the adaptive adjustment parameter in order to realize the adaptive adjustment of the medical staff conversation analysis processor 36.

[0044] It is particularly noted that, in practical implementation, the invention may be realized, either partially or completely based on hardware, e.g., one or more components in the system may be implemented via an integrated circuit chip, System on Chip (SoC), Complex Programmable Logic Device (CPLD), Field Programmable Logic Device (CPLD), Field Programmable Logic Array (Field Programmable Logic Device (CPLD), etc. The non-transient computer-readable storage media of the invention, which uploads computer-readable instructions (or referred to as computer program instructions) for enabling the processor to implement various aspects of the invention, the non-transient computer-readable storage media may be a tangible device that can hold and store instructions for use by the command execution equipment. Non-transient computer-readable storage media may be, but are not limited to, electrical storage equipment, magnetic storage equipment, optical storage equipment, electromagnetic storage equipment, semiconductor storage equipment, or any suitable combination of the above. More specific examples of computer-readable storage media (not an exhaustive list) include: hard drives, random access memory, read-only memory, flash memory, optical discs, floppy disks, and any suitable combination of the above. The non-transient computer-readable storage media used here are not construed as instantaneous signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated through waveguides or other transmission media (e.g., optical signals via fiber optic cables), or electrical signals transmitted through wires. In addition, the computer-readable commands described here can be downloaded from non-transient computer-readable storage media to various computing / processing equipment, or to external computer equipment or external storage equipment via a network, such as: an Internet, LAN, WAN, and / or wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, hubs and / or gateways. The network card or network interface in each computing / processing equipment receives computer-readable instructions from the network and forwards these computer-readable instructions for storage in the non-transient computer-readable storage media in the respective computing / processing equipment. Computer-readable instructions to perform operations of the present invention may be combination language instructions, instruction set architecture instructions, machine instructions, machine-related instructions, microinstructions, firmware instructions, or source code or object code (Object Code) written in any combination of one or more programming languages, including object-oriented programming languages, such as Common Lisp, Python, C++, Objective-C, Smalltalk, Delphi, Java, Swift, C#, Perl, Ruby and PHP, etc., as well as conventional procedural (Procedural) programming languages, such as C language or similar programming languages.

[0045] Next, the operation method of the present invention will be described below, and please refer to Figure 3A and Figure 3B at the same time. Figure 3A and Figure 3B are flowcharts of the communication interaction assessment method based on multimodal data fusion of doctor-patient interaction of the present invention.

[0046] The communication interaction assessment method based on multimodal data fusion of doctor-patient interaction disclosed in this invention includes the following steps:

[0047] First, the audio-visual capture device acquires the voice and video of the conversation between the doctor and the patient in real time, with each voice and video having its own timestamp (step 501); next, the communication interaction analysis server connects to the audio-visual capture device and acquires the voice and video of the conversation from the audio-visual capture device (step 502); next, the communication interaction analysis server uses speech-to-text technology and voice recognition technology to convert the real-time voice conversation into text information, with the text information having a corresponding timestamp set according to the voice conversation (step 503); next, the communication interaction analysis server uses facial expression analysis technology to perform facial expression analysis on the real-time video conversation to obtain the expressions of the doctor and patient respectively, and when the expressions of the doctor and patient are mutually exclusive, real-time facial expression adjustment suggestion information is generated (step 504); next, the communication interaction analysis server uses emotion analysis technology to perform emotion analysis on the real-time video conversation and text information to obtain the emotions of the doctor and patient, and when the emotions of the doctor and patient are mutually exclusive, real-time emotion adjustment suggestion information is generated. Emotional adjustment suggestion information (step 505); Next, the communication interaction analysis server uses natural language processing technology to perform semantic and word analysis on the medical staff conversation content in the conversation text information to obtain the medical staff conversation analysis results. When the medical staff conversation analysis result is inappropriate conversation, real-time conversation adjustment suggestion information is generated (step 506); Next, the communication interaction analysis server connects to the medical staff prompting device and provides real-time facial expression adjustment suggestion information, real-time emotion adjustment suggestion information, and / or real-time conversation adjustment suggestion information to the medical staff prompting device and displays it, so as to provide medical staff with real-time adjustment of dialogue, facial expressions, and emotions (step 507); Next, the communication interaction analysis server uses the communication interaction analysis model to perform interaction pattern analysis on all medical staff facial expressions, patient facial expressions, medical staff emotions, patient emotions, and medical staff conversation analysis results within a unit time period to obtain the medical staff communication interaction score for the unit time period (step 508); And, the communication interaction analysis server provides the medical staff communication interaction score within the unit time period to the medical staff prompting device and displays it (step 509).

[0048] In summary, the communication and interaction analysis server obtains real-time conversational audio and video from the audio-visual capture device. The communication and interaction analysis server uses facial expression analysis technology, emotion analysis technology, and natural language processing technology to obtain the analysis results of medical staff facial expressions, patient facial expressions, medical staff emotions, patient emotions, and medical staff conversations. When medical staff facial expressions and patient facial expressions are mutually exclusive, it generates real-time facial expression adjustment suggestions; when medical staff emotions and patient emotions are mutually exclusive, it generates real-time emotion adjustment suggestions; or when the medical staff conversation analysis result is inappropriate conversation, it generates real-time conversation adjustment suggestions. The communication and interaction analysis server provides real-time facial expression adjustment suggestions, real-time emotion adjustment suggestions, and / or real-time conversation adjustment suggestions to the medical staff prompting device and displays them, so as to enable medical staff to adjust their conversations, facial expressions, and emotions in real time. The communication and interaction analysis server uses the communication and interaction analysis model to perform interaction pattern analysis on all medical staff facial expressions, patient facial expressions, medical staff emotions, patient emotions, and medical staff conversation analysis results within a unit time period to obtain the medical staff communication and interaction score for that unit time period. The communication and interaction analysis server provides the medical staff communication and interaction score for that unit time period to the medical staff prompting device and displays it.

[0049] This technical means can solve the problem of the lack of objectivity in the assessment of communication interaction between medical staff and patients in the previous technology, thereby achieving the technical effect of providing objective assessment of communication interaction between medical staff and patients.

[0050] Although the embodiments disclosed in this invention are as described above, the content described is not intended to directly limit the scope of patent protection of this invention. Anyone skilled in the art to which this invention pertains may make some modifications in form and detail of the implementation without departing from the spirit and scope disclosed in this invention. The scope of patent protection of this invention shall still be determined by the appended claims. [Simplified Explanation of the Diagram]

[0051] Figure 1 is a system block diagram of the communication interaction assessment system based on multimodal data fusion of doctor-patient interaction according to the present invention. Figure 2A is a schematic diagram of the doctor-patient communication interaction scoring chart of the communication interaction assessment system based on multimodal data fusion of doctor-patient interaction according to the present invention. Figure 2C is a schematic diagram of the patient scoring chart of the communication interaction assessment system based on multimodal data fusion of doctor-patient interaction according to the present invention. Figure 2B is a schematic diagram of the medical staff scoring chart of the communication interaction assessment system based on multimodal data fusion of doctor-patient interaction according to the present invention. Figures 3A and 3B are flowcharts of the communication interaction assessment method based on multimodal data fusion of doctor-patient interaction according to the present invention.

Claims

1. A communication interaction assessment system based on multimodal data fusion of doctor-patient interaction, comprising: an audio-visual capture device for real-time acquisition of a conversation between a doctor and a patient, wherein the audio-visual conversation and the video conversation have their respective timestamps; a medical staff prompting device for receiving and displaying real-time facial expression adjustment suggestions, real-time emotion adjustment suggestions, and / or real-time conversation adjustment suggestions, to provide medical staff with real-time adjustments to their dialogue, facial expressions, and emotions, and for receiving and displaying a medical staff communication interaction score within a unit time period; and a communication interaction analysis server connected to the audio-visual capture device and the medical staff prompting device, wherein the communication interaction analysis server further comprises: a non-transitory computer-readable storage medium for storing multiple computer-readable instructions; A transmission processor, electrically connected to the non-transitory computer-readable storage medium to execute the computer-readable instructions, acquires the conversational voice and the conversational video from the audio-visual capture device, provides real-time facial expression adjustment suggestion information, real-time emotion adjustment suggestion information, and / or real-time conversation adjustment suggestion information to the medical staff prompting device, and provides the medical staff communication interaction score for the unit time period to the medical staff prompting device; a speech-to-text processor, electrically connected to the non-transitory computer-readable storage medium to execute the computer-readable instructions, uses speech-to-text technology and voice recognition technology to convert the real-time conversational voice into conversational text information, the conversational text information being set with a corresponding timestamp based on the conversational voice; a doctor-patient facial expression analysis processor, electrically connected to the non-transitory computer-readable storage medium to execute the computer-readable instructions, uses facial expression analysis technology to perform facial expression analysis on the real-time conversational video to obtain a doctor-patient facial expression and a patient facial expression respectively, and when the doctor-patient facial expression and the patient facial expression are mutually exclusive, generates the real-time facial expression adjustment suggestion information; A doctor-patient emotion analysis processor, electrically connected to the non-transitory computer-readable storage medium to execute the computer-readable instructions, uses an emotion analysis technology to perform emotion analysis on the real-time conversational audio, video, and text information to obtain a doctor-caregiver emotion and a patient emotion. When the doctor-caregiver emotion and the patient emotion are mutually exclusive, it generates real-time emotion adjustment suggestion information. A doctor-patient conversation analysis processor, electrically connected to the non-transitory computer-readable storage medium to execute the computer-readable instructions, uses a natural language processing technology to perform semantic and word analysis on the doctor-caregiver conversation content in the text information to obtain a doctor-caregiver conversation analysis result. When the doctor-caregiver conversation analysis result indicates inappropriate conversation, it generates real-time conversation adjustment suggestion information.A communication interaction analysis processor, electrically connected to the non-transitory computer-readable storage medium to execute the computer-readable instructions, uses a communication interaction analysis model to perform interaction pattern analysis on all medical staff facial expressions, patient facial expressions, medical staff emotions, patient emotions, and medical staff conversation analysis results within the unit time period to obtain the medical staff communication interaction score for the unit time period.

2. The communication interaction assessment system based on multimodal data fusion of doctor-patient interaction as described in claim 1, wherein the communication interaction analysis processor uses the communication interaction analysis model to perform interaction pattern analysis on all the doctor-staff facial expressions, the patient facial expressions, the doctor-staff emotions, the patient emotions, and the analysis results of the doctor-staff conversation to obtain a total doctor-staff communication interaction score, and integrates the doctor-patient communication interaction scores corresponding to each unit time period according to time sequence into a doctor-patient communication interaction score graph, and the transmission processor provides the total doctor-staff communication interaction score and / or the doctor-patient communication interaction score graph to the doctor-staff prompting device for display.

3. The communication interaction assessment system based on multimodal data fusion of doctor-patient interaction as described in claim 1, wherein the communication interaction analysis processor further comprises calculating a difference between the doctor-patient communication interaction score in the current unit time period and the doctor-patient communication interaction score in the previous unit time period, generating an adaptive adjustment parameter based on the difference when the absolute value of the difference is greater than or equal to an adaptive threshold, and the doctor-patient expression analysis processor performing adaptive adjustment based on the adaptive adjustment parameter; the communication interaction analysis processor further comprises calculating the difference between the doctor-patient communication interaction score in the current unit time period and the doctor-patient communication interaction score in the previous unit time period, generating the adaptive adjustment parameter based on the difference when the absolute value of the difference is greater than or equal to the adaptive threshold, and the doctor-patient emotion analysis processor performing adaptive adjustment based on the adaptive adjustment parameter; and the communication interaction analysis processor further comprises calculating the difference between the doctor-patient communication interaction score in the current unit time period and the doctor-patient communication interaction score in the previous unit time period, generating the adaptive adjustment parameter based on the difference when the absolute value of the difference is greater than or equal to the adaptive threshold, and the doctor-patient conversation analysis processor performing adaptive adjustment based on the adaptive adjustment parameter.

4. The communication interaction assessment system based on multimodal data fusion of doctor-patient interaction as described in claim 1, wherein the communication interaction analysis processor uses the communication interaction analysis model to perform interaction pattern analysis on the doctor-patient facial expressions, doctor-patient emotions, and doctor-patient conversation analysis results within a unit time period to obtain a doctor-patient score within a unit time period, and then integrates the doctor-patient scores corresponding to each unit time period according to the time sequence to generate a doctor-patient score graph, and provides the doctor-patient score graph to the doctor-patient prompting device for display through the transmission processor; the communication interaction analysis processor uses the communication interaction analysis model to perform interaction pattern analysis on the patient facial expressions and patient emotions within a unit time period to obtain a patient score within a unit time period, and then integrates the patient scores corresponding to each unit time period according to the time sequence to generate a patient score graph, and provides the patient score graph to the doctor-patient prompting device for display through the transmission processor.

5. The communication interaction assessment system based on multimodal data fusion of doctor-patient interaction as described in claim 1, wherein the doctor-patient expression analysis processor further includes, when the doctor's expression and the patient's expression are mutually exclusive, extracting a frame of the real-time conversation video as a real-time expression warning image based on the timestamps corresponding to the doctor's expression and the patient's expression; the doctor-patient emotion analysis processor further includes, when the doctor's emotion and the patient's emotion are mutually exclusive, extracting a frame of the real-time conversation video as a real-time emotion warning image based on the timestamps corresponding to the doctor's emotion and the patient's emotion; the doctor-patient conversation analysis processor further includes, when the doctor-patient conversation analysis result is inappropriate conversation, extracting a real-time conversation voice as a real-time conversation warning voice based on the timestamps corresponding to the doctor-patient conversation analysis result.

6. A communication interaction assessment method based on multimodal data fusion of doctor-patient interaction, comprising the following steps: A video capture device acquires in real-time a conversational audio and a conversational video between a doctor and a patient, wherein the audio and video have their own timestamps; A communication interaction analysis server is connected to the video capture device and acquires the audio and video from the video capture device; The communication interaction analysis server uses speech-to-text technology and voice recognition technology to convert the real-time audio into text information, wherein the text information is set with a corresponding timestamp based on the audio; The communication interaction analysis server uses facial expression analysis technology to perform facial expression analysis on the real-time conversational video to obtain a doctor's facial expression and a patient's facial expression respectively; when the doctor's facial expression and the patient's facial expression are mutually exclusive, real-time facial expression adjustment suggestion information is generated; The communication interaction analysis server uses an emotion analysis technology to perform emotion analysis on the real-time conversational audio, video, and text information to obtain a medical staff emotion and a patient emotion. When the medical staff emotion and the patient emotion are mutually exclusive, an real-time emotion adjustment suggestion is generated. The communication interaction analysis server uses a natural language processing technology to perform semantic and word analysis on the medical staff conversation content in the text information to obtain a medical staff conversation analysis result. When the medical staff conversation analysis result is inappropriate conversation, an real-time conversation adjustment suggestion is generated. The communication interaction analysis server is connected to a medical staff prompting device, providing the real-time facial expression adjustment suggestion, the real-time emotion adjustment suggestion, and / or the real-time conversation adjustment suggestion to the medical staff prompting device and displaying it, so as to enable medical staff to adjust their dialogue, facial expressions, and emotions in real time. The communication interaction analysis server uses a communication interaction analysis model to perform interaction pattern analysis on all medical staff facial expressions, patient facial expressions, medical staff emotions, patient emotions, and medical staff conversation analysis results within a unit time period to obtain the medical staff communication interaction score for that unit time period; and the communication interaction analysis server provides the medical staff communication interaction score for that unit time period to the medical staff prompting device and displays it.

7. The communication interaction assessment method based on multimodal data fusion of doctor-patient interaction as described in claim 6, wherein the communication interaction analysis server uses the communication interaction analysis model to perform interaction pattern analysis on all the doctor-staff facial expressions, the patient facial expressions, the doctor-staff emotions, the patient emotions, and the doctor-staff conversation analysis results to obtain a total doctor-staff communication interaction score, and integrates the doctor-patient communication interaction scores corresponding to each unit time period according to time sequence to form a doctor-patient communication interaction score graph, and the communication interaction analysis server provides the total doctor-staff communication interaction score and / or the doctor-patient communication interaction score graph to the doctor-staff prompting device for display.

8. The communication interaction assessment method based on multimodal data fusion of doctor-patient interaction as described in claim 6, wherein the communication interaction analysis server further includes calculating a difference between the doctor-patient communication interaction score in the current unit time period and the doctor-patient communication interaction score in the previous unit time period; when the absolute value of the difference is greater than or equal to an adaptive threshold, an adaptive adjustment parameter is generated based on the difference; the communication interaction analysis server adaptively adjusts the facial expression analysis technology used based on the adaptive adjustment parameter; the communication interaction analysis server further includes calculating the doctor-patient communication interaction score in the current unit time period and the doctor-patient communication interaction score in the previous unit time period. The communication interaction analysis server adaptively adjusts the emotion analysis technology based on the adaptive adjustment parameters when the absolute value of the difference is greater than or equal to the adaptive threshold; and the communication interaction analysis server further includes calculating the difference between the medical staff communication interaction score in the current unit time period and the medical staff communication interaction score in the previous unit time period, generating the adaptive adjustment parameters based on the difference when the absolute value of the difference is greater than or equal to the adaptive threshold, and the communication interaction analysis server adaptively adjusts the natural language processing technology based on the adaptive adjustment parameters.

9. The communication interaction assessment method based on multimodal data fusion of doctor-patient interaction as described in claim 6, wherein the communication interaction analysis server uses the communication interaction analysis model to perform interaction pattern analysis on the doctor-patient facial expressions, doctor-patient emotions, and doctor-patient conversation analysis results within a unit time period to obtain a doctor-patient score within a unit time period, and then integrates the doctor-patient scores corresponding to each unit time period according to the time sequence to generate a doctor-patient score graph, and provides the doctor-patient score graph to the doctor-patient prompting device for display through the communication interaction analysis server; the communication interaction analysis server uses the communication interaction analysis model to perform interaction pattern analysis on the patient facial expressions and patient emotions within a unit time period to obtain a patient score within a unit time period, and then integrates the patient scores corresponding to each unit time period according to the time sequence to generate a patient score graph, and provides the patient score graph to the doctor-patient prompting device for display through the communication interaction analysis server.

10. The communication interaction assessment method based on multimodal data fusion of doctor-patient interaction as described in claim 6, wherein the communication interaction analysis server further includes, when the doctor's facial expression and the patient's facial expression are mutually exclusive, extracting a frame of the real-time conversation video as a real-time facial expression warning image based on the timestamps corresponding to the doctor's facial expression and the patient's facial expression; the communication interaction analysis server further includes, when the doctor's emotion and the patient's emotion are mutually exclusive, extracting a frame of the real-time conversation video as a real-time emotion warning image based on the timestamps corresponding to the doctor's emotion and the patient's emotion; the communication interaction analysis server further includes, when the doctor-patient conversation analysis result is inappropriate conversation, extracting a real-time conversation voice as a real-time conversation warning voice based on the timestamps corresponding to the doctor-patient conversation analysis result.

Images