Doctor-patient experience real-time monitoring method based on cloud data analysis platform

Abstract

The invention belongs to the technical field of medical information, and particularly relates to a doctor-patient experience real-time monitoring method based on a cloud data analysis platform. According to the scheme, image information in the treatment process is monitored in real time, and voice information input by a patient terminal and a medical care terminal is collected; the expressions and behaviors of the patient and the medical staff in the image information are recognized and processed into a first emotion factor, and the voice of the patient or the medical staff in the voice information is recognized and processed into a second emotion factor; and finally, the emotion of the patient or the medical staff is quantified into an emotion value through the first emotion factor and the second emotion factor, and the emotion value is monitored and managed in real time. When the emotion value of the patient or the medical staff exceeds a preset emotion threshold value, information of consulting room replacement and emotion pacifying is sent to the patient terminal and the medical terminal. According to the scheme, the problem that doctor-patient experience data cannot be timely and accurately obtained in the doctor seeing process of a patient is solved.

Description

Technical Field

[0001] This solution belongs to the field of medical information technology, and specifically involves a real-time monitoring method for doctor-patient experience based on a cloud data analysis platform. Background Art

[0002] With the development of the medical industry, patient experience has become one of the important indicators of hospital management. Traditional methods of monitoring doctor-patient experience usually rely on regular questionnaires and satisfaction assessments. These methods often have problems such as delayed feedback and untimely data updates, making it difficult to reflect the true feelings of patients during the medical treatment process. During the medical treatment process, patients' feedback is often concentrated on certain key time nodes rather than real-time dynamic changes. Due to the lag in feedback collection, the feedback raised by patients cannot be processed and satisfied in a timely manner, making the hospital lack of pertinence in service improvement.

[0003] With the advancement of medical informatization, although many hospitals have begun to introduce information systems for data management, most systems are still unable to achieve real-time collection and analysis of data directly related to patient experience, and patients' demands are still not effectively responded to and processed in a timely manner, resulting in the management being unable to grasp changes in service quality in a timely manner. Existing technologies lack effective integration methods and cannot aggregate and analyze data scattered in different systems, resulting in information islands.

[0004] In this context, cloud data analysis technology has gradually attracted attention. Cloud computing has powerful data processing and storage capabilities and can handle large-scale medical data. However, there is currently no effective method to apply cloud technology to real-time monitoring of doctor-patient experience, resulting in hospital managers not having a comprehensive understanding of patients' real needs and feelings. Therefore, developing a real-time monitoring method for doctor-patient experience based on a cloud data analysis platform can provide hospitals with more timely and accurate doctor-patient experience data, which has become an urgent need to improve the quality of medical services. Summary of the invention

[0005] The purpose of this solution is to provide a real-time monitoring method for doctor-patient experience based on a cloud data analysis platform to solve the problem of being unable to obtain doctor-patient experience data in a timely and accurate manner during the patient's medical treatment process.

[0006] In order to achieve the above objectives, this solution provides a real-time monitoring method for doctor-patient experience based on a cloud data analysis platform, including the following steps: Real-time monitoring of image information during the medical consultation process, and collection of voice information entered by the patient terminal and medical terminal; Recognize and process the facial expressions and behaviors of patients and medical staff in the image information as the first emotional factor, and recognize and process the voices of patients or medical staff in the voice information as the second emotional factor; The emotions of patients or medical staff are quantified into emotion values ​​through the first emotion factor and the second emotion factor, and the emotion values ​​are monitored and managed in real time; When the emotional value of a patient or medical staff exceeds a preset emotional threshold, information about changing the consulting room and emotional comfort is sent to the patient terminal and the medical staff terminal.

[0007] The principle and technical effect of this scheme are: this scheme monitors the image information during the medical treatment process in real time, identifies the expressions and behaviors of patients and medical staff in the image information (patients usually have tense muscles around their lips, close their lips, and touch their faces with their hands to wipe their tears when they are sad and crying; when patients are angry, their eyes are wide open, their hands hit the table or point at medical staff; when patients are anxious, their brows are furrowed, their hands scratch or hold other objects tightly), and then combines the voice (conversation) information of patients and medical staff to more completely collect various information during the medical treatment process, so that the quantified (quantified by the expressions, behaviors and voice conversations of patients and medical staff) emotional values ​​can more accurately reflect the emotions of patients and medical staff. On the basis of the above, this scheme monitors the emotional values ​​of patients and medical staff in real time, can obtain the emotional information of patients and medical staff in real time, and thus obtain the medical experience data during the medical treatment process, and intervene in the medical treatment process in time when the medical experience data is not ideal, so as to provide patients and medical staff with a better medical experience. At the same time, it can also reduce the judgment time of medical experience data in a quantitative way. This solution quantifies the emotions of patients and medical staff into emotion values, through which the emotional changes and current emotional conditions of patients and medical staff can be clearly obtained. Managers can monitor multiple medical processes at the same time through observation data, while managers can only monitor one medical process in the traditional way (relying on observation and monitoring) (it is difficult for managers to watch multiple monitoring screens with one pair of eyes), thereby improving the work efficiency of managers (observing medical processes).

[0008] In summary, this solution solves the problem of patients being unable to obtain timely and accurate doctor-patient experience data during their medical treatment.

[0009] Furthermore, different facial expressions, behaviors, intonations, volumes and rhythms are assigned values, the weight of the first emotional factor is calculated using the numerical values ​​corresponding to the facial expressions and behaviors, the weight of the second emotional factor is calculated using the numerical values ​​corresponding to the intonations, volume and rhythm, and a multi-factor calculation model of the first emotional factor and the second emotional factor is constructed. The multi-factor calculation model is used to calculate the emotional value based on the first emotional factor and the second emotional factor.

[0010] By assigning values ​​to different dimensions such as facial expressions, behaviors, intonations, volume, and rhythm, and calculating the corresponding weights of emotional factors based on these values, the limitation of evaluating emotions based on a single dimension has been changed. For example, judging emotions only by facial expressions may lead to misjudgment due to individual differences or disguised expressions, while combining multiple factors such as behaviors, voice intonation, etc. can capture emotional clues from multiple angles, thereby more accurately quantifying emotions, which are relatively subjective and complex, into specific values, making the measurement of emotions more objective and scientific.

[0011] Furthermore, the facial feature information of patients and medical staff in the image information is identified and annotated, and the shapes of the facial features in the facial feature information are processed as facial emotions; the body contours of patients and medical staff in the image information are identified and annotated, and the position changes of the body contours within a preset behavior judgment time are processed as body emotions; the facial emotions are processed as the first emotional factor, and the body emotions are used to increase the value of the first emotional factor.

[0012] Further, the voice information is processed into dialogue content and sound features, the dialogue content is processed into natural language text, and the text emotions are extracted and identified, and the intonation, volume and rhythm in the sound features are processed into sound emotions; if the text emotions can be extracted and identified when the dialogue content is processed into natural language text, the text emotions are used as the second emotion factor, and the sound emotions are increased by the value of the second emotion factor; if the text emotions cannot be extracted and identified when the dialogue content is processed into natural language text, it is determined whether the intonation is a recognizable language type. If it is a recognizable language type, the volume and rhythm in the sound features are processed into the values ​​of the second emotion factor, and the value of the second emotion factor is increased according to the amplitude of the intonation; if it is an unrecognizable language type, the volume and rhythm in the sound features are processed into the values ​​of the second emotion factor.

[0013] Through the above series of operations such as the relationship between emotion types and numerical settings, image information processing, voice information processing, etc., the emotional state of patients and medical staff in specific situations can be quantified and analyzed more comprehensively, so as to better understand their mental state, and then provide targeted help in medical services, psychological counseling and other aspects.

[0014] Furthermore, when the patient's emotion value exceeds the emotion threshold, a soothing prompt is issued to the patient terminal, and a prompt to soothe the patient is issued to the medical terminal at the same time, the medical process is monitored, and it is determined whether the patient terminal or the medical terminal has recorded the voice information of the medical staff soothing the patient within the preset soothing time, or whether the image information has captured the medical staff's behavior of soothing the patient. If not detected, a prompt is issued to the medical staff again, and the preset soothing time is waited. If the patient terminal still has not recorded the voice information of the medical staff soothing the patient, or the image information has not captured the medical staff's behavior of soothing the patient, the patient's emotion value, the voice information within the two preset soothing times, and the image information are sent to the management terminal; if the patient's emotion value drops within two consecutive preset soothing times, silent soothing is regarded as the medical staff's soothing behavior, and the preset soothing time is waited again.

[0015] On the one hand, this plan recognizes the positive impact of possible atypical comfort methods on patients' emotions, gives certain affirmation to medical staff, and avoids misjudgment of medical staff due to failure to detect conventional comfort behaviors; on the other hand, waiting for the preset comfort time again can further observe the changing trend of patients' emotions and ensure that patients' emotions can continue to improve steadily. If the patient's emotions remain stable or continue to decline, it means that this silent comfort may have indeed worked. If the patient's emotions fluctuate and rise again, medical staff may need to take more specific comfort measures. The above methods can more effectively protect the emotional health of patients during the medical treatment process, prompting medical staff to pay attention to and comfort patients' emotions in a timely manner. At the same time, it also provides a basis for the management side to comprehensively evaluate and improve medical services, which helps to improve the overall quality of medical services and patient satisfaction.

[0016] Furthermore, when the voice information input by the patient terminal is recognized, it is extracted and determined whether the conversation content contains negative descriptions or comments on the medical environment. If so, the conversation content is sent to the management terminal.

[0017] Furthermore, if the patient's emotion value suddenly increases and the medical staff's emotion value does not change, it is determined whether the medical staff's voice recorded by the medical terminal contains a description of the patient's condition. If not, the patient's gaze range in the image information before and after the increase in the patient's emotion value is identified and marked, and it is determined whether there are stains or garbage in the medical environment within the patient's gaze range. If not, a prompt to confirm the medical environment is sent to the medical terminal. If so, the image information with stains or garbage is transmitted to the management terminal.

[0018] If it is determined that there are no stains or garbage within the patient's gaze range, a prompt to confirm the medical environment will be sent to the medical terminal. This helps medical staff to reconfirm whether the patient's emotional changes are related to other factors in the medical environment, such as whether the light is too bright or too dark, the temperature is not suitable, etc., prompting medical staff to more comprehensively investigate environmental factors that may affect the patient's mood, so as to solve the problem in time and stabilize the patient's mood. If stains or garbage are found within the patient's gaze range, the image information of the stains or garbage will be transmitted to the management terminal. After receiving this information, the management terminal can immediately arrange for cleaning staff to clean up, and at the same time, it can supervise and evaluate the personnel responsible for cleaning work in the area to ensure that the medical environment is always kept clean and tidy, avoid affecting the patient's mood due to poor environment, and improve the quality and management level of the overall medical environment.

[0019] Furthermore, the number of soothing times is initialized to 0. When the patient's voice emotion value exceeds the emotion threshold, the patient is regarded as a dangerous patient, the position of the dangerous patient's terminal is collected, and the patient terminals and medical terminals within a preset onlooker radius are regarded as onlooker terminals with the position of the dangerous patient as the center, and it is judged whether the limb outline of the dangerous patient is close to the limb outline of the medical staff; according to the voice characteristics of the dangerous patient, the onlooker terminals that have not recognized the patient's voice are set as ordinary terminals. If the limb outline of the dangerous patient is close to the limb outline of the medical staff, a request for verbal soothing of the dangerous patient is sent to the onlooker terminals that have collected the dangerous patient's voice characteristics. If the patient's limb outline is not close to the limb outline of the medical staff, the management terminal sends a prompt to soothe the patient to the medical terminal and sends a soothing message to the patient terminal; wait A preset comforting time is set, and the number of comforting times is accumulated, and then it is determined whether the number of comforting times exceeds the preset maximum waiting times. If it exceeds the preset maximum waiting times, the management terminal sends a request for medical care intervention to the dangerous patient terminal, and the management terminal sends a safety reminder to the onlooker terminal, generates a safety alarm based on the location information of the dangerous patient terminal, and sends it to the public security maintenance terminal; if it does not exceed the preset maximum waiting times, it is determined whether the dangerous patient's emotional value rises. If the emotional value rises, the management terminal sends a request for medical care intervention to the dangerous patient terminal, and the management terminal sends a safety reminder to the onlooker terminal, generates a safety alarm based on the location information of the dangerous patient terminal, and sends it to the public security maintenance terminal; if the dangerous patient's emotional value does not rise, it is determined whether the patient's emotional value exceeds the emotional threshold.

[0020] This process mainly establishes a relatively complete response mechanism for situations where the patient's voice emotion value exceeds the threshold, aiming to promptly deal with possible dangerous patients' emotional out-of-control situations, ensure the overall safety of medical staff, other patients and the scene, and try to stabilize the patient's emotions through reasonable comfort measures to improve the safety and orderliness of the medical environment. Collecting the location information of dangerous patients can quickly lock in key individuals who may have emotional problems, and at the same time grasp their specific location in the medical environment and their relative position with medical staff, providing basic data for subsequent precise response measures. By sending a verbal comfort request to the onlooker terminal and mobilizing the power of the surrounding people to try to verbally comfort and persuade the dangerous patient, it is possible to alleviate the excessive emotions of the dangerous patient in the first place and prevent their emotions from further deteriorating, thereby ensuring the personal safety of medical staff and the order of the scene. At the same time, the locking of the onlooker terminal can send an evacuation reminder to the onlooker terminal when the dangerous patient's emotions deteriorate.

[0021] Furthermore, the confession value is initialized. When the patient terminal records voice information, the patient and medical staff are distinguished by sound features, and the interval time when the patient and medical staff's voices alternate is counted. When the interval time is less than the preset communication interval time, it is determined whether the features recorded after the voice alternation belong to the medical staff. If they belong to the medical staff, it is determined whether the patient's emotional value rises when the voice alternates. If it rises, the patient's increased emotional value is accumulated into the confession value. When the patient's confession value exceeds the preset confession threshold, a reminder to listen to the patient is issued to the medical staff; after the patient finishes confessing, it is determined whether the difference in the patient's emotional value is less than the confession value. If it is not less than the confession value, a reminder is issued to the patient to pay attention to the length of the consultation and the confession value is cleared; if it is less than the confession value, the difference and the confession value are accumulated as anxiety emotions into the emotional value, and a comforting message, consultation time reminder information and psychological diagnosis and treatment suggestions are sent to the patient.

[0022] This process aims to promote more effective communication between medical staff and patients through dynamic monitoring of the voice interaction between patients and medical staff and the emotional values ​​of patients, pay attention to the emotional changes and psychological needs of patients in a timely manner, ensure the good experience of patients during the treatment process, and intervene appropriately for possible emotional problems. Accurately distinguishing patients and medical staff through voice characteristics provides a basis for subsequent analysis of the communication mode between the two, and helps to deal with problems that arise in communication between the two parties in a targeted manner. When the interval time is less than the preset communication interval time, the interruption of the patient's confession by the medical staff and the patient is judged by the word order of the medical staff and the patient. If the patient has accumulated more negative emotions due to multiple interruptions by the medical staff, the medical staff is prompted to listen to the patient, so that more attention can be paid to the patient's emotions and expression content, and avoid ignoring the patient's emotional needs due to busyness and other reasons. This helps to improve the quality of communication, make patients feel valued, thereby alleviating possible negative emotions of patients and improving the doctor-patient relationship. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Flowchart of the real-time monitoring method of doctor-patient experience based on cloud data analysis platform.

[0024] Figure 2 A flow chart for processing the second emotional factor.

[0025] Figure 3 A flow chart for reassuring patients at risk. DETAILED DESCRIPTION

[0026] The following will clearly and completely describe the concept and technical effects of the present invention in combination with the embodiments, so as to fully understand the purpose, features and effects of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, other embodiments obtained by those skilled in the art without creative work are all within the scope of protection of the present invention: like Figure 1 As shown, the real-time monitoring method of doctor-patient experience based on the cloud data analysis platform includes the following steps: Real-time monitoring of image information during the medical consultation process, and collection of voice information entered by the patient terminal and medical terminal; Recognize and process the facial expressions and behaviors of patients and medical staff in the image information as the first emotional factor, and recognize and process the voices of patients or medical staff in the voice information as the second emotional factor; The emotions of patients or medical staff are quantified into emotion values ​​through the first emotion factor and the second emotion factor, and the emotion values ​​are monitored and managed in real time; When the emotional value of a patient or medical staff exceeds a preset emotional threshold, information about changing the consulting room and emotional comfort is sent to the patient terminal and the medical staff terminal.

[0027] Among them, the image information of patients and medical staff can be obtained through the hospital's surveillance cameras. If the patient and medical staff's permission is required, the voice information can be collected through the patient's or medical staff's terminal (mobile phone, computer or other mobile terminal). If the terminal permission is obtained, the patient's or medical staff's voice information can be obtained through the microphone of the mobile phone, computer or other mobile terminal, and then the voice information is uploaded to the cloud.

[0028] Among them, by assigning values ​​to different dimensions such as facial expressions, behaviors, intonations, volume, and rhythm, and calculating the corresponding weights of emotional factors based on these values, the limitations of evaluating emotions in the past with a single dimension have been changed. For example, judging emotions only by facial expressions may lead to misjudgment due to individual differences or facial expressions disguise, while combining multiple factors such as behavior, voice intonation, etc., can capture emotional clues from multiple angles, thereby more accurately quantifying the relatively subjective and complex state of emotions into specific values, making the measurement of emotions more objective and scientific.

[0029] In one embodiment of the present solution, the preset emotion threshold can be set by an administrator, and the administrator adjusts different emotion thresholds according to the patient's condition.

[0030] In another embodiment of the present solution, the present solution uses an adaptive algorithm to adjust the threshold according to the patient's historical emotional change trajectory, and sets a dynamic threshold based on statistical analysis of historical data or patient group characteristics (such as age, disease type).

[0031] Identify and annotate the facial feature information of patients and medical staff in the image information, and process the facial features in the facial feature information as facial emotions; identify and annotate the body contours of patients and medical staff in the image information, and process the position changes of the body contours within the preset behavior judgment time as body emotions; process facial emotions as the first emotional factor, and use body emotions to increase the value of the first emotional factor.

[0032] Among them, the Bayesian inference model or weighted average method is used to process facial emotions and body emotions as the first emotional factor.

[0033] like Figure 2As shown, the voice information is processed into dialogue content and sound features, the dialogue content is processed by natural language text processing, and the text emotions are extracted and identified, and the intonation, volume and rhythm in the sound features are processed as sound emotions; if the text emotions can be extracted and identified when the dialogue content is processed by natural language text processing, the text emotions are used as the second emotion factor, and the sound emotions are increased by the value of the second emotion factor; if the text emotions cannot be extracted and identified when the dialogue content is processed by natural language text processing, it is determined whether the intonation is a recognizable language type. If it is a recognizable language type, the volume and rhythm in the sound features are processed as the values ​​of the second emotion factor, and the value of the second emotion factor is increased according to the amplitude of the intonation; if it is an unrecognizable language type, the volume and rhythm in the sound features are processed as the values ​​of the second emotion factor.

[0034] In the embodiment of this solution, a special intelligent monitoring system is set up in the consultation room to collect and analyze the image information and voice information during the interaction between the patient and the medical staff. Both the patient terminal and the medical staff terminal are equipped with high-quality microphones, which can accurately record the voice information of both parties during the communication process.

[0035] Emotion types include: anger, disgust, fear, happiness, neutrality, sadness, and surprise. Different values ​​are set for different types of emotions, with anger being 90, disgust being 70, fear being 50, happiness being 10, neutrality being 30, sadness being 50, and surprise being 40. When the patient's emotions are positive (happy), the patient's need for comfort is low. When the patient's emotions are negative (anger, disgust, fear, etc.), the patient's need for comfort is high. Different values ​​are set for the types of emotions according to whether the patient needs comfort. The higher the patient's need for comfort, the higher the value, so that the patient can be comforted more accurately.

[0036] In another embodiment of the present solution, a mathematical model of multidimensional spatial factors is constructed based on facial expressions, behaviors, intonations, volume, and rhythms, and the mathematical model calculates the emotion value from a variety of emotion features through a linear weighted model or a deep learning model.

[0037] First, the facial features of the patient and medical staff in the image information are accurately identified and annotated. When the medical staff was communicating with the patient about the condition, they were caught frowning slightly and looking focused. According to the preset correspondence between facial features and emotions, the frowning facial features were processed as expressions of possible worry or seriousness. On the patient's side, the corners of his mouth were turned down and his eyes were dim, which may be due to the trouble of the condition. They were identified as facial features that may be in a sad mood. During the communication process of a few minutes, the preset behavior judgment time was set to 5 seconds. It was found that the patient's body slightly retracted and his hands were unconsciously clenched. This change in the position of the body contour within 5 seconds was processed as a body emotion, which may mean that the patient had a certain degree of anxiety or resistance. The medical staff leaned forward and placed their hands naturally, showing an open and concerned body emotion. The value corresponding to the facial emotion of the medical staff just identified was 30, which is a neutral emotion (because frowning and other expressions are serious but generally stable), and the patient's facial emotion was identified as a sad emotion, with a corresponding value of 50. Then, use body emotion to increase the value of the first emotion factor. The body emotion of the medical staff is positive attention, which can be assigned a value of -10 (assuming that the body emotion of positive attention has a corresponding bonus value in this setting system), so the final value of the first emotion factor of the medical staff is 30-10=20. The body emotion of the patient is uneasy and resistant, which can be assigned a value of +10 (also with a corresponding minus value in the setting system), so the final value of the first emotion factor of the patient is 50+10=60. During the conversation, the medical staff clearly explained the condition to the patient and expressed comfort and consolation. The patient responded with a statement expressing fear of the condition. The emotional value of the medical staff is set to 10, and the emotional value of the patient is set to 50. The tone of the medical staff's speech is steady, the volume is moderate, and the rhythm is soothing. Comprehensively judging this voice emotion, it can be assigned a value of -5 (assuming that there is such a corresponding assignment method in the setting system). When the patient speaks, the tone of voice is a little trembling, the volume is slightly higher, and the rhythm is slightly faster. Comprehensively judging this voice emotion, it can be assigned a value of +5 (because the trembling, slightly higher volume, and slightly faster rhythm reflect uneasiness).

[0038] For medical staff, because the text emotion can be successfully extracted and recognized, the second emotion factor should be determined by comprehensively considering the text emotion and the voice emotion. The text emotion of the medical staff is 10, and the voice emotion is -5, so the value of the second emotion factor of the medical staff is 10-5=5. For patients, because the text emotion can be clearly extracted and recognized when the conversation content is processed by natural language text (50 corresponding to the emotion of fear), and the voice emotion is +5, the value of the second emotion factor of the patient is 50+5=55.

[0039] Combining the first and second emotional factors, the comprehensive emotional value of the medical staff is the sum of the first emotional factor value of 20 and the second emotional factor value of 5, that is, 20+5=25. From the overall value, the medical staff is in a relatively positive and stable emotional state, and can better deal with the patient's situation and provide professional services. The patient's comprehensive emotional value is the sum of the first emotional factor value of 60 and the second emotional factor value of 55, that is, 60+55=105. This value is relatively high, indicating that the patient is currently in a relatively strong negative emotion, and the medical staff may need to provide more comfort and explanation work to alleviate the patient's fear and anxiety so that he can better cooperate with the treatment.

[0040] When the patient's emotional value exceeds the emotional threshold, a soothing prompt is sent to the patient terminal, and a soothing prompt is sent to the medical terminal at the same time. The medical process is monitored to determine whether the patient terminal or the medical terminal has recorded the medical staff's voice information soothing the patient within the preset soothing time, or whether the image information has captured the medical staff's soothing behavior. If not detected, a prompt is sent to the medical staff again, and the preset soothing time is waited. If the patient terminal still has not recorded the medical staff's voice information soothing the patient, or the image information has not captured the medical staff's soothing behavior, the patient's emotional value, the voice information within the two preset soothing times, and the image information are sent to the management terminal; if the patient's emotional value drops within two consecutive preset soothing times, silent soothing is used as the medical staff's soothing behavior, and the preset soothing time is waited again.

[0041] In the embodiment of this scheme, the patient's worries about the treatment plan made him very anxious, and his emotion value reached 80, exceeding the emotion threshold. The preset comfort time set by the hospital is 15 minutes. During these 15 minutes, the medical treatment process is continuously monitored, including the voice information input of the patient terminal used by the patient and the medical terminal of the medical staff, as well as the image information collection in the consultation room. However, after 15 minutes, no voice information of the medical staff to comfort the patient was detected, and no behavior of the medical staff to comfort the patient (such as patiently talking to the patient, patting the patient's shoulder, etc.) was found in the image information. Therefore, a prompt was issued again to the medical staff (medical terminal) responsible for the patient, emphasizing that the patient is still in an emotionally unstable state and needs to be comforted as soon as possible. Another 15 minutes passed, and the relevant information continued to be checked. The patient terminal still did not record the voice information of the medical staff comforting the patient, and the image information did not collect the medical staff's behavior of comforting the patient. Then, the patient's current emotion value of 80, the two voice information within 15 minutes (30 minutes in total) (including the patient's recorded voice information) and image information (the picture of the patient talking and the picture of the medical staff listening silently) were sent to the management terminal. After receiving the information, the management terminal began to pay attention to this situation and prepared to question the relevant medical staff.

[0042] In another embodiment of the present scheme, the confession value is initialized. When the patient terminal records voice information, the patient and the medical staff are distinguished by the sound characteristics, and the interval time when the patient and the medical staff alternate their voices is counted. When the interval time is less than the preset communication interval time, it is determined whether the characteristics recorded after the voice alternation belong to the medical staff. If it belongs to the medical staff, it is determined whether the patient's emotional value rises when the voice alternates. If it rises, the patient's increased emotional value is accumulated into the confession value. When the patient's confession value exceeds the preset confession threshold, the management terminal sends a reminder to the medical staff to listen to the patient; after the patient finishes confessing, it is determined whether the difference in the patient's emotional value is less than the confession value. If it is not less than the confession value, a reminder is issued to the patient to pay attention to the length of the consultation time, and the confession value is cleared; if it is less than the confession value, the difference and the confession value difference are accumulated as anxiety emotions into the emotional value, and a comforting message, consultation time reminder information and psychological diagnosis and treatment suggestions are sent to the patient.

[0043] In another embodiment of the present scheme, during the consultation, the patient's initial confession value is 0. During a certain period of time, the interval between the medical staff and the patient's conversation is 0.4 seconds, which is less than the preset communication interval (1 second). It is determined that the person who speaks after the interval of 0.4 seconds is the medical staff (i.e., the medical staff interrupts the patient's confession), and the patient stops talking and begins to frown, and the emotion increases by 10. The patient's increased emotion value of 10 is accumulated into the confession value (the confession value becomes 10). In the subsequent communication, the medical staff interrupted the patient's confession 5 times, and the patient's confession value accumulated to 60. The patient's confession value reaches the preset confession threshold (60). The management terminal sends a reminder to the medical staff to listen to the patient. After that, the medical staff no longer interrupts the patient's confession. After the patient finishes confessing, the patient's eyebrows relax and the emotion value drops by 80. The management terminal sends a reminder to the patient to pay attention to the length of the consultation and clears the confession value. After checking the reminder, the patient stops confessing and listens to the medical staff's explanation of the condition.

[0044] In another embodiment of the present solution, the present solution uses a mature human posture estimation algorithm, such as OpenPose, to process the preprocessed image and identify the key joints of the human body (such as joints at the head, shoulders, elbows, wrists, hips, knees, ankles, etc.). Through the coordinate information of these key joints, a human skeleton model can be constructed to describe the position and shape of the limbs. By calculating the slight displacement changes of the head joints in the vertical direction to determine whether there is a nodding action, a reasonable displacement threshold is set (such as the vertical displacement range is between 1-3 cm within a certain period of time), and the frequency of nodding (such as 5-10 times per minute) is combined to determine whether it is a slight nodding action when listening. At the same time, the relative position relationship and changes of the hip, knee and ankle joints are analyzed to determine whether the sitting posture is stable. For example, if the coordinate fluctuation range of these joints is small (within the preset coordinate difference range) within a period of time, it can be identified as a stable sitting posture. The stability of the sitting posture can be quantified as a numerical value. For example, the smaller the coordinate fluctuation, the closer the numerical value is to 1, indicating that the sitting posture is more stable. For each extracted limb feature, construct time series data in chronological order. For example, record the value of the slightly nodding feature (nodding is 1, no nodding is 0) and the quantitative value of the stable sitting feature every second to form a feature sequence that changes over time. Use time series analysis methods, such as sliding window technology combined with statistical analysis, to analyze the constructed time series data. Set a suitable time window length (for example, 5-10 minutes), and count the frequency and duration of limb features in each time window. If "low emotional interference actions" such as slightly nodding and stable sitting continue to appear in this time window and reach a certain frequency threshold (such as the frequency of occurrence exceeds 70%), it is determined that such "low emotional interference actions" exist and are identified as an atypical soothing behavior. During the doctor-patient interaction process, the emotional value is continuously updated at a certain time interval (such as once a minute) to form the time series data of the emotional value, and the change of emotions is monitored in real time. Set a time window (for example, 3-5 minutes) to observe whether the patient's emotional value continues to decrease in this time window. If the emotion value shows a monotonically decreasing trend (i.e., the emotion value at each time interval is lower than the previous one), and the amplitude of the decrease exceeds a certain threshold (e.g., the cumulative decrease in the emotion value exceeds 2 points), it is preliminarily determined that silence may have a soothing effect, and the next step of confirmation combined with physiological signals is entered. At the same time, the patient's physiological signal data is collected, such as obtaining heart rate data through wearable devices (smart bracelets, etc.). Within the time window when the emotion value decreases, observe whether the heart rate also shows a downward trend. A threshold for the decrease in heart rate can be set (e.g., a decrease of 5-10 beats per minute). When the emotion value decreases and the heart rate also decreases and meets the corresponding threshold requirements, it is confirmed that silence has played a role in soothing the patient's emotions during this time period, and it is identified as an effective atypical soothing behavior.

[0045] When the voice information input by the patient terminal is recognized, it is extracted to determine whether the conversation content contains negative descriptions or comments on the medical environment. If so, the conversation content is sent to the management terminal.

[0046] If the patient's emotion value suddenly increases and the medical staff's emotion value does not change, it is determined whether the medical staff's voice recorded by the medical terminal contains a description of the patient's condition. If not, the patient's gaze range in the image information before and after the increase in the patient's emotion value is identified and marked, and it is determined whether there is stain or garbage in the medical environment within the patient's gaze range. If not, a prompt to confirm the medical environment is sent to the medical terminal. If so, the image information with stains or garbage is transmitted to the management terminal.

[0047] In the embodiment of this solution, during the communication process between the patient and the medical staff, the patient enters voice information through the patient terminal (such as a mobile phone, which is used to interact with the hospital and record some medical information). When the patient's voice information is recognized, it is extracted and judged that the conversation content entered by the patient contains complaints about the crowded environment, so the conversation content is immediately sent to the management terminal.

[0048] In the embodiment of the present scheme, the patient's emotion value suddenly increases during the medical treatment process. It is determined through monitoring that the emotion value of the medical staff has not changed at this time. Then, it is determined that the voice of the medical staff recorded by the medical terminal does not contain a description of the patient's condition. Therefore, the patient's gaze range in the image information before and after the increase of the patient's emotion value is further identified and marked. Through the analysis of the camera shooting pictures in the clinic, it is found that the patient is mainly looking at the floor, corners and the position of the trash can next to the clinic. Then, it is determined that there are stains or garbage in the medical environment within the patient's gaze range (the trash can is also overflowing, and some garbage is scattered around), and the image information of the presence of stains or garbage is transmitted to the management terminal. After receiving it, the management terminal immediately notifies the cleaning staff to go to the clinic for cleaning, and also supervises and reminds the personnel responsible for cleaning in the area, requiring them to strengthen daily cleaning work to ensure the cleanliness of the medical environment.

[0049] In another embodiment of the present solution, environmental factors such as light, noise, and temperature are introduced into the multi-factor analysis to establish priorities for environmental issues and to identify which environmental factors require urgent attention. For example, "stains" are classified as high-priority issues, while "slight noise" are classified as medium- and low-priority issues.

[0050] Light sensors are installed in places where doctors and patients interact (such as wards and consulting rooms) to measure the intensity of ambient light and record it quantitatively in Lux. Noise sensors are deployed to detect the decibel value of ambient noise and distinguish noise components in different frequency ranges, so as to more accurately assess the impact of noise on doctors and patients. Temperature sensors are installed to obtain ambient temperature data in real time and record it in degrees Celsius (℃). For visual environmental problems such as "stains", image recognition technology can be used in combination with camera monitoring to detect them. The camera regularly captures images in the venue and uses a pre-trained image recognition model to identify whether there are stains and the area and location of the stains. Light factors are quantified according to different light intensity ranges. For example, too dark light (less than 100 Lux) may affect the patient's mood and the accuracy of medical staff's operations, which can be quantified as -1; moderate light (100-500 Lux) is quantified as 0; too strong light (more than 500 Lux) may cause visual discomfort, which is quantified as 1. Quantification is performed according to the decibel value of noise. For example, noise below 40 decibels can be considered a quiet environment and quantified as 0; 40-60 decibels is slight noise and quantified as 1; 60-80 decibels is moderate noise and quantified as 2; and above 80 decibels is strong noise and quantified as 3. At the same time, the quantization value is further adjusted according to the noise frequency component. If the noise frequency is concentrated in the human sensitive frequency band (such as 2-4kHz), the weight of the quantization value is appropriately increased. Determine the appropriate temperature range (such as 22-26℃), within which the temperature factor is quantified as 0. If the temperature is too low (below 22℃) or too high (above 26℃), it is quantified according to the degree of deviation. For example, for every 1℃ deviation from the appropriate temperature, the quantization value increases or decreases by 0.5. Quantify according to the area and location of the stain. If the stain area is large (exceeding a certain threshold, such as 1 square meter) and located in a conspicuous location (such as near a bed or around diagnostic and treatment equipment), it is quantified as 3, indicating a high priority; if the stain area is small (less than 1 square meter) and the location is relatively inconspicuous, it is quantified as 2, indicating a medium priority.

[0051] like Figure 3As shown, the initial number of soothing times is 0. When the patient's voice emotion value exceeds the emotion threshold, the patient is regarded as a dangerous patient, the position of the dangerous patient's terminal is collected, and the patient terminals and medical terminals within the preset onlooker radius are regarded as onlooker terminals with the position of the dangerous patient as the center, and it is judged whether the limb outline of the dangerous patient is close to the limb outline of the medical staff; according to the voice characteristics of the dangerous patient, the onlooker terminals that have not recognized the patient's voice are set as ordinary terminals. If the limb outline of the dangerous patient is close to the limb outline of the medical staff, a request for verbal soothing of the dangerous patient is sent to the onlooker terminals that have collected the voice characteristics of the dangerous patient. If the limb outline of the patient is not close to the limb outline of the medical staff, the management terminal sends a prompt to soothe the patient to the medical terminal and sends a soothing message to the patient terminal; wait for the preset Set the comfort time, accumulate the number of comfort times, and then determine whether the number of comfort times exceeds the preset maximum waiting times. If it exceeds the preset maximum waiting times, the management terminal sends a request for medical care intervention to the dangerous patient terminal, and the management terminal sends a safety reminder to the onlooker terminal, generates a safety alarm based on the location information of the dangerous patient terminal, and sends it to the public security maintenance terminal; if it does not exceed the preset maximum waiting times, determine whether the dangerous patient's emotional value rises. If the emotional value rises, the management terminal sends a request for medical care intervention to the dangerous patient terminal, and the management terminal sends a safety reminder to the onlooker terminal, generates a safety alarm based on the location information of the dangerous patient terminal, and sends it to the public security maintenance terminal; if the dangerous patient's emotional value does not rise, determine whether the patient's emotional value exceeds the emotional threshold.

[0052] In an embodiment of the present scheme, the initialization number of soothing times is 0, the preset emotion threshold is set to 80 (assuming that the full score of the sound emotion value is 100), the preset onlooker radius is set to 5 meters, the preset soothing time is 10 minutes, and the preset maximum waiting times is 3 times.

[0053] The patient was in severe abdominal pain. After waiting for a while, he was extremely anxious when communicating with the medical staff. The emotional value of his voice reached 85, exceeding the preset emotional threshold. At this time, the patient (assuming that the patient is "Mr. Zhang") is immediately identified as a dangerous patient, and further monitoring actions are started. Through the camera in the waiting area, it is quickly determined whether the limb outline of the dangerous patient is close to the limb outline of the medical staff (for example, the finger is pointing to the medical staff), and the location information of the patient terminal (mobile phone, whose positioning function is connected to the hospital) carried by the dangerous patient is obtained. With the location of the dangerous patient as the center, other patient terminals and medical terminals of medical staff within a radius of 5 meters are regarded as onlooker terminals. In this process, according to the sound characteristics (timbre, tone, etc.) such as the sound propagation range of the dangerous patient, some onlooker terminals (mainly some patient terminals sitting in the corner) that are slightly far away and blocked by the crowd and do not recognize the voice of the dangerous patient are set as ordinary terminals, while other terminals that can clearly hear Mr. Zhang's voice are still normal onlooker terminals.

[0054] After a while, the dangerous patient became more and more agitated and began to approach a passing nurse, with the outline of his limbs gradually approaching the nurse's limbs. At this time, the situation was detected, and a request for verbal comfort of the dangerous patient was quickly sent to all the onlookers, prompting the surrounding patients and medical staff to help say some comforting words to stabilize the dangerous patient's emotions, and began to wait for the preset comfort time of 10 minutes, while accumulating the number of comforts, and the number of comforts became 1 at this time. However, during these 10 minutes, the pain of the dangerous patient was not relieved, and his mood became worse, and the voice emotion value rose to 90. After the management terminal detected this situation, it immediately sent a request for medical service intervention to the patient terminal of the dangerous patient, reminding him that medical staff would come to treat his condition soon. At the same time, the management terminal sent a safety reminder to all the onlookers, telling everyone to keep a distance and pay attention to their own safety. In addition, a safety alarm was generated based on the location information of the patient terminal of the dangerous patient and sent to the hospital's security maintenance terminal (the receiving device of the hospital security personnel). After the security maintenance terminal received the alarm, the security personnel quickly rushed to the scene to maintain order and prevent possible accidents.

[0055] In another embodiment of the present scheme, voice emotion factors, body proximity, and facial expression changes are comprehensively judged, and a danger level classification mechanism is added to the result of the comprehensive judgment. For example, dangerous patients are divided into mild, moderate, and severe categories based on emotional values ​​and behavioral characteristics, and different comfort or safety measures are taken respectively.

[0056] Among them, the speech emotion features, body proximity features and facial expression features are fused to construct a comprehensive feature vector. For example, the MFCC feature vector of speech, feature values ​​such as volume and speech speed, the distance-related feature values ​​of body proximity, and the category code and intensity value of facial expression can be spliced ​​into a high-dimensional vector in a certain order. Collect a large number of multimodal data samples labeled with danger levels (mild, moderate, and severe). These samples contain the speech, body proximity, and facial expression data of different patients in different situations and the corresponding danger level annotations. Use machine learning or deep learning algorithms to build a comprehensive judgment model, such as support vector machine (SVM), multi-layer perceptron (MLP), or deep learning-based fusion network. Train the model with the comprehensive feature vector as the model input and the danger level as the output label. By adjusting the parameters of the model, the model can learn the complex relationship between different modal features and danger levels, thereby improving the accuracy and generalization ability of the model.

[0057] In one embodiment of the present solution, the confession value is initialized. When the patient terminal records voice information, the patient and the medical staff are distinguished by the sound features, and the interval time when the patient and the medical staff alternate their voices is counted. When the interval time is less than the preset communication interval time, it is determined whether the features recorded after the voice alternation belong to the medical staff. If it belongs to the medical staff, it is determined whether the patient's emotional value rises when the voice alternates. If it rises, the patient's increased emotional value is accumulated into the confession value. When the patient's confession value exceeds the preset confession threshold, a reminder to listen to the patient is sent to the medical staff. After the patient finishes talking, it is determined whether the difference in the patient's emotional value is less than the talking value. If it is not less than the talking value, the patient will be reminded to pay attention to the length of the consultation and the talking value will be cleared. If it is less than the talking value, the difference between the difference and the talking value will be accumulated as anxiety into the emotional value, and the patient will be sent a comforting message, a consultation time reminder, and psychological treatment suggestions.

[0058] In one embodiment of the present solution, during the patient's confession, the confession time is monitored in real time and compared with the set time limit. When approaching or exceeding the time limit, different processing is performed according to the previous feedback information of the medical staff. If the medical staff marks that the patient's confession content is critical, the medical staff is reminded whether to continue to extend the time; if the medical staff marks that the patient's emotions are excessively vented, a tactful reminder message is sent to the patient, such as "The medical staff also needs to know other important information, you can briefly summarize the key points", to guide the patient to speed up the pace of confession.

[0059] In one embodiment of the present solution, the patient terminal or medical terminal encrypts the data before transmitting it, and introduces a differential privacy mechanism during encryption to ensure that the patient's personal information cannot be identified during data analysis.

[0060] The above is only an embodiment of the present invention, and the common knowledge such as the known specific structure and characteristics in the scheme is not described in detail here. It should be pointed out that for those skilled in the art, several deformations and improvements can be made without departing from the structure of the present invention, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A real-time monitoring method for doctor-patient experience based on a cloud data analysis platform, characterized in that: The following steps are involved: Real-time monitoring of image information during the medical consultation process, and collection of voice information entered by the patient terminal and medical terminal; Recognize and process the facial expressions and behaviors of patients and medical staff in the image information as the first emotional factor, and recognize and process the voices of patients or medical staff in the voice information as the second emotional factor; The emotions of patients or medical staff are quantified into emotion values ​​through the first emotion factor and the second emotion factor, and the emotion values ​​are monitored and managed in real time; When the emotional value of a patient or medical staff exceeds a preset emotional threshold, information about changing the consulting room and emotional comfort is sent to the patient terminal and the medical staff terminal.

2. The real-time monitoring method of doctor-patient experience based on a cloud data analysis platform according to claim 1 is characterized in that: Assign values ​​to different facial expressions, behaviors, intonations, volumes and rhythms, calculate the weight of the first emotional factor through the numerical values ​​corresponding to the facial expressions and behaviors, calculate the weight of the second emotional factor through the numerical values ​​corresponding to the intonations, volume and rhythm, and construct a multi-factor calculation model for the first emotional factor and the second emotional factor. The multi-factor calculation model is used to calculate the emotional value based on the first emotional factor and the second emotional factor.

3. The method for real-time monitoring of doctor-patient experience based on a cloud data analysis platform according to claim 2, characterized in that: Identify and annotate the facial feature information of patients and medical staff in the image information, and process the facial features in the facial feature information as facial emotions; identify and annotate the body contours of patients and medical staff in the image information, and process the position changes of the body contours within the preset behavior judgment time as body emotions; process facial emotions as the first emotional factor, and use body emotions to increase the value of the first emotional factor.

4. The method for real-time monitoring of doctor-patient experience based on a cloud data analysis platform according to claim 3 is characterized in that: Process the speech information into dialogue content and sound features, perform natural language text processing on the dialogue content, extract and identify text emotions, and process the intonation, volume and rhythm in the sound features into sound emotions; If the text emotion can be extracted and recognized during natural language text processing of the conversation content, the text emotion is used as the second emotion factor, and the voice emotion is increased by the value of the second emotion factor; If the text emotion cannot be extracted and recognized during natural language text processing of the conversation content, it is determined whether the intonation is a recognizable language type. If it is a recognizable language type, the volume and rhythm in the sound features are processed as the numerical value of the second emotion factor, and the numerical value of the second emotion factor is increased according to the amplitude of the intonation; if it is an unrecognizable language type, the volume and rhythm in the sound features are processed as the numerical value of the second emotion factor.

5. The real-time monitoring method of doctor-patient experience based on a cloud data analysis platform according to claim 4 is characterized in that: When the patient's emotional value exceeds the emotional threshold, a soothing reminder is sent to the patient terminal, and a soothing reminder is sent to the medical terminal at the same time, the medical process is monitored, and it is determined whether the patient terminal or the medical terminal has recorded the voice information of the medical staff soothing the patient within the preset soothing time, or whether the image information has collected the behavior of the medical staff soothing the patient. If not detected, a reminder is sent to the medical staff again, and the preset soothing time is waited. If the patient terminal still has not recorded the voice information of the medical staff soothing the patient, or the image information has not collected the behavior of the medical staff soothing the patient, the patient's emotional value, the voice information within the two preset soothing times, and the image information are sent to the management terminal; If the patient's emotional value decreases during the preset soothing time for two consecutive times, silent soothing will be used as the medical staff's soothing behavior, and the preset soothing time will be waited again.

6. The method for real-time monitoring of doctor-patient experience based on a cloud data analysis platform according to claim 5, characterized in that: When the voice information input by the patient terminal is recognized, it is extracted and determined whether the conversation content contains negative descriptions or comments on the medical environment. If so, the conversation content is sent to the management terminal.

7. The real-time monitoring method of doctor-patient experience based on a cloud data analysis platform according to claim 6 is characterized in that: If the patient's emotion value suddenly increases and the medical staff's emotion value does not change, it is determined whether the medical staff's voice recorded by the medical terminal contains a description of the patient's condition. If not, the patient's gaze range in the image information before and after the increase in the patient's emotion value is identified and marked, and it is determined whether there is stain or garbage in the medical environment within the patient's gaze range. If not, a prompt to confirm the medical environment is sent to the medical terminal. If so, the image information with stains or garbage is transmitted to the management terminal.

8. The method for real-time monitoring of doctor-patient experience based on a cloud data analysis platform according to claim 7, characterized in that: The number of initial soothing times is 0. When the patient's voice emotion value exceeds the emotion threshold, the patient is regarded as a dangerous patient, the position of the dangerous patient's terminal is collected, and the patient terminals and medical terminals within the preset onlooker radius are regarded as onlooker terminals with the dangerous patient's position as the center, and it is judged whether the limb outline of the dangerous patient is close to the limb outline of the medical staff; according to the voice characteristics of the dangerous patient, the onlooker terminals that have not recognized the patient's voice are set as ordinary terminals. If the limb outline of the dangerous patient is close to the limb outline of the medical staff, a request for verbal soothing of the dangerous patient is sent to the onlooker terminal that has collected the dangerous patient's voice characteristics. If the patient's limb outline is not close to the limb outline of the medical staff, the management terminal sends a prompt to soothe the patient to the medical terminal and sends a soothing message to the patient terminal; wait for the preset Soothing time, cumulative soothing times, and then judging whether the soothing times exceed the preset maximum waiting times. If so, the management terminal sends a request for medical care intervention to the dangerous patient terminal, and the management terminal sends a safety reminder to the onlooker terminal, generates a safety alarm based on the location information of the dangerous patient terminal, and sends it to the public security maintenance terminal; if it does not exceed the preset maximum waiting times, then judging whether the dangerous patient's emotion value rises. If so, the management terminal sends a request for medical care intervention to the dangerous patient terminal, and the management terminal sends a safety reminder to the onlooker terminal, generates a safety alarm based on the location information of the dangerous patient terminal, and sends it to the public security maintenance terminal; if the dangerous patient's emotion value does not rise, then judging whether the patient's emotion value exceeds the emotion threshold.

9. The real-time monitoring method of doctor-patient experience based on a cloud data analysis platform according to claim 8 is characterized in that: Initialize the confession value. When the patient terminal records voice information, distinguish the patient and the medical staff through the sound characteristics, and count the interval time when the patient and the medical staff alternate their voices. When the interval time is less than the preset communication interval time, determine whether the recorded features after the voice alternation belong to the medical staff. If it belongs to the medical staff, determine whether the patient's emotional value rises when the voice alternates. If it rises, the patient's rising emotional value is accumulated into the confession value. When the patient's confession value exceeds the preset confession threshold, send a reminder to the medical staff to listen to the patient; After the patient finishes talking, it is determined whether the difference in the patient's emotional value is less than the talking value. If it is not less than the talking value, the patient is prompted to pay attention to the length of the consultation and the talking value is cleared; If it is less than the confession value, the difference between the difference and the confession value will be accumulated into the emotion value as anxiety, and the patient will be sent a comforting message, a reminder of the consultation time, and suggestions for psychological diagnosis and treatment.

Images