Doctor-patient communication method based on head-mounted display and head-mounted display

Abstract

The invention discloses a doctor-patient communication method based on a head-mounted display and the head-mounted display, and the method comprises the steps: obtaining the biological characteristics of a patient and the gestures of a doctor based on the model data collected by the head-mounted display; a point position mark corresponding to the operation point position shown by the gesture of the doctor is added on a human body three-dimensional model constructed by the head-mounted display based on the biological characteristics of the patient so as to record the operation steps of the doctor; obtaining language information of a doctor-patient communication process based on the language data collected by the head-mounted display; displaying a doctor-patient dialogue text while displaying the human body three-dimensional model; determining emotional characteristics of the patient based on the patient biological characteristics and the patient language information; performing emotion prediction on the emotion features by using an emotion model to obtain the emotion of the patient; and displaying an emotion identifier corresponding to the emotion of the patient. According to the method, point position marking on the human body three-dimensional model and real-time tracking of the emotion of the patient can be achieved, effective visual help is provided for treatment means of doctors, doctor-patient communication is more convenient and simpler, and energy of the doctors in the treatment process is reduced.

Description

Technical Field

[0001] The present application relates to the field of oral medicine, and in particular to a doctor-patient communication method based on a head-mounted display and a head-mounted display. Background Art

[0002] Traditional doctor-patient communication is usually face-to-face dialogue. However, when patients undergo oral medical diagnosis, they find it difficult to speak normally due to gum pain, which leads to problems in doctor-patient communication such as language barriers (patients have slurred speech), information asymmetry (patients find it difficult to express the degree of pain due to toothache), time pressure (there are many patients in the hospital and each patient has limited consultation time), and emotional factors (patients have greater mood swings due to toothache). These problems lead to inefficient doctor-patient communication, affect the doctor's diagnosis of the patient's condition, and make it impossible to provide a good treatment plan. In addition, traditional doctor-patient communication lacks full-time recording methods, which brings many inconveniences to tracing back the condition and formulating treatment plans.

[0003] With the advancement of Internet technology, online consultation methods have emerged in doctor-patient communication. However, the patient data collected by this online consultation method comes from a single source, and only relies on video communication to obtain patient expressions. The recognition results cannot be corrected or supplemented by other data, and are only flat data. The recognition results are not stable. In addition, patients are required to provide the corresponding dedicated electronic equipment themselves. The dedicated electronic equipment provided by patients themselves may have unstable recognition results due to differences in device models and component parameters. In addition, the scope of use of this diagnosis is small, and it is only suitable for online consultation scenarios. When there is a delay problem in the network, the patient's expression changes too quickly, making it more difficult for doctors to diagnose.

[0004] Obviously, face-to-face communication and online consultations cannot substantially solve communication problems caused by language barriers, information asymmetry, time pressure, and emotional factors. They still rely more on the doctor's skills, experience, and regulations. Doctors need to patiently guide patients to communicate, which consumes a lot of doctors' energy during the treatment process. Summary of the invention

[0005] The present application provides a method and a head-mounted display (HMD)-based doctor-patient communication method, with the aim of providing a convenient way for doctor-patient communication to reduce the doctor's energy during the treatment process.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] A method for doctor-patient communication based on head display, comprising:

[0008] Based on the model data collected in real time by the head display, the patient's biometric characteristics and the doctor's gestures in the doctor-patient communication process are obtained; the wearers of the head display include at least the doctor;

[0009] Adding point marks corresponding to the operation points indicated by the doctor's gestures on the three-dimensional human body model constructed by the head display based on the patient's biometrics to record the doctor's operation steps during the doctor-patient communication process;

[0010] Based on the language data collected in real time by the head display, language information in the doctor-patient communication process is obtained; the language information includes doctor language information and patient language information;

[0011] While displaying the three-dimensional human body model, displaying a doctor-patient dialogue text; the doctor-patient dialogue text is obtained by performing language recognition on the language information;

[0012] Determining the patient's emotional characteristics based on the patient's biological characteristics and the patient's language information;

[0013] Using a pre-trained emotion model, emotion prediction is performed on the emotion characteristics of the patient to obtain the patient's emotion;

[0014] The emotion identifier corresponding to the patient's emotion is displayed.

[0015] Optionally, the method further includes:

[0016] Using a pre-trained pain level model, predicting the pain level of the patient's emotional characteristics to obtain the patient's pain level;

[0017] When the patient's pain level reaches a specified level, a corresponding prompt message is sent to the doctor.

[0018] Optionally, based on the model data collected in real time by the headset, obtain the patient's biometrics and the doctor's gestures during the doctor-patient communication process, including:

[0019] Obtain model data collected in real time by the head display;

[0020] Preprocessing the model data to obtain a target image that meets usage requirements;

[0021] Performing human body key point detection on the target image to obtain human body posture feature points and facial feature points of the patient;

[0022] Based on the human body posture feature points and the facial feature points, identifying the patient's biometric features during the doctor-patient communication process;

[0023] Perform gesture recognition on the target image to obtain the doctor's gesture during the doctor-patient communication process.

[0024] Optionally, adding point marks corresponding to the operation points indicated by the doctor's gestures to the three-dimensional human body model constructed by the head display based on the patient's biometrics to record the doctor's operation steps during the doctor-patient communication process includes:

[0025] Based on the human body posture feature points and the facial feature points, construct a three-dimensional human body model of the patient;

[0026] Rendering the three-dimensional human body model and the doctor's gesture to generate an interactive screen of the three-dimensional human body model and the doctor's gesture;

[0027] Based on the contact position between the three-dimensional human body model and the doctor's gesture in the interactive screen, determining the operation point of the doctor's gesture;

[0028] Point marks corresponding to the operation points are added to the three-dimensional human body model to record the doctor's operation steps during the doctor-patient communication process.

[0029] Optionally, based on the language data collected in real time by the head display, language information in the doctor-patient communication process is obtained, including:

[0030] Obtaining language data collected in real time by the head display;

[0031] Preprocessing the language data to obtain language information that meets language quality requirements;

[0032] Using a preset digital human model, the language information is subjected to digital human identity recognition to obtain the doctor's language information and the patient's language information; the digital human model is modeled based on the doctor's language features;

[0033] Based on the doctor's language information and the patient's language information, the language information in the doctor-patient communication process is identified.

[0034] Optionally, determining the patient's emotional characteristics based on the patient's biological characteristics and the patient's language information includes:

[0035] Determine the patient's biometric features and patient language information at the same time stamp; the patient's biometric features include the patient's body posture feature points and facial feature points;

[0036] Extracting feature points from the patient's language information to obtain language feature points of the patient;

[0037] The human body posture feature points, the facial feature points and the language feature points are subjected to feature fusion to obtain the emotional features of the patient.

[0038] Optionally, the training process of the emotion model and the pain level model includes:

[0039] Pre-building a multi-task learning model; the multi-tasks include an emotion prediction task and a pain level prediction task;

[0040] Obtaining training data for the multi-task learning model; the training data includes emotion feature samples preset with emotion labels and pain level labels; the emotion feature samples are determined based on human posture feature points, facial feature points, and language feature points of sample patients in previous doctor-patient communication processes;

[0041] Based on the training data, determining a corresponding training set and a validation set;

[0042] Using the training set, training the multi-task learning model to obtain a trained multi-task learning model;

[0043] The trained multi-task learning model is verified using the verification set to obtain the emotion model and the pain level model.

[0044] Optionally, the method further includes:

[0045] Generate a doctor-patient communication record with a corresponding timestamp based on the patient's biometrics, operation points, doctor's language information, patient's language information, patient's emotions, and patient's pain level with the same timestamp;

[0046] The doctor-patient communication records with different time stamps during the doctor-patient communication process are saved in the patient's medical records; the medical records are used to assist the doctor in retrospective diagnosis and treatment.

[0047] Optionally, while displaying the three-dimensional human body model, a text of the doctor-patient dialogue is displayed, including:

[0048] While displaying the three-dimensional human body model, translating the language information to obtain target information that can be understood by the doctor;

[0049] Generate a doctor-patient dialogue text corresponding to the target information, and display the doctor-patient dialogue text.

[0050] A head display, comprising:

[0051] A data analysis unit, configured to obtain biometric features of the patient and the doctor's gestures during the doctor-patient communication process based on the model data collected in real time by the head display; the wearers of the head display include at least the doctor;

[0052] A step recording unit, configured to add point marks corresponding to the operation points indicated by the doctor's gestures on the three-dimensional human body model constructed by the head display based on the patient's biometrics, so as to record the doctor's operation steps during the doctor-patient communication process;

[0053] A language analysis unit, configured to obtain language information in the doctor-patient communication process based on language data collected in real time by the head display; the language information includes doctor language information and patient language information;

[0054] An information display unit, used to display the doctor-patient dialogue text while displaying the three-dimensional human body model; the doctor-patient dialogue text is obtained by performing language recognition on the language information;

[0055] a feature determination unit, configured to determine an emotional feature of the patient based on the biological feature of the patient and the language information of the patient;

[0056] An emotion recognition unit, used to use a pre-trained emotion model to perform emotion prediction on the patient's emotion characteristics to obtain the patient's emotion;

[0057] The information display unit is also used to display the emotion identifier corresponding to the patient's emotion.

[0058] The technical solution provided by the present application obtains the patient's biometrics and the doctor's gestures based on the model data collected by the head display. On the three-dimensional human body model constructed by the head display based on the patient's biometrics, point marks corresponding to the operation points indicated by the doctor's gestures are added to record the doctor's operation steps. Based on the language data collected by the head display, the language information of the doctor-patient communication process is obtained. While displaying the three-dimensional human body model, the doctor-patient dialogue text is displayed. Based on the patient's biometrics and the patient's language information, the patient's emotional characteristics are determined. Using the emotion model, the patient's emotional characteristics are predicted to obtain the patient's emotions. The emotion identifier corresponding to the patient's emotions is displayed. The present application can realize the point marking on the three-dimensional human body model and the real-time tracking of the patient's emotions, provide effective and intuitive help for the doctor's treatment methods, make doctor-patient communication more convenient and simple, and reduce the doctor's energy during the treatment process. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 A schematic diagram of a flow chart of a method for doctor-patient communication based on a head-mounted display provided in an embodiment of the present application;

[0061] Figure 2 A schematic diagram of a flow chart of another method for doctor-patient communication based on a head-mounted display provided in an embodiment of the present application;

[0062] Figure 3 A schematic diagram of a flow chart of another method for doctor-patient communication based on a head-mounted display provided in an embodiment of the present application;

[0063] Figure 4 A schematic diagram of a flow chart of another method for doctor-patient communication based on a head-mounted display provided in an embodiment of the present application;

[0064] Figure 5 A schematic diagram of a flow chart of another method for doctor-patient communication based on a head-mounted display provided in an embodiment of the present application;

[0065] Figure 6 A schematic diagram of a flow chart of another method for doctor-patient communication based on a head-mounted display provided in an embodiment of the present application;

[0066] Figure 7 A schematic diagram of the architecture of a head-mounted display provided in an embodiment of the present application;

[0067] Figure 8 A schematic diagram of a method for doctor-patient communication based on a head-mounted display provided in an embodiment of the present application;

[0068] Fig. 9 A schematic diagram of another method for doctor-patient communication based on a head-mounted display provided in an embodiment of the present application;

[0069] Fig.10 A schematic diagram of a model training provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0071] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0072] like Figure 1 As shown, it is a flowchart of a method for doctor-patient communication based on a head display provided in an embodiment of the present application, including the following steps.

[0073] S101: Based on the model data collected in real time by the head display, the patient's biometric characteristics and the doctor's gestures during the doctor-patient communication process are obtained.

[0074] Among them, the wearers of the headset include at least doctors.

[0075] In some examples, the types of head displays include, but are not limited to, head-mounted displays (such as virtual reality devices, augmented reality devices, mixed reality devices) and smart glasses. Generally speaking, the head display and its operating system provide the basic hardware and computing resources required for three-dimensional modeling, and the basic hardware includes, but is not limited to, cameras (such as binocular cameras), microphones (such as microphone arrays), and the like.

[0076] In some examples, the types of model data include but are not limited to point cloud data, video data, image data, iToF data, etc. Specifically, point cloud data can be collected based on the lidar on the headset, and image data and video data (such as video image frames) can be collected based on the camera on the headset.

[0077] In a possible implementation, during the doctor-patient communication process, the doctor needs to wear the headset in real time so that the camera of the headset can capture the images seen by the doctor in the current doctor-patient communication scenario in real time. The camera generates a corresponding video stream based on the real-time captured images, and the headset then extracts corresponding video image frames from the video stream according to a preset sampling interval.

[0078] Optionally, based on the model data collected in real time by the head display, the patient's biometrics and the doctor's gestures during the doctor-patient communication process can be obtained. Figure 2 The steps are shown and the corresponding explanations.

[0079] S102: Add point marks corresponding to the operation points indicated by the doctor's gestures to the three-dimensional human body model constructed by the head display based on the patient's biometrics, so as to record the doctor's operation steps during the doctor-patient communication process.

[0080] Among them, the head display includes one or more mixed virtual visual technologies such as augmented reality, virtual reality, and mixed reality, which can be used to establish a three-dimensional human body model of the patient based on the patient's biological characteristics.

[0081] It should be noted that the operation points indicated by the doctor's gestures can be regarded as the doctor's diagnostic methods for the patient, such as acupuncture, pressing, touching and other methods. For this reason, it is necessary to identify the operation points of the diagnostic methods on the patient in order to record the doctor's operating steps during the doctor-patient communication process, so as to facilitate the doctor to know the diagnostic methods used at the time during subsequent retrospective diagnosis and treatment.

[0082] Optionally, add point marks corresponding to the operation points indicated by the doctor's gestures on the human body 3D model constructed by the head display based on the patient's biometrics to record the implementation process of the doctor's operation steps during the doctor-patient communication process. Figure 3 The steps are shown and the corresponding explanations.

[0083] In a possible embodiment, in combination with Figure 3 The process shown in the figure, which is the process of establishing a three-dimensional human body model and effectively recording the doctor's operating steps during the doctor-patient communication process, can be found in Figure 8 shown.

[0084] S103: Based on the language data collected in real time by the head display, language information in the doctor-patient communication process is obtained.

[0085] The language information includes doctor language information and patient language information.

[0086] It is understandable that there will inevitably be dialogue between doctors and patients during the medical communication process. For this reason, the microphone of the headset can be used to collect the language data of the dialogue in real time, and the specific content of the dialogue can be recorded throughout the process to facilitate doctors to conduct retrospective diagnosis and treatment.

[0087] In some examples, the types of language data include, but are not limited to: spoken data, sign language data, and voice or text information output by other devices, etc. Therefore, the types of language information obtained in the doctor-patient communication process may also include spoken language, sign language, etc. Specifically, spoken data can be collected based on the microphone on the head display, sign language data can be collected based on the camera on the head display, and other types of language data can be obtained through data interaction between the head display and other devices.

[0088] Optionally, the process of obtaining language information during doctor-patient communication based on the language data collected in real time by the head display can be found in Figure 4 The steps are shown and the corresponding explanations.

[0089] S104: While displaying the three-dimensional model of the human body, display the text of the doctor-patient dialogue.

[0090] Among them, the doctor-patient dialogue text is obtained by language recognition of language information.

[0091] In some examples, the doctor-patient conversation text includes doctor text and patient text. After obtaining the doctor language information DV and the patient language information PV, language recognition (such as speech recognition, sign language recognition, etc.) can be performed on the doctor language information DV to obtain the doctor text, and language recognition can be performed on the patient language information PV to obtain the patient text. Corresponding identity tags and display effects are set for the doctor text and patient text, respectively, so that the doctor text and patient text can be displayed distinguishably.

[0092] It should be noted that displaying the text of the doctor-patient dialogue while displaying the three-dimensional model of the human body can help doctors clearly understand the patient's verbal expression and avoid the doctor being unclear about the patient's intention in the verbal expression.

[0093] In addition, the types of language information may include spoken language, sign language, etc., and the patient's language is not understood by the doctor (for example, the patient is a foreigner, etc.). Therefore, the language information needs to be processed to obtain language that can be understood by the doctor.

[0094] Optionally, the implementation process of displaying the doctor-patient dialogue text while displaying the three-dimensional human body model can be: while displaying the three-dimensional human body model, translating the language information to obtain target information that can be understood by the doctor, generating the doctor-patient dialogue text corresponding to the target information, and displaying the doctor-patient dialogue text.

[0095] S105: Determine the patient's emotional characteristics based on the patient's biological characteristics and the patient's language information.

[0096] Among them, the patient's biological characteristics include the patient's body posture feature points and facial feature points, which are not in the same data dimension as the patient's language information. Therefore, a series of processing of the patient's biological characteristics and the patient's language information is required to obtain the emotional characteristics that clearly express the patient's emotions.

[0097] Optionally, the implementation process of determining the patient's emotional characteristics based on the patient's biological characteristics and the patient's language information can be found in Figure 5 The steps are shown and the corresponding explanations.

[0098] In a possible embodiment, in combination with Figure 4 , Figure 5The process shown in FIG. 1 is to identify the patient's language feature point V from the patient's language information, and to obtain the doctor's language information and the patient's language information from the language data collected by the head display. For example, Fig. 9 shown.

[0099] S106: Using the pre-trained emotion model, perform emotion prediction on the patient's emotional characteristics to obtain the patient's emotions.

[0100] Among them, the emotional model is based on the patient's emotional characteristics as input and the patient's emotions as output, which can effectively identify the patient's emotional changes during the doctor-patient communication process.

[0101] In some examples, the types of emotion models include but are not limited to various deep learning models (such as CNN models, support vector machines, etc.). For details, the training process of the emotion model can be found in Figure 6 The steps are shown and the corresponding explanations.

[0102] S107: Display the emotion identifier corresponding to the patient's emotion.

[0103] Among them, the real-time display of emotional markers corresponding to the patient's emotions can help doctors make corresponding treatment measures in a timely manner. For example, during the diagnosis and treatment process, the doctor determines that the patient is in pain based on the currently displayed emotional markers. This may be because the current treatment method is too severe (for example, the wound is pressed too hard). The doctor can quickly make a corresponding diagnosis, such as canceling the current treatment method to reduce the patient's pain.

[0104] In some examples, the expression forms of emotion markers include but are not limited to: text, numerical values, expressions, etc.

[0105] It is understandable that the patient's pain is the best reference for evaluating treatment methods. Therefore, further evaluation of the patient's pain can provide a more reliable reference for the doctor's diagnosis.

[0106] Optionally, a pre-trained pain level model may be used to predict the pain level of the patient's emotional characteristics to obtain the patient's pain level. When the patient's pain level reaches a specified level, a corresponding prompt message is sent to the doctor.

[0107] It should be noted that the pain level model is based on the patient's emotional characteristics as input and the patient's pain level as output, and can effectively identify the patient's pain level during the doctor-patient communication process.

[0108] In some examples, the types of pain level models include but are not limited to various deep learning models (such as CNN models, support vector machines, etc.). Specifically, the training process of the pain level model can be found in Figure 6 The steps are shown and the corresponding explanations.

[0109] Generally speaking, when the patient's pain level reaches a specified level, it can be considered that the treatment measures currently taken by the doctor cause great pain to the patient, and a corresponding reminder message needs to be sent to the doctor so that the doctor can cancel the current treatment measures or use other treatment measures to reduce the patient's pain.

[0110] Optionally, when the patient's pain level reaches a specified level, the point mark obtained at the same timestamp may be specially displayed, such as highlighting the point mark or displaying it in red, to remind the doctor that the treatment method causes greater pain to the patient.

[0111] In addition, in order to facilitate doctors to conduct retrospective diagnosis and treatment, various types of information obtained in real time need to be saved in the corresponding database.

[0112] Optionally, based on the patient's biometrics, operation points, doctor's language information, patient's language information, patient's emotions and patient's pain level with the same timestamp, a doctor-patient communication record with a corresponding timestamp is generated, and the doctor-patient communication records with different timestamps during the doctor-patient communication process are saved in the patient's medical file. The medical file is used to assist doctors in retrospective diagnosis and treatment.

[0113] In a possible implementation, if a doctor performs an oral examination on a patient, because the patient cannot speak, the procedural pain caused by the doctor's use of instruments cannot be clearly expressed verbally (but the patient's body posture reacts violently and facial expressions also change significantly). Based on the scheme shown in the embodiment of the present application, the head display can prompt the patient that he is in a state of severe pain so that the doctor can adjust the operation in time to relieve the patient's pain.

[0114] In a possible implementation, the patient has a language barrier, but the patient's limbs can move. Based on the implementation of the scheme shown in this application, the corresponding emotional characteristics can be obtained from the patient's limb movements, thereby achieving effective tracking of the patient's emotional changes and helping doctors achieve effective communication.

[0115] Compared with the existing doctor-patient communication methods, the solution shown in the embodiment of the present application has the following advantages: 1. Convenience: The doctor wears a head display, freeing his hands, and there is no need for the doctor to observe the treatment instrument while observing the patient, which is of obvious help to dentists; 2. Efficiency: It can reduce doctor-patient dialogue and communication, and intuitively display the patient's status (including the three-dimensional model of the human body and the patient's emotions); 3. System integrity: It can realize the entire process from collecting patient expressions to data processing and then to the doctor receiving feedback; 4. Scalability and adaptability: It is not limited to oral medical diagnosis, but can be applied to all doctor-patient communication scenarios.

[0116] The process shown in S101-S107 above utilizes the model data collected in real time by the head display and the text of the doctor-patient conversation to realize the point marking on the three-dimensional model of the human body and the real-time tracking of the patient's emotions, providing effective and intuitive assistance for the doctor's treatment methods, making the doctor-patient communication more convenient and simple, so as to reduce the doctor's energy during the treatment process.

[0117] like Figure 2 As shown, it is a flowchart of another method for doctor-patient communication based on head display provided in an embodiment of the present application, including the following steps.

[0118] S201: Obtain model data collected by the head display in real time.

[0119] When the model data is obtained, the timestamp of the model data is also recorded so as to determine the timestamp of the patient's biometric features and the doctor's gesture obtained based on the model data.

[0120] S202: Preprocess the model data to obtain a target image that meets usage requirements.

[0121] If the model data is a video image frame, the means of preprocessing the video image frame include but are not limited to: scaling, cropping, normalization, etc. It is understandable that by preprocessing the video image frame, the image quality of the target image can be effectively improved so that the target image meets the use requirements.

[0122] In a possible implementation, the usage requirement may generally be the input image quality required when establishing a three-dimensional human body model (including a human body posture model and a three-dimensional facial model).

[0123] S203: Performing human body key point detection on the target image to obtain the patient's human body posture feature points and facial feature points.

[0124] Among them, the types of human body key points include human body posture key points and human face facial key points.

[0125] In some examples, for human body posture key points, a corresponding human body posture key point detection algorithm may be used to perform human body posture key point detection on a target image to obtain a feature point set A, where the feature point set A includes at least one human body posture feature point of the patient.

[0126] In some examples, for facial key points of a human face, a corresponding facial key point detection algorithm may be used to perform facial key point detection on a target image to obtain a feature point set F, where the feature point set F includes at least one facial feature point of the patient.

[0127] S204: Based on the human body posture feature points and facial feature points, the patient's biometric features in the doctor-patient communication process are identified.

[0128] Among them, the human body posture feature points and facial feature points based on the same timestamp are identified as the patient's biometric features corresponding to the timestamp.

[0129] It should be noted that during the doctor-patient communication process, the doctor may perform diagnosis and treatment on the patient, causing the patient to experience pain or surprise, which in turn causes changes in the patient's body posture and facial expression. For example, when the patient feels pain, he or she may involuntarily punch the patient and his or her facial expression may become ferocious. Therefore, the body posture feature points at the same timestamp can be used to record the body posture of the patient when he or she feels pain, and the facial feature points at the same timestamp can be used to record the patient's facial expression when he or she feels pain. To this end, based on the body posture feature points and facial feature points at the same timestamp, the patient's biometric features marked as the corresponding timestamp can be used to record the body posture and facial expression of the patient when the patient's feelings change in real time.

[0130] S205: Perform gesture recognition on the target image to obtain the doctor's gestures during the doctor-patient communication process.

[0131] From the doctor's perspective, in the process of diagnosing a patient, it is necessary to touch the patient with the hand to achieve the diagnosis. Therefore, the real-time image captured by the head-mounted display camera includes not only the patient, but also the doctor's hand and consultation tools (such as a stethoscope, etc.). A gesture recognition algorithm can be used to perform gesture recognition on the doctor's hand area in the target image to obtain the corresponding doctor's gesture.

[0132] It should be noted that real-time recording of the doctor's gestures during the doctor-patient communication process can well record the diagnostic methods (such as acupuncture, pressing, etc.) used by the doctor on the patient during the doctor-patient communication process, and provide an effective reference for the doctor's subsequent retrospective diagnosis and treatment.

[0133] The process shown in S201-S205 above obtains the target image from the model data collected in real time by the head display, and obtains the patient's biometric features and the doctor's gestures based on the target image recognition, so as to record in real time the human body posture and facial expressions when the patient's feelings change during the doctor-patient communication process, as well as the corresponding doctor's diagnosis methods.

[0134] like Figure 3 As shown, it is a flowchart of another method for doctor-patient communication based on head display provided in an embodiment of the present application, including the following steps.

[0135] S301: Constructing a three-dimensional human body model of the patient based on human body posture feature points and facial feature points.

[0136] Among them, the feature point reconstruction and matching algorithm can be used to reconstruct and match the human body posture feature points and facial feature points to obtain the patient's three-dimensional human body model.

[0137] S302: Rendering the three-dimensional human body model and the doctor's gestures to generate an interactive screen of the three-dimensional human body model and the doctor's gestures.

[0138] Among them, if the headset is equipped with a rendering engine, the interactive screen of the human body three-dimensional model and the doctor's gestures generated in real time by the rendering engine can be regarded as a virtual scene reconstruction of the doctor's diagnosis of the patient.

[0139] S303: Determine the operation point of the doctor's gesture based on the contact position between the three-dimensional human body model and the doctor's gesture in the interactive screen.

[0140] Among them, the point information of the contact position between the three-dimensional human body model and the doctor's gesture in the interactive screen can be extracted, and the point information can be identified as the operation point of the doctor's gesture.

[0141] In some examples, the contact position between the three-dimensional human body model and the doctor's gesture may be the contact position between the doctor's fingers or other hand areas and the three-dimensional human body model, or the contact position between the diagnostic tools in the doctor's hand (such as needles used in acupuncture) and the three-dimensional human body model.

[0142] S304: Add point marks corresponding to the operation points on the three-dimensional human body model to record the doctor's operation steps during the doctor-patient communication process.

[0143] Among them, after obtaining the operation points, point marks corresponding to the operation points can be generated. When there are a large number of operation points, the point marks corresponding to each operation point can be set with number indexes (such as 1, 2, 3, etc.). Based on the order of the number indexes, the occurrence sequence of the operation points can be reflected, thereby further reflecting the implementation order of each diagnostic method, so as to accurately record the doctor's operating steps in the doctor-patient communication process.

[0144] In some examples, point marks corresponding to operation points are added to the three-dimensional human body model, and the display effect of the point marks is different from the display effects of other areas on the three-dimensional human body model. For example, the point marks can be displayed in yellow, and other areas can be displayed in green.

[0145] The process shown in S301-S304 above utilizes the three-dimensional model of the human body constructed by the head display to add point marks corresponding to the operation points indicated by the doctor's gestures, so as to accurately record the doctor's operation steps during the doctor-patient communication process.

[0146] like Figure 4 As shown, it is a flowchart of another method for doctor-patient communication based on head display provided in an embodiment of the present application, including the following steps.

[0147] S401: Obtain language data collected by the head display in real time.

[0148] Among them, since the doctor wears the headset in real time and the headset is pre-authorized and enabled, language data can be collected in real time during the doctor-patient communication process. In addition, the timestamp of the language data needs to be recorded in order to determine the doctor's language information and the patient's language information at the same timestamp.

[0149] S402: Preprocess the language data to obtain language information that meets language quality requirements.

[0150] Among them, there may be other noises in the doctor-patient communication scene (such as air conditioning sound, external walking sound, etc.). Therefore, the language data needs to be preprocessed to obtain language information that meets the language quality requirements.

[0151] In some examples, the pre-processing methods adopted for the language data include, but are not limited to: high-pass filtering, low-pass filtering, etc.

[0152] S403: Using a preset digital human model, perform digital human identity recognition on the language information to obtain the doctor's language information and the patient's language information.

[0153] The digital human model is modeled based on the language features of the doctor. Generally speaking, the doctor can input his own speech language in the head display in advance, and use the language feature recognition algorithm preset in the head display to recognize the doctor's speech language to obtain the doctor's language features, and finally model the doctor's language features (such as timbre, frequency, etc.) to obtain a digital human model. The digital human model can recognize the doctor's language information that matches the doctor's language features in the language information, thereby distinguishing the doctor's language information from the patient's language information.

[0154] S404: Based on the doctor's language information and the patient's language information, the language information in the doctor-patient communication process is identified.

[0155] Among them, the doctor's language information and the patient's language information based on the same timestamp can be identified as the language information of the corresponding timestamp.

[0156] The process shown in S401-S404 above can obtain the doctor's language information and the patient's language information based on the language data collected in real time by the head display, so as to record the language information in the doctor-patient communication process in real time and provide an effective reference for the doctor's retrospective diagnosis and treatment.

[0157] like Figure 5 As shown, it is a flowchart of another method for doctor-patient communication based on head display provided in an embodiment of the present application, including the following steps.

[0158] S501: Determine the patient's biometric characteristics and patient language information at the same time stamp.

[0159] The patient's biological characteristics include the patient's body posture feature points and facial feature points.

[0160] It can be understood that the patient's biological characteristics and the patient's language information at the same timestamp can be determined based on the model data and language data at the same timestamp.

[0161] S502: Extract feature points from the patient's language information to obtain the patient's language feature points.

[0162] Among them, the language model preset in the head display (such as the Bert model) can be used to extract feature points of the patient's language information to obtain the patient's language feature points.

[0163] S503: Perform feature fusion on human posture feature points, facial feature points and language feature points to obtain the patient's emotional features.

[0164] Among them, feature fusion of human posture feature points, facial feature points and language feature points at the same timestamp can obtain the emotional features of the corresponding timestamp, thereby tracking the patient's emotional changes during doctor-patient communication in real time.

[0165] In some examples, the types of emotions include but are not limited to suspicion, disappointment, awe, concentration, surprise, pain, and sadness. Accordingly, the emotional features corresponding to different types of emotions are different, that is, the human body posture feature points, facial feature points, and language feature points corresponding to different types of emotions are different.

[0166] The process shown in S501-S503 above can utilize the patient's biological characteristics and the patient's language information to determine the patient's emotional characteristics, thereby providing a favorable reference for identifying the patient's emotions.

[0167] like Figure 6 As shown, it is a flowchart of another method for doctor-patient communication based on head display provided in an embodiment of the present application, including the following steps.

[0168] S601: Pre-build a multi-task learning model.

[0169] Among them, the multi-task includes emotion prediction task and pain level prediction task.

[0170] S602: Obtain training data for a multi-task learning model.

[0171] Among them, the training data includes emotion feature samples with preset emotion labels and pain level labels. The emotion feature samples are determined based on the body posture feature points, facial feature points and language feature points of sample patients in previous doctor-patient communication processes.

[0172] S603: Determine a corresponding training set and validation set based on the training data.

[0173] The data volume of the training set and the validation set can be set by the technical staff according to the actual situation. Generally speaking, the data volume of the validation set can be less than that of the training set.

[0174] S604: Train the multi-task learning model using the training set to obtain a trained multi-task learning model.

[0175] Among them, the multi-task learning model can be regarded as a supervised learning model. The training process of the supervised learning model is a common technical means in this field and will not be repeated here.

[0176] S605: Using the validation set, verify the trained multi-task learning model to obtain an emotion model and a pain level model.

[0177] Among them, using the validation set to verify the trained multi-task learning model can effectively improve the model performance of the emotion model and the pain level model.

[0178] For some examples of the training process of the emotion model and the pain level model, and the real-time use of the emotion model and the pain level model for new patients, see Fig.10 As shown ( Fig.10 The speech feature points shown in are a specific manifestation of language feature points).

[0179] The process shown in S601-S605 above can use the emotion feature samples with preset emotion labels and pain level labels to train the supervised learning model to obtain the emotion model and the pain level model, thereby providing an effective basis for the identification of the patient's emotion and the patient's pain level.

[0180] like Figure 7 , which is a schematic diagram of the architecture of a head display provided in an embodiment of the present application, including the units shown below.

[0181] The data analysis unit 100 is used to obtain the patient's biometric characteristics and the doctor's gestures during the doctor-patient communication process based on the model data collected in real time by the head display; the wearer of the head display includes at least the doctor.

[0182] Optionally, the data analysis unit 100 is specifically used to: obtain model data collected by the head display in real time; pre-process the model data to obtain a target image that meets the use requirements; perform human body key point detection on the target image to obtain the patient's human body posture feature points and facial feature points; identify the patient's biometric features in the doctor-patient communication process based on the human body posture feature points and facial feature points; perform gesture recognition on the target image to obtain the doctor's gestures in the doctor-patient communication process.

[0183] The step recording unit 200 is used to add point marks corresponding to the operation points indicated by the doctor's gestures on the three-dimensional human body model constructed by the head display based on the patient's biometrics, so as to record the doctor's operation steps during the doctor-patient communication process.

[0184] Optionally, the step recording unit 200 is specifically used to: construct a three-dimensional human body model of the patient based on human posture feature points and facial feature points; render the three-dimensional human body model and the doctor's gestures to generate an interactive screen of the three-dimensional human body model and the doctor's gestures; determine the operation point of the doctor's gesture based on the contact position between the three-dimensional human body model and the doctor's gesture in the interactive screen; add point marks corresponding to the operation points on the three-dimensional human body model to record the doctor's operation steps during the doctor-patient communication process.

[0185] The language analysis unit 300 is used to obtain language information in the doctor-patient communication process based on the language data collected in real time by the head display; the language information includes the doctor's language information and the patient's language information.

[0186] Optionally, the language analysis unit 300 is specifically used to: obtain language data collected by the head display in real time; pre-process the language data to obtain language information that meets language quality requirements; use a preset digital human model to perform digital human identity recognition on the language information to obtain doctor language information and patient language information; the digital human model is modeled based on the doctor's language characteristics; based on the doctor's language information and the patient's language information, identify it as language information in the doctor-patient communication process.

[0187] The information display unit 400 is used to display the doctor-patient dialogue text while displaying the three-dimensional human body model; the doctor-patient dialogue text is obtained by performing language recognition on language information.

[0188] Optionally, the information display unit 400 is specifically used to: translate language information while displaying the three-dimensional model of the human body to obtain target information that can be understood by the doctor; generate a doctor-patient dialogue text corresponding to the target information, and display the doctor-patient dialogue text.

[0189] The information display unit 400 is also used to display the emotion identifier corresponding to the patient's emotion.

[0190] The feature determination unit 500 determines the patient's emotional features based on the patient's biological features and the patient's language information.

[0191] Optionally, the feature determination unit 500 is specifically used to: determine the patient's biometric features and patient language information at the same timestamp; the patient's biometric features include the patient's body posture feature points and facial feature points; perform feature point extraction on the patient's language information to obtain the patient's language feature points; perform feature fusion on the body posture feature points, facial feature points and language feature points to obtain the patient's emotional features.

[0192] The emotion recognition unit 600 is used to use a pre-trained emotion model to perform emotion prediction on the patient's emotion characteristics to obtain the patient's emotion.

[0193] The pain recognition unit 700 is used to: use the pre-trained pain level model to predict the pain level of the patient's emotional characteristics to obtain the patient's pain level; when the patient's pain level reaches a specified level, send a corresponding prompt message to the doctor.

[0194] Optionally, the training process of the emotion model and the pain level model includes: pre-building a multi-task learning model; the multi-tasks include emotion prediction tasks and pain level prediction tasks; obtaining training data for the multi-task learning model; the training data includes emotion feature samples with preset emotion labels and pain level labels; the emotion feature samples are determined based on the human posture feature points, facial feature points and language feature points of sample patients in previous doctor-patient communication processes; based on the training data, determining the corresponding training set and validation set; using the training set, training the multi-task learning model to obtain the trained multi-task learning model; using the validation set, validating the trained multi-task learning model to obtain the emotion model and the pain level model.

[0195] The information storage unit 800 is used to: generate doctor-patient communication records with corresponding timestamps based on the patient's biological characteristics, operation points, doctor's language information, patient's language information, patient's emotions and patient's pain level with the same timestamp; save the doctor-patient communication records with different timestamps during the doctor-patient communication process into the patient's medical records; the medical records are used to assist doctors in retrospective diagnosis and treatment.

[0196] Each of the above-mentioned units utilizes the model data collected in real time by the head display and the text of the doctor-patient conversation to realize point marking on the three-dimensional model of the human body and real-time tracking of the patient's emotions, providing effective and intuitive assistance for the doctor's treatment methods, making doctor-patient communication more convenient and simple, thereby reducing the doctor's energy during the treatment process.

[0197] The present application also provides a computer-readable storage medium, which includes a stored program, wherein the program executes the head-mounted display-based doctor-patient communication method provided by the present application.

[0198] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. Certain features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0199] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A method for doctor-patient communication based on head-mounted display, characterized in that: include: Based on the model data collected in real time by the head-mounted display, the patient's biometric characteristics and the doctor's gestures during the doctor-patient communication process are obtained; The wearer of the head display includes at least a doctor; Adding point marks corresponding to the operation points indicated by the doctor's gestures on the three-dimensional human body model constructed by the head display based on the patient's biometrics to record the doctor's operation steps during the doctor-patient communication process; Based on the language data collected in real time by the head display, language information in the doctor-patient communication process is obtained; the language information includes doctor language information and patient language information; While displaying the three-dimensional human body model, displaying the doctor-patient dialogue text; The doctor-patient dialogue text is obtained by performing language recognition on the language information; Determining the patient's emotional characteristics based on the patient's biological characteristics and the patient's language information; Using a pre-trained emotion model, emotion prediction is performed on the emotion characteristics of the patient to obtain the patient's emotion; The emotion identifier corresponding to the patient's emotion is displayed.

2. The method according to claim 1, characterized in that The method further comprises: Using a pre-trained pain level model, predicting the pain level of the patient's emotional characteristics to obtain the patient's pain level; When the patient's pain level reaches a specified level, a corresponding prompt message is sent to the doctor.

3. The method according to claim 1, characterized in that Based on the model data collected in real time by the headset, the patient's biometrics and the doctor's gestures during the doctor-patient communication process are obtained, including: Obtain model data collected in real time by the head display; Preprocessing the model data to obtain a target image that meets usage requirements; Performing human body key point detection on the target image to obtain human body posture feature points and facial feature points of the patient; Based on the human body posture feature points and the facial feature points, identifying the patient's biometric features during the doctor-patient communication process; Perform gesture recognition on the target image to obtain the doctor's gesture during the doctor-patient communication process.

4. The method according to claim 3, characterized in that Adding point marks corresponding to the operation points indicated by the doctor's gestures on the three-dimensional human body model constructed by the head display based on the patient's biological characteristics to record the doctor's operation steps during the doctor-patient communication process includes: Based on the human body posture feature points and the facial feature points, construct a three-dimensional human body model of the patient; Rendering the three-dimensional human body model and the doctor's gesture to generate an interactive screen of the three-dimensional human body model and the doctor's gesture; Based on the contact position between the three-dimensional human body model and the doctor's gesture in the interactive screen, determining the operation point of the doctor's gesture; Point marks corresponding to the operation points are added to the three-dimensional human body model to record the doctor's operation steps during the doctor-patient communication process.

5. The method according to claim 1, characterized in that Based on the language data collected in real time by the head display, language information in the doctor-patient communication process is obtained, including: Obtaining language data collected in real time by the head display; Preprocessing the language data to obtain language information that meets language quality requirements; Using a preset digital human model, the language information is subjected to digital human identity recognition to obtain the doctor's language information and the patient's language information; the digital human model is modeled based on the doctor's language features; Based on the doctor's language information and the patient's language information, the language information in the doctor-patient communication process is identified.

6. The method according to claim 1, characterized in that Determining the patient's emotional characteristics based on the patient's biological characteristics and the patient's language information includes: Determine the patient's biometric features and patient language information at the same time stamp; the patient's biometric features include the patient's body posture feature points and facial feature points; Extracting feature points from the patient's language information to obtain language feature points of the patient; The human body posture feature points, the facial feature points and the language feature points are subjected to feature fusion to obtain the emotional features of the patient.

7. The method according to claim 2, characterized in that The training process of the emotion model and the pain level model includes: Pre-building a multi-task learning model; the multi-tasks include an emotion prediction task and a pain level prediction task; Obtaining training data for the multi-task learning model; the training data includes emotion feature samples preset with emotion labels and pain level labels; the emotion feature samples are determined based on human posture feature points, facial feature points, and language feature points of sample patients in previous doctor-patient communication processes; Based on the training data, determining a corresponding training set and a validation set; Using the training set, training the multi-task learning model to obtain a trained multi-task learning model; The trained multi-task learning model is verified using the verification set to obtain the emotion model and the pain level model.

8. The method according to claim 2, characterized in that: The method further comprises: Generate a doctor-patient communication record with a corresponding timestamp based on the patient's biometrics, operation points, doctor's language information, patient's language information, patient's emotions, and patient's pain level with the same timestamp; The doctor-patient communication records with different time stamps during the doctor-patient communication process are saved in the patient's medical records; the medical records are used to assist the doctor in retrospective diagnosis and treatment.

9. The method according to claim 1, characterized in that: While displaying the three-dimensional human body model, the doctor-patient dialogue text is displayed, including: While displaying the three-dimensional human body model, translating the language information to obtain target information that can be understood by the doctor; Generate a doctor-patient dialogue text corresponding to the target information, and display the doctor-patient dialogue text.

10. A head mounted display, characterized in that: include: A data analysis unit, which is used to obtain the patient's biometrics and the doctor's gestures during the doctor-patient communication process based on the model data collected in real time by the head display; The wearer of the head display includes at least a doctor; A step recording unit, configured to add point marks corresponding to the operation points indicated by the doctor's gestures on the three-dimensional human body model constructed by the head display based on the patient's biometrics, so as to record the doctor's operation steps during the doctor-patient communication process; A language analysis unit, configured to obtain language information in the doctor-patient communication process based on language data collected in real time by the head display; the language information includes doctor language information and patient language information; An information display unit, used to display the doctor-patient dialogue text while displaying the three-dimensional human body model; The doctor-patient dialogue text is obtained by performing language recognition on the language information; a feature determination unit, configured to determine an emotional feature of the patient based on the biological feature of the patient and the language information of the patient; An emotion recognition unit, used to use a pre-trained emotion model to perform emotion prediction on the patient's emotion characteristics to obtain the patient's emotion; The information display unit is also used to display the emotion identifier corresponding to the patient's emotion.

Images