Video consultation interface updating method, device, electronic device and storage medium

Abstract

The present invention relates to the field of digital medical technology, and provides a method, device, electronic device and medium for updating the interface of a video consultation, wherein the method comprises: obtaining audio data of a preset duration in a patient's video consultation at preset intervals; extracting multiple first voice feature vectors of the patient and the emotion label of each first voice feature vector; inputting multiple first voice feature vectors and the emotion label of each first voice feature vector and the emotion calculation rule carried by each emotion label into a preset first emotion recognition model for identification to obtain multiple emotion values; determining a final emotion label according to the multiple emotion values ​​and the emotion label of each emotion value; and updating the interface color of the video consultation based on the final emotion label. The present invention timely updates the interface color of the video consultation by calculating the patient's final target emotion label in the audio data, and the patient adjusts his emotion in time according to the interface color, thereby improving the consultation efficiency of the video consultation.

Description

Technical Field

[0001] The present invention relates to the field of digital medical technology, and in particular to a method, device, electronic equipment and storage medium for updating an interface of a video consultation. Background Art

[0002] At present, online consultations mainly include text and picture consultations and video consultations. Among them, video consultations are consultations between patients and doctors via video. During the consultation process, patients will show different emotions when facing the type or severity of the disease. In the existing video consultation process, patients cannot perceive their own emotions, and doctors cannot grasp the patients' emotions, resulting in low efficiency of video consultations. In addition, the existing video consultation interface has a single color, which can easily make patients feel depressed, resulting in low patient satisfaction, which in turn affects the efficiency of video consultations. Summary of the invention

[0003] In view of the above content, it is necessary to propose a method, device, electronic device and storage medium for updating the interface of video consultation. By calculating the patient's final target emotion identifier in the audio data, the interface color of the video consultation is updated in time. The patient adjusts his emotion in time according to the interface color, thereby improving the consultation efficiency of the video consultation.

[0004] A first aspect of the present invention provides a method for updating a video consultation interface, the method comprising:

[0005] Acquire audio data of a preset duration in the patient's video consultation at a preset period;

[0006] Extracting a plurality of first speech feature vectors of the patient and an emotion label of each first speech feature vector from the audio data;

[0007] Inputting the plurality of first speech feature vectors, the emotion label of each first speech feature vector, and the emotion calculation rule carried by each emotion label into a preset first emotion recognition model for recognition to obtain a plurality of emotion values ​​of the patient;

[0008] Determining a final emotion identifier of the patient according to the multiple emotion values ​​of the patient and the emotion identifier of each emotion value;

[0009] The interface color of the video consultation is updated based on the final emotion identification of the patient.

[0010] Optionally, determining the final emotion identifier of the patient according to the multiple emotion values ​​of the patient and the emotion identifier of each emotion value includes:

[0011] Calculate the number of acquisitions of audio data of a preset duration in a video consultation of a patient;

[0012] Calculate the average value of each emotion according to the acquisition times and the multiple emotion values;

[0013] Obtaining an emotion identifier for each emotion according to the emotion average value of each emotion;

[0014] Get the pixel value of the emotion tag for each emotion;

[0015] Calculate the product of the pixel value of the emotion identifier of each emotion and the corresponding preset weight to obtain the target pixel value of each emotion;

[0016] An average value of the plurality of target pixel values ​​is calculated to obtain a final target pixel value, and a final emotion identification of the patient is obtained according to the final target pixel value.

[0017] Optionally, extracting a plurality of first speech feature vectors of the patient from the audio data comprises:

[0018] extracting a plurality of first speech segments of the patient from the audio data;

[0019] Feature extraction is performed on the multiple first speech segments to obtain multiple first speech feature vectors.

[0020] Optionally, the process of training the first emotion recognition model includes:

[0021] Obtaining multiple historical speech feature vectors from multiple audio data in multiple historical video consultations;

[0022] Obtain the emotion label of each historical speech feature vector and the emotion calculation rule corresponding to the emotion label of each historical speech feature vector as a sample data set;

[0023] Dividing the sample data set into a training set and a test set;

[0024] Inputting the training set into a preset neural network for training to obtain a first emotion recognition model;

[0025] Inputting the test set into the first emotion recognition model for testing, and calculating the test pass rate;

[0026] If the test pass rate is greater than a preset pass rate threshold, it is determined that the training of the first emotion recognition model is completed; if the test pass rate is less than the preset pass rate threshold, the number of training sets is increased and the first emotion recognition model is trained again.

[0027] Optionally, after updating the interface color of the video consultation based on the final emotion identifier of the patient, the method further includes:

[0028] Recording the frequency of color changes of the video consultation interface;

[0029] Comparing the frequency of the color change of the video consultation interface with a preset frequency threshold;

[0030] When the frequency of the interface color change is greater than or equal to the preset frequency threshold, pausing the video consultation;

[0031] When the frequency of the interface color change is less than the preset frequency threshold, the video consultation is maintained.

[0032] Optionally, after pausing the video consultation, the method further includes:

[0033] Analyze the audio data in the video consultation to obtain an analysis result;

[0034] When the analysis result shows that the reason for suspending the video consultation is the doctor's fault, an alarm message is sent to the doctor's terminal, and the video consultation is maintained;

[0035] When the analysis result is that the reason for suspending the video consultation is the patient's reason, a suggestion for terminating the video consultation is sent to the patient's terminal.

[0036] Optionally, parsing the audio data in the video consultation to obtain the parsing result includes:

[0037] Extracting a plurality of second voice segments of the doctor from the audio data in the video consultation;

[0038] Performing feature extraction on the plurality of second speech segments to obtain a plurality of second speech feature vectors;

[0039] Inputting the plurality of second speech feature vectors and the corresponding emotion tags and the emotion calculation rules carried by each emotion tag into a preset second emotion recognition model for recognition to obtain a plurality of emotion values ​​of the doctor;

[0040] Calculating an average of the multiple emotion values ​​as the doctor's emotion value;

[0041] When it is determined that the doctor's emotion value is greater than or equal to a preset emotion threshold, determining that the analysis result is that the reason for pausing the video consultation is the doctor's reason;

[0042] When it is determined that the doctor's emotion value is less than the preset emotion threshold, the analysis result is determined to be that the reason for pausing the video consultation is the patient's reason.

[0043] A second aspect of the present invention provides a device for updating an interface of a video consultation, the device comprising:

[0044] An acquisition module, used for acquiring audio data of a preset duration in a patient's video consultation at a preset period;

[0045] An extraction module, used for extracting a plurality of first speech feature vectors of the patient and an emotion label of each first speech feature vector from the audio data;

[0046] A recognition module, used for inputting the plurality of first speech feature vectors, the emotion label of each first speech feature vector, and the emotion calculation rule carried by each emotion label into a preset first emotion recognition model for recognition to obtain a plurality of emotion values ​​of the patient;

[0047] A determination module, configured to determine a final emotion identifier of the patient according to the multiple emotion values ​​of the patient and the emotion identifier of each emotion value;

[0048] An updating module is used to update the interface color of the video consultation based on the final emotion identification of the patient.

[0049] A third aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the processor is configured to implement the video consultation interface update method when executing a computer program stored in the memory.

[0050] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, and the computer program, when executed by a processor, implements the method for updating the interface of the video consultation.

[0051] In summary, the interface updating method, device, electronic device and medium of the video consultation of the present invention, on the one hand, obtains the audio data of the preset time length in the video consultation of the patient at every preset period, calculates the final target emotion identifier of the patient corresponding to the audio data, and timely updates the interface color of the video consultation according to the final target emotion identifier, so that the patient can intuitively see the changes in his or her emotions according to the interface color and make timely emotional adjustments. At the same time, the doctor can also adjust the consultation strategy according to the patient's emotions in time, thereby improving the consultation efficiency of the video consultation and the patient's satisfaction; on the other hand, by setting different emotion calculation rules for different emotion tags, and inputting the emotion tag of each speech feature vector and the emotion calculation rule corresponding to each emotion tag into the first emotion recognition model for identification to obtain the emotion value, instead of directly identifying the emotion value according to multiple first speech feature vectors and corresponding emotion tags, the accuracy of the identified emotion value is improved; finally, by obtaining the multiple emotion values ​​and the corresponding emotion identifiers, the final emotion identifier of the patient is determined according to the emotion identifiers of the multiple emotions, instead of directly determining the final emotion of the patient according to the multiple emotion values, multiple dimensions are considered to improve the accuracy of obtaining the final emotion identifier of the patient. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flow chart of a method for updating the interface of a video consultation provided in the first embodiment of the present invention.

[0053] Figure 2 It is a structural diagram of the interface updating device for video medical consultation provided in the second embodiment of the present invention.

[0054] Figure 3 It is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0055] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0057] Embodiment 1

[0058] Figure 1This is a flow chart of a method for updating the interface of a video consultation provided in the first embodiment of the present invention.

[0059] In this embodiment, the video consultation interface update method can be applied to electronic devices. For electronic devices that need to perform video consultation interface updates, the video consultation interface update function provided by the method of the present invention can be directly integrated into the electronic device, or run in the electronic device in the form of a software development kit (Software Development Kit, SKD).

[0060] like Figure 1 As shown, the method for updating the interface of the video consultation specifically includes the following steps. According to different requirements, the order of the steps in the flowchart can be changed, and some can be omitted.

[0061] S11, obtaining audio data of a preset duration in the patient's video consultation at a preset period.

[0062] In this embodiment, a period for obtaining audio data of a preset duration in a video consultation of a patient can be preset, such as 30 seconds, with 1 minute as one acquisition period. The duration for obtaining audio data can be preset, and the preset duration is set based on historical video consultations. During the video consultation between the doctor and the patient, the camera device is controlled to obtain the audio data of the patient in the video consultation once every preset period. Specifically, the camera device can be provided by the terminal device or pre-installed in the terminal device.

[0063] S12, extracting a plurality of first speech feature vectors of the patient and an emotion label of each first speech feature vector from the audio data.

[0064] In this embodiment, during the video consultation between the doctor and the patient, multiple voice segments in the audio stream are identified to obtain corresponding multiple first voice feature vectors and emotion labels of each first voice feature vector, wherein each first voice feature vector corresponds to an emotion label. Specifically, the emotion label may include: neutral, happy, sad, angry, etc., and the emotion label may represent the patient's current emotion.

[0065] Optionally, extracting a plurality of first speech feature vectors of the patient from the audio data comprises:

[0066] extracting a plurality of first speech segments of the patient from the audio data;

[0067] Feature extraction is performed on the multiple first speech segments to obtain multiple first speech feature vectors.

[0068] In this embodiment, a plurality of first speech segments of the patient may be extracted from the audio data, and features may be extracted from each first speech segment to obtain a plurality of first speech feature vectors of the patient.

[0069] S13, inputting the multiple first speech feature vectors, the emotion label of each first speech feature vector, and the emotion calculation rule carried by each emotion label into a preset first emotion recognition model for recognition to obtain multiple emotion values ​​of the patient.

[0070] In this embodiment, the first emotion recognition model can be pre-trained. After obtaining the patient's speech feature vector, the multiple first speech feature vectors and the corresponding emotion tags and the emotion calculation rules carried by each emotion tag are input into the preset first emotion recognition model for recognition, and multiple emotion values ​​output by the first emotion recognition model are obtained.

[0071] Specifically, the process of training the first emotion recognition model includes:

[0072] 21) obtaining a plurality of historical speech feature vectors from a plurality of audio data in a plurality of historical video consultations;

[0073] 22) Obtaining the emotion label of each historical speech feature vector and the emotion calculation rule corresponding to the emotion label of each historical speech feature vector as a sample data set;

[0074] 23) dividing the sample data set into a training set and a test set;

[0075] 24) inputting the training set into a preset neural network for training to obtain a first emotion recognition model;

[0076] 25) inputting the test set into the first emotion recognition model for testing, and calculating the test pass rate;

[0077] 26) If the test pass rate is greater than the preset pass rate threshold, it is determined that the training of the first emotion recognition model is completed; if the test pass rate is less than the preset pass rate threshold, the number of training sets is increased and the first emotion recognition model is trained again.

[0078] In this embodiment, a corresponding emotion calculation rule is set for each emotion label according to different speech feature vectors. For example, the emotion calculation rule set for the happy emotion label is: the absolute value of the speech feature vector ≥ N, and the corresponding happy emotion value is 10; N-1≤the absolute value of the speech feature vector≤M, and the corresponding happy emotion value is 8; M-1≤the absolute value of the speech feature vector≤P, and the corresponding happy emotion value is 5; the absolute value of the speech feature vector≤P-1, and the corresponding happy emotion value is 0.

[0079] In this embodiment, multiple historical video consultations and multiple audio data in each historical video consultation are obtained. Specifically, the historical video consultation may be a video consultation reported by a medical institution, or may be a video consultation obtained by a server from multiple preset data sources. Multiple historical voice feature vectors in multiple audio data in multiple historical voice feature vectors are extracted from the multiple historical video consultations, and the emotion label of each voice feature vector and the emotion calculation rule corresponding to each emotion label are input into the first emotion recognition model to identify the emotion value corresponding to each voice feature vector. In the subsequent prediction process, each voice feature vector and the corresponding emotion label are used as new data to increase the number of the data set, and the first emotion recognition model is retrained based on the new data set, and the first emotion recognition model is continuously updated, thereby continuously improving the emotion recognition rate.

[0080] In this embodiment, different emotion calculation rules are set for different emotion labels, and the emotion label of each speech feature vector and the emotion calculation rule corresponding to each emotion label are input into the first emotion recognition model to identify and obtain the emotion value, rather than directly identifying the emotion value based on multiple first speech feature vectors and the corresponding emotion labels, thereby improving the accuracy of the identified emotion value.

[0081] S14, determining a final emotion identifier of the patient according to the multiple emotion values ​​of the patient and the emotion identifier of each emotion value.

[0082] In this embodiment, the emotion identifier is used to uniquely identify the patient's emotional state, and different emotion values ​​correspond to different emotion identifiers. Specifically, the emotion identifier may include: blue, orange, black, red, etc.

[0083] Optionally, determining the final emotion identifier of the patient according to the multiple emotion values ​​of the patient and the emotion identifier of each emotion value includes:

[0084] Calculate the number of acquisitions of audio data of a preset duration in a video consultation of a patient;

[0085] Calculate the average value of each emotion according to the acquisition times and the multiple emotion values;

[0086] Obtaining an emotion identifier for each emotion according to the emotion average value of each emotion;

[0087] Get the pixel value of the emotion tag for each emotion;

[0088] Calculate the product of the pixel value of the emotion identifier of each emotion and the corresponding preset weight to obtain the target pixel value of each emotion;

[0089] An average value of the plurality of target pixel values ​​is calculated to obtain a final target pixel value, and a final emotion identification of the patient is obtained according to the final target pixel value.

[0090] For example, assuming that during a 5-minute video consultation, the patient's voice data is acquired every 30 seconds, then the voice-first emotion recognition model can output 10 emotion values, including 3 happy emotion values ​​and 7 sad emotion values. The 3 happy emotion values ​​are added up and divided by 10 to obtain the average emotion value of the happy emotion, and the emotion mark of the happy emotion is obtained according to the average emotion value of the happy emotion: blue, and the 7 sad emotion values ​​are added up and divided by 10 to obtain the average emotion value of the sad emotion, and the emotion mark of the sad emotion is obtained according to the average emotion value of the sad emotion: black.

[0091] In other embodiments, the emotion identifier may also be set to different facial expressions, or the emotion identifier may also be set to other identifiers that can identify emotions. Different settings may be made according to different situations, and the present invention is not limited thereto.

[0092] In this embodiment, the final emotion identification is used to identify the final emotion of the patient in the audio data of the preset duration. The pixel value of the emotion identification of each emotion is multiplied by the corresponding preset weight, and then multiple target pixel values ​​of multiple emotions are accumulated and the average is calculated to obtain the final target pixel value. The final emotion identification of the patient is then obtained by matching the final target pixel value, rather than directly determining the final emotion of the patient based on multiple emotion values. This method considers multiple dimensions and improves the accuracy of obtaining the final emotion identification of the patient.

[0093] S15, updating the interface color of the video consultation based on the final emotion identification of the patient.

[0094] In this embodiment, the interface color of the video consultation is updated in real time according to the final emotion mark of the patient, and the interface color of the video consultation represents the emotion of the patient during the re-consultation process.

[0095] In this embodiment, by obtaining audio data of a preset duration in the patient's video consultation at preset intervals, the final target emotion identifier of the patient corresponding to the audio data is calculated, and the interface color of the video consultation is timely updated according to the final target emotion identifier. The patient can intuitively see the changes in his or her emotions according to the interface color and make timely emotional adjustments. At the same time, the doctor can also adjust the consultation strategy in time according to the patient's emotions, thereby improving the consultation efficiency of the video consultation and the patient's satisfaction.

[0096] Further, after updating the interface color of the video consultation based on the final emotion identification of the patient, the method further includes:

[0097] Recording the frequency of color changes of the video consultation interface;

[0098] Comparing the frequency of the color change of the video consultation interface with a preset frequency threshold;

[0099] When the frequency of the interface color change is greater than or equal to the preset frequency threshold, pausing the video consultation;

[0100] When the frequency of the interface color change is less than the preset frequency threshold, the video consultation is maintained.

[0101] In this embodiment, a frequency threshold of the interface color change can be set in advance. When the frequency of the interface color change is greater than or equal to the preset frequency threshold, it is determined that the patient's emotions are very unstable and the video consultation needs to be paused to facilitate the patient to adjust his emotions, thereby improving the experience and satisfaction of the video consultation.

[0102] Furthermore, after pausing the video consultation, the method further includes:

[0103] Analyze the audio data in the video consultation to obtain an analysis result;

[0104] When the analysis result shows that the reason for suspending the video consultation is the doctor's fault, an alarm message is sent to the doctor's terminal, and the video consultation is maintained;

[0105] When the analysis result is that the reason for suspending the video consultation is the patient's reason, a suggestion for terminating the video consultation is sent to the patient's terminal.

[0106] In this embodiment, when it is determined that the reason for suspending the video consultation is due to the doctor's reasons, an alarm message is sent to the doctor's terminal to promptly remind the doctor to control his emotions and maintain the video consultation, thereby improving the consultation efficiency of the video consultation; when the analysis result is that the reason for suspending the video consultation is due to the patient's reasons, a suggestion to terminate the video consultation is sent to the patient's terminal, and the patient can decide whether to terminate the video consultation based on his current emotions, thereby improving the patient's satisfaction and at the same time improving the consultation efficiency of the video consultation.

[0107] Furthermore, the parsing of the audio data in the video consultation to obtain the parsing result includes:

[0108] Extracting a plurality of second voice segments of the doctor from the audio data in the video consultation;

[0109] Performing feature extraction on the plurality of second speech segments to obtain a plurality of second speech feature vectors;

[0110] Inputting the plurality of second speech feature vectors and the corresponding emotion tags and the emotion calculation rules carried by each emotion tag into a preset second emotion recognition model for recognition to obtain a plurality of emotion values ​​of the doctor;

[0111] Calculating an average of the multiple emotion values ​​as the doctor's emotion value;

[0112] When it is determined that the doctor's emotion value is greater than or equal to a preset emotion threshold, determining that the analysis result is that the reason for pausing the video consultation is the doctor's reason;

[0113] When it is determined that the doctor's emotion value is less than the preset emotion threshold, the analysis result is determined to be that the reason for pausing the video consultation is the patient's reason.

[0114] In this embodiment, since the video consultation is an interactive process, that is, there are voice clips of the doctor and voice clips of the patient, voice clips of different interactive parties are extracted according to different scenarios. When the doctor's emotion analysis is required, multiple second voice clips of the doctor are extracted, and second voice feature vectors are extracted based on the multiple voice clips of the doctor. The extracted multiple second voice feature vectors and the corresponding emotion tags and the emotion calculation rules carried by each emotion tag are input into a preset second emotion recognition model for identification to obtain multiple emotion values ​​of the doctor. The training process of the second emotion recognition model is the same as the training process of the first emotion recognition model. By calculating the average value of the multiple emotion values ​​output by the second emotion recognition model as the emotion value of the doctor, the doctor's emotion can be quickly determined, and it can be judged whether the doctor's emotion exceeds the preset emotion threshold. The reason for pausing the video consultation is determined according to the judgment result, thereby improving the efficiency of determining the reason for pausing the video consultation, thereby improving the consultation efficiency of the video consultation.

[0115] To summarize, the method for updating the interface of video consultation described in this embodiment obtains audio data of a preset length in the patient's video consultation at preset intervals; extracts multiple first voice feature vectors of the patient and the emotion label of each first voice feature vector from the audio data; inputs the multiple first voice feature vectors and the emotion label of each first voice feature vector and the emotion calculation rule carried by each emotion label into a preset first emotion recognition model for identification to obtain multiple emotion values ​​of the patient; determines the patient's final emotion identifier based on the patient's multiple emotion values ​​and the emotion identifier of each emotion value; and updates the interface color of the video consultation based on the patient's final emotion identifier.

[0116] In this embodiment, on the one hand, by obtaining the audio data of a preset time in the patient's video consultation at every preset period, the final target emotion identifier of the patient corresponding to the audio data is calculated, and the interface color of the video consultation is updated in time according to the final target emotion identifier. The patient can intuitively see the changes in his or her emotions according to the interface color and make timely emotional adjustments. At the same time, the doctor can also adjust the consultation strategy in time according to the patient's emotions, thereby improving the consultation efficiency of the video consultation and the patient's satisfaction. On the other hand, by setting different emotion calculation rules for different emotion tags, and inputting the emotion tag of each speech feature vector and the emotion calculation rule corresponding to each emotion tag into the first emotion recognition model for identification to obtain the emotion value, instead of directly identifying the emotion value according to multiple first speech feature vectors and corresponding emotion tags, the accuracy of the identified emotion value is improved. In addition, by obtaining the multiple emotion values ​​and the corresponding emotion identifiers, the final emotion identifier of the patient is determined according to the emotion identifiers of the multiple emotions, instead of directly determining the final emotion of the patient according to multiple emotion values, multiple dimensions are considered to improve the accuracy of obtaining the final emotion identifier of the patient.

[0117] Embodiment 2

[0118] Figure 2 It is a structural diagram of the interface updating device for video medical consultation provided in the second embodiment of the present invention.

[0119] In some embodiments, the video consultation interface update device 20 may include a plurality of functional modules composed of program code segments. The program code of each program segment in the video consultation interface update device 20 may be stored in the memory of the electronic device and executed by the at least one processor to perform (see Figure 1 Description) Video consultation interface updated.

[0120] In this embodiment, the interface updating device 20 of the video consultation can be divided into multiple functional modules according to the functions it performs. The functional modules may include: an acquisition module 201, an extraction module 202, an identification module 203, a determination module 204, an update module 205, a comparison module 206 and a parsing module 207. The module referred to in the present invention refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0121] The acquisition module 201 is used to acquire the audio data of a preset duration in the video consultation of the patient at every preset period.

[0122] In this embodiment, the acquisition cycle can be preset to acquire the audio data of a preset duration in the patient's video consultation, such as 30 seconds, with 1 minute as one acquisition cycle. The duration for acquiring audio data can be preset, and the preset duration is set based on historical video consultations. During the video consultation between the doctor and the patient, the camera device is controlled to acquire the audio data of the patient in the video consultation once every preset period. Specifically, the camera device can be built-in to the terminal device, or can be pre-installed in the terminal device.

[0123] The extraction module 202 is used to extract a plurality of first speech feature vectors of the patient and an emotion label of each first speech feature vector from the audio data.

[0124] In this embodiment, during the video consultation between the doctor and the patient, multiple voice segments in the audio stream are identified to obtain corresponding multiple first voice feature vectors and emotion labels of each first voice feature vector, wherein each first voice feature vector corresponds to an emotion label. Specifically, the emotion label may include: neutral, happy, sad, angry, etc., and the emotion label may represent the patient's current emotion.

[0125] Optionally, the extraction module 202 extracts a plurality of first speech feature vectors of the patient from the audio data, including:

[0126] extracting a plurality of first speech segments of the patient from the audio data;

[0127] Feature extraction is performed on the multiple first speech segments to obtain multiple first speech feature vectors.

[0128] In this embodiment, a plurality of first speech segments of the patient may be extracted from the audio data, and features may be extracted from each first speech segment to obtain a plurality of first speech feature vectors of the patient.

[0129] The recognition module 203 is used to input the multiple first speech feature vectors, the emotion label of each first speech feature vector, and the emotion calculation rule carried by each emotion label into a preset first emotion recognition model for recognition to obtain multiple emotion values ​​of the patient.

[0130] In this embodiment, the first emotion recognition model can be pre-trained. After obtaining the patient's speech feature vector, the multiple first speech feature vectors and the corresponding emotion tags and the emotion calculation rules carried by each emotion tag are input into the preset first emotion recognition model for recognition, and multiple emotion values ​​output by the first emotion recognition model are obtained.

[0131] Specifically, the process of training the first emotion recognition model includes:

[0132] 21) obtaining a plurality of historical speech feature vectors from a plurality of audio data in a plurality of historical video consultations;

[0133] 22) Obtaining the emotion label of each historical speech feature vector and the emotion calculation rule corresponding to the emotion label of each historical speech feature vector as a sample data set;

[0134] 23) dividing the sample data set into a training set and a test set;

[0135] 24) inputting the training set into a preset neural network for training to obtain a first emotion recognition model;

[0136] 25) inputting the test set into the first emotion recognition model for testing, and calculating the test pass rate;

[0137] 26) If the test pass rate is greater than the preset pass rate threshold, it is determined that the training of the first emotion recognition model is completed; if the test pass rate is less than the preset pass rate threshold, the number of training sets is increased and the first emotion recognition model is trained again.

[0138] In this embodiment, a corresponding emotion calculation rule is set for each emotion label according to different speech feature vectors. For example, the emotion calculation rule set for the happy emotion label is: the absolute value of the speech feature vector ≥ N, and the corresponding happy emotion value is 10; N-1≤the absolute value of the speech feature vector≤M, and the corresponding happy emotion value is 8; M-1≤the absolute value of the speech feature vector≤P, and the corresponding happy emotion value is 5; the absolute value of the speech feature vector≤P-1, and the corresponding happy emotion value is 0.

[0139] In this embodiment, multiple historical video consultations and multiple audio data in each historical video consultation are obtained. Specifically, the historical video consultation may be a video consultation reported by a medical institution, or may be a video consultation obtained by a server from multiple preset data sources. Multiple historical voice feature vectors in multiple audio data in multiple historical voice feature vectors are extracted from the multiple historical video consultations, and the emotion label of each voice feature vector and the emotion calculation rule corresponding to each emotion label are input into the first emotion recognition model to identify the emotion value corresponding to each voice feature vector. In the subsequent prediction process, each voice feature vector and the corresponding emotion label are used as new data to increase the number of the data set, and the first emotion recognition model is retrained based on the new data set, and the first emotion recognition model is continuously updated, thereby continuously improving the emotion recognition rate.

[0140] In this embodiment, different emotion calculation rules are set for different emotion labels, and the emotion label of each speech feature vector and the emotion calculation rule corresponding to each emotion label are input into the first emotion recognition model to identify and obtain the emotion value, rather than directly identifying the emotion value based on multiple first speech feature vectors and the corresponding emotion labels, thereby improving the accuracy of the identified emotion value.

[0141] The determination module 204 is used to determine a final emotion identification of the patient according to the multiple emotion values ​​of the patient and the emotion identification of each emotion value.

[0142] In this embodiment, the emotion identifier is used to uniquely identify the patient's emotional state, and different emotion values ​​correspond to different emotion identifiers. Specifically, the emotion identifier may include: blue, orange, black, red, etc.

[0143] Optionally, the determining module 204 determines the final emotion identification of the patient according to the multiple emotion values ​​of the patient and the emotion identification of each emotion value, including:

[0144] Calculate the number of acquisitions of audio data of a preset duration in a video consultation of a patient;

[0145] Calculate the average value of each emotion according to the acquisition times and the multiple emotion values;

[0146] Obtaining an emotion identifier for each emotion according to the emotion average value of each emotion;

[0147] Get the pixel value of the emotion tag for each emotion;

[0148] Calculate the product of the pixel value of the emotion identifier of each emotion and the corresponding preset weight to obtain the target pixel value of each emotion;

[0149] An average value of the plurality of target pixel values ​​is calculated to obtain a final target pixel value, and a final emotion identification of the patient is obtained according to the final target pixel value.

[0150] For example, assuming that during a 5-minute video consultation, the patient's voice data is acquired every 30 seconds, then the voice-first emotion recognition model can output 10 emotion values, including 3 happy emotion values ​​and 7 sad emotion values. The 3 happy emotion values ​​are added up and divided by 10 to obtain the average emotion value of the happy emotion, and the emotion mark of the happy emotion is obtained according to the average emotion value of the happy emotion: blue, and the 7 sad emotion values ​​are added up and divided by 10 to obtain the average emotion value of the sad emotion, and the emotion mark of the sad emotion is obtained according to the average emotion value of the sad emotion: black.

[0151] In other embodiments, the emotion identifier may also be set to different facial expressions, or the emotion identifier may also be set to other identifiers that can identify emotions. Different settings may be made according to different situations, and the present invention is not limited thereto.

[0152] In this embodiment, the final emotion identification is used to identify the final emotion of the patient in the audio data of the preset duration. The pixel value of the emotion identification of each emotion is multiplied by the corresponding preset weight, and then multiple target pixel values ​​of multiple emotions are accumulated and the average is calculated to obtain the final target pixel value. The final emotion identification of the patient is then obtained by matching the final target pixel value, rather than directly determining the final emotion of the patient based on multiple emotion values. This method considers multiple dimensions and improves the accuracy of obtaining the final emotion identification of the patient.

[0153] The updating module 205 is used to update the interface color of the video consultation based on the final emotion identification of the patient.

[0154] In this embodiment, the interface color of the video consultation is updated in real time according to the final emotion mark of the patient, and the interface color of the video consultation represents the emotion of the patient during the re-consultation process.

[0155] In this embodiment, by obtaining audio data of a preset duration in the patient's video consultation at preset intervals, the final target emotion identifier of the patient corresponding to the audio data is calculated, and the interface color of the video consultation is timely updated according to the final target emotion identifier. The patient can intuitively see the changes in his or her emotions according to the interface color and make timely emotional adjustments. At the same time, the doctor can also adjust the consultation strategy in time according to the patient's emotions, thereby improving the consultation efficiency of the video consultation and the patient's satisfaction.

[0156] Furthermore, after updating the interface color of the video consultation based on the final emotion identifier of the patient, the updating module 205 records the frequency of the interface color change of the video consultation.

[0157] The comparison module 206 is used to compare the frequency of the interface color change of the video consultation with a preset frequency threshold; when the frequency of the interface color change is greater than or equal to the preset frequency threshold, the video consultation is paused; when the frequency of the interface color change is less than the preset frequency threshold, the video consultation is maintained.

[0158] In this embodiment, a frequency threshold of the interface color change can be set in advance. When the frequency of the interface color change is greater than or equal to the preset frequency threshold, it is determined that the patient's emotions are very unstable and the video consultation needs to be paused to facilitate the patient to adjust his emotions, thereby improving the experience and satisfaction of the video consultation.

[0159] Furthermore, after pausing the video consultation, the analysis module 207 is used to analyze the audio data in the video consultation to obtain an analysis result; when the analysis result is that the reason for pausing the video consultation is the doctor's reason, an alarm message is sent to the doctor's terminal and the video consultation is maintained; when the analysis result is that the reason for pausing the video consultation is the patient's reason, a suggestion to terminate the video consultation is sent to the patient's terminal.

[0160] In the present embodiment, when it is determined that the reason for suspending the video consultation is due to the doctor's reasons, an alarm message is sent to the doctor's terminal to promptly remind the doctor to control his emotions and maintain the video consultation, thereby improving the efficiency of the video consultation; when the analysis result shows that the reason for suspending the video consultation is due to the patient's reasons, a suggestion to terminate the video consultation is sent to the patient's terminal, and the patient can decide whether to terminate the video consultation based on his current emotions, thereby improving the patient's satisfaction and at the same time improving the efficiency of the video consultation.

[0161] Furthermore, the parsing of the audio data in the video consultation to obtain the parsing result includes:

[0162] Extracting a plurality of second voice segments of the doctor from the audio data in the video consultation;

[0163] Performing feature extraction on the plurality of second speech segments to obtain a plurality of second speech feature vectors;

[0164] Inputting the plurality of second speech feature vectors and the corresponding emotion tags and the emotion calculation rules carried by each emotion tag into a preset second emotion recognition model for recognition to obtain a plurality of emotion values ​​of the doctor;

[0165] Calculating an average of the multiple emotion values ​​as the doctor's emotion value;

[0166] When it is determined that the doctor's emotion value is greater than or equal to a preset emotion threshold, determining that the analysis result is that the reason for pausing the video consultation is the doctor's reason;

[0167] When it is determined that the doctor's emotion value is less than the preset emotion threshold, the analysis result is determined to be that the reason for pausing the video consultation is the patient's reason.

[0168] In this embodiment, since the video consultation is an interactive process, that is, there are voice clips of the doctor and voice clips of the patient, voice clips of different interactive parties are extracted according to different scenarios. When the doctor's emotion analysis is required, multiple second voice clips of the doctor are extracted, and second voice feature vectors are extracted based on the multiple voice clips of the doctor. The extracted multiple second voice feature vectors and the corresponding emotion tags and the emotion calculation rules carried by each emotion tag are input into a preset second emotion recognition model for identification to obtain multiple emotion values ​​of the doctor. The training process of the second emotion recognition model is the same as the training process of the first emotion recognition model. By calculating the average value of the multiple emotion values ​​output by the second emotion recognition model as the emotion value of the doctor, the doctor's emotion can be quickly determined, and it can be judged whether the doctor's emotion exceeds the preset emotion threshold. The reason for pausing the video consultation is determined according to the judgment result, thereby improving the efficiency of determining the reason for pausing the video consultation, thereby improving the consultation efficiency of the video consultation.

[0169] In summary, the interface update device of the video consultation described in this embodiment obtains audio data of a preset length in the patient's video consultation at preset intervals; extracts multiple first voice feature vectors of the patient and the emotion label of each first voice feature vector from the audio data; inputs the multiple first voice feature vectors and the emotion label of each first voice feature vector and the emotion calculation rule carried by each emotion label into a preset first emotion recognition model for identification to obtain multiple emotion values ​​of the patient; determines the patient's final emotion identification according to the patient's multiple emotion values ​​and the emotion identification of each emotion value; and updates the interface color of the video consultation based on the patient's final emotion identification.

[0170] In this embodiment, on the one hand, by obtaining the audio data of a preset time in the patient's video consultation at every preset period, the final target emotion identifier of the patient corresponding to the audio data is calculated, and the interface color of the video consultation is updated in time according to the final target emotion identifier. The patient can intuitively see the changes in his or her emotions according to the interface color and make timely emotional adjustments. At the same time, the doctor can also adjust the consultation strategy in time according to the patient's emotions, thereby improving the consultation efficiency of the video consultation and the patient's satisfaction. On the other hand, by setting different emotion calculation rules for different emotion tags, and inputting the emotion tag of each speech feature vector and the emotion calculation rule corresponding to each emotion tag into the first emotion recognition model for identification to obtain the emotion value, instead of directly identifying the emotion value according to multiple first speech feature vectors and corresponding emotion tags, the accuracy of the identified emotion value is improved. In addition, by obtaining the multiple emotion values ​​and the corresponding emotion identifiers, the final emotion identifier of the patient is determined according to the emotion identifiers of the multiple emotions, instead of directly determining the final emotion of the patient according to multiple emotion values, multiple dimensions are considered to improve the accuracy of obtaining the final emotion identifier of the patient.

[0171] Embodiment 3

[0172] See also Figure 3 FIG. 2 is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. In a preferred embodiment of the present invention, the electronic device 3 includes a memory 31 , at least one processor 32 , at least one communication bus 33 and a transceiver 34 .

[0173] Those skilled in the art should understand that Figure 3 The structure of the electronic device shown does not constitute a limitation of the embodiments of the present invention, and may be a bus structure or a star structure. The electronic device 3 may also include more or less other hardware or software than shown in the figure, or a different component arrangement.

[0174] In some embodiments, the electronic device 3 is an electronic device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits, programmable gate arrays, digital processors, and embedded devices. The electronic device 3 may also include client devices, which include but are not limited to any electronic product that can interact with a client through a keyboard, mouse, remote control, touchpad, or voice-controlled device, such as a personal computer, tablet computer, smart phone, digital camera, etc.

[0175] It should be noted that the electronic device 3 is only an example, and other existing or future electronic products that are suitable for the present invention should also be included in the protection scope of the present invention and included here by reference.

[0176] In some embodiments, the memory 31 is used to store program codes and various data, such as the interface update device 20 of the video consultation installed in the electronic device 3, and to achieve high-speed and automatic access to programs or data during the operation of the electronic device 3. The memory 31 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0177] In some embodiments, the at least one processor 32 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The at least one processor 32 is the control core (Control Unit) of the electronic device 3, and uses various interfaces and lines to connect various components of the entire electronic device 3, and executes various functions and processes data of the electronic device 3 by running or executing programs or modules stored in the memory 31, and calling data stored in the memory 31.

[0178] In some embodiments, the at least one communication bus 33 is configured to implement connection and communication between the memory 31 and the at least one processor 32, etc.

[0179] Although not shown, the electronic device 3 may also include a power source (such as a battery) for supplying power to each component. Optionally, the power source may be logically connected to the at least one processor 32 through a power management device, so that the power management device can manage charging, discharging, and power consumption management. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 3 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0180] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0181] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, electronic device, or network device, etc.) or a processor to execute a part of the method described in each embodiment of the present invention.

[0182] In a further embodiment, in combination Figure 2 The at least one processor 32 can execute the operating device of the electronic device 3 and various installed application programs (such as the interface updating device 20 of the video consultation), program codes, etc., for example, the above-mentioned modules.

[0183] The memory 31 stores program codes, and the at least one processor 32 can call the program codes stored in the memory 31 to execute related functions. For example, Figure 2 The modules described are program codes stored in the memory 31 and are executed by the at least one processor 32, thereby realizing the functions of the modules to achieve the purpose of updating the interface of the video consultation.

[0184] In one embodiment of the present invention, the memory 31 stores a plurality of instructions, and the plurality of instructions are executed by the at least one processor 32 to implement the function of updating the interface of the video consultation.

[0185] Specifically, the specific implementation method of the at least one processor 32 for the above instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0186] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0187] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0188] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0189] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "including" does not exclude other units or, and the singular does not exclude the plural. The multiple units or devices stated in the present invention can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0190] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for updating the interface of a video consultation, characterized in that: The method comprises: Acquire audio data of a preset duration in the patient's video consultation at a preset period; Extracting a plurality of first speech feature vectors of the patient and an emotion label of each first speech feature vector from the audio data; Inputting the multiple first speech feature vectors, the emotion label of each first speech feature vector, and the emotion calculation rule carried by each emotion label into a preset first emotion recognition model for identification to obtain multiple emotion values ​​of the patient, wherein a corresponding emotion calculation rule is set for each emotion label according to different first speech feature vectors; Determining the final emotion identification of the patient according to the multiple emotion values ​​of the patient and the emotion identification of each emotion value, including: calculating the number of times audio data of a preset duration in the video consultation of the patient is obtained; calculating the emotion average value of each emotion according to the number of acquisitions and the multiple emotion values; obtaining the emotion identification of each emotion according to the emotion average value of each emotion; obtaining the pixel value of the emotion identification of each emotion; calculating the product of the pixel value of the emotion identification of each emotion and the corresponding preset weight to obtain the target pixel value of each emotion; calculating the average value of multiple target pixel values ​​to obtain the final target pixel value, and obtaining the final emotion identification of the patient according to the final target pixel value; The interface color of the video consultation is updated based on the final emotion identification of the patient.

2. The method for updating the interface of video consultation according to claim 1, characterized in that: The extracting a plurality of first speech feature vectors of the patient from the audio data comprises: extracting a plurality of first speech segments of the patient from the audio data; Feature extraction is performed on the multiple first speech segments to obtain multiple first speech feature vectors.

3. The method for updating the interface of video consultation according to claim 1, characterized in that: The process of training the first emotion recognition model includes: Obtaining multiple historical speech feature vectors from multiple audio data in multiple historical video consultations; Obtain the emotion label of each historical speech feature vector and the emotion calculation rule corresponding to the emotion label of each historical speech feature vector as a sample data set; Dividing the sample data set into a training set and a test set; Inputting the training set into a preset neural network for training to obtain a first emotion recognition model; Inputting the test set into the first emotion recognition model for testing, and calculating the test pass rate; If the test pass rate is greater than a preset pass rate threshold, it is determined that the training of the first emotion recognition model is completed; if the test pass rate is less than the preset pass rate threshold, the number of training sets is increased and the first emotion recognition model is trained again.

4. The method for updating the interface of video consultation according to claim 1, characterized in that: After updating the interface color of the video consultation based on the final emotion identifier of the patient, the method further includes: Recording the frequency of color changes of the video consultation interface; Comparing the frequency of the color change of the video consultation interface with a preset frequency threshold; When the frequency of the interface color change is greater than or equal to the preset frequency threshold, pausing the video consultation; When the frequency of the interface color change is less than the preset frequency threshold, the video consultation is maintained.

5. The method for updating the interface of video consultation according to claim 4, characterized in that: After pausing the video consultation, the method further includes: Analyze the audio data in the video consultation to obtain an analysis result; When the analysis result shows that the reason for suspending the video consultation is the doctor's fault, an alarm message is sent to the doctor's terminal, and the video consultation is maintained; When the analysis result is that the reason for suspending the video consultation is the patient's reason, a suggestion for terminating the video consultation is sent to the patient's terminal.

6. The method for updating the interface of video consultation according to claim 5, characterized in that: The analyzing the audio data in the video consultation to obtain the analysis result includes: Extracting a plurality of second voice segments of the doctor from the audio data in the video consultation; Performing feature extraction on the plurality of second speech segments to obtain a plurality of second speech feature vectors; Inputting the plurality of second speech feature vectors and the corresponding emotion tags and the emotion calculation rules carried by each emotion tag into a preset second emotion recognition model for recognition to obtain a plurality of emotion values ​​of the doctor; Calculating an average of the multiple emotion values ​​as the doctor's emotion value; When it is determined that the doctor's emotion value is greater than or equal to a preset emotion threshold, determining that the analysis result is that the reason for pausing the video consultation is the doctor's reason; When it is determined that the doctor's emotion value is less than the preset emotion threshold, the analysis result is determined to be that the reason for pausing the video consultation is the patient's reason.

7. A device for updating the interface of a video consultation, characterized in that: The device comprises: An acquisition module, used for acquiring audio data of a preset duration in a patient's video consultation at a preset period; An extraction module, used for extracting a plurality of first speech feature vectors of the patient and an emotion label of each first speech feature vector from the audio data; an identification module, configured to input the plurality of first speech feature vectors, the emotion label of each first speech feature vector, and the emotion calculation rule carried by each emotion label into a preset first emotion recognition model for identification to obtain a plurality of emotion values ​​of the patient, wherein a corresponding emotion calculation rule is set for each emotion label according to different first speech feature vectors; A determination module, for determining a final emotion identification of the patient according to the patient's multiple emotion values ​​and the emotion identification of each emotion value, comprising: calculating the number of acquisitions of audio data of a preset duration in a video consultation of the patient; calculating the emotion average value of each emotion according to the acquisition number and the multiple emotion values; obtaining the emotion identification of each emotion according to the emotion average value of each emotion; obtaining the pixel value of the emotion identification of each emotion; calculating the product of the pixel value of the emotion identification of each emotion and the corresponding preset weight to obtain the target pixel value of each emotion; calculating the average value of multiple target pixel values ​​to obtain a final target pixel value, and obtaining the final emotion identification of the patient according to the final target pixel value; An updating module is used to update the interface color of the video consultation based on the final emotion identification of the patient.

8. An electronic device, characterized in that: The electronic device includes a processor and a memory, and the processor is used to implement the video consultation interface update method as described in any one of claims 1 to 6 when executing the computer program stored in the memory.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for updating the interface of video consultation as described in any one of claims 1 to 6 is implemented.

Images