A medical information interaction terminal and interaction method based on AI analysis

Abstract

The present invention relates to the field of interaction, and specifically to a medical information interaction terminal and interaction method based on AI analysis, which extracts target audio features and text from request voice information, and then performs differential calculations on its target audio features and reference audio features to eliminate the influence of emotion recognition caused by personal vocal characteristics. Then, an AI model is used to perform emotion analysis on the reference audio features, and accurate emotion recognition results are obtained by combining the emotion entities extracted from the text. Then, based on the emotion analysis results and the extracted key entities, determine which interaction method to use. When the patient or family member is emotionally stable and intends to ask questions, a question and answer template or a large language model is used to respond. When the patient or family member is emotionally unstable or intends to make a request, the request voice information is forwarded to the corresponding medical staff. It can not only help medical staff reduce responses, but also avoid the stimulation of doctor-patient conflicts caused by the use of a single question and answer robot.

Description

Technical Field

[0001] The present invention relates to the field of interactive services, and specifically to a medical information interactive terminal and an interactive method based on AI analysis. Background Art

[0002] Currently, many hospitals are equipped with service terminals for inpatients. Patients can make service requests through the service terminals. For example, they can send different instructions by touching the corresponding cases on the bedside terminal to call medical staff or caregivers.

[0003] In some cases, patients' needs are relatively simple, or they just ask some questions. In this case, providing only a way to interact by calling medical staff will bring unnecessary burden to medical staff, thereby increasing their workload. However, if only Q&A robots are used to answer patients' questions, in some cases it will also cause dissatisfaction among patients or their families, and easily trigger conflicts between doctors and patients. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a medical information interaction terminal and interaction method based on AI analysis to solve the problems in the background technology.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A medical information interaction terminal based on AI analysis of the present invention comprises the steps of:

[0007] The audio collection module is used to collect the request voice information from the user when receiving the request instruction from the user;

[0008] A feature extraction module is used to convert the request voice information into text, extract the target audio features in the request voice information; and perform entity extraction on the text based on a pre-built key entity library to obtain key entities, wherein the key entity library includes request entities, inquiry entities, symptom entities and emotion entities, and the types of the target audio features are predetermined;

[0009] The emotion analysis module is used to determine the user's reference audio features based on the user's personal information and a pre-built reference audio feature library, or to extract reference audio features from the user's sampled audio; and to perform differential calculation based on the reference audio features and the target audio features to obtain differential features; and to input the differential features into a pre-built emotion recognition model to obtain the user's emotion prediction results; and to determine the emotion analysis results based on the emotion prediction results and key entities;

[0010] An interaction module is used to forward the request voice information to a target object when the emotion prediction result is abnormal; forward the request voice information to a target object when the emotion prediction result is not abnormal and the key entity is a request entity or a symptom entity; and perform interaction based on the text and a pre-built question and answer template when the emotion prediction result is not abnormal and the key entity is an inquiry entity, or perform interaction based on the text and a called language model.

[0011] In an embodiment of the present application, extracting target audio features from the request voice information includes:

[0012] Performing noise reduction and pre-emphasis on the request voice information to obtain pre-processed voice information;

[0013] Converting the preprocessed voice information into PCM data, extracting spectrum data of a plurality of voice frames in the preprocessed voice information, and extracting Mel-frequency cepstral coefficients of a plurality of voice frames in the preprocessed voice information;

[0014] Extracting speech segments and non-speech segments from the PCM data, and extracting the timestamp of each word in the speech segment; calculating the speech rate based on the timestamp of each word and the number of words, and calculating the pause ratio based on the speech segment and the non-speech segment;

[0015] Extracting fundamental frequencies of the plurality of speech frames from the spectrum data of the plurality of speech frames, calculating average values ​​and variances of the fundamental frequencies of the plurality of speech frames, and obtaining fundamental frequency mean values ​​and fundamental frequency variances;

[0016] Extracting first N mel-frequency cepstral values ​​from the mel-frequency cepstral coefficients of the multiple speech frames, and calculating the variance of all mel-frequency cepstral values ​​with the same bit order to obtain a flatness characteristic coefficient;

[0017] The target audio feature is constructed based on the speaking rate, the pause ratio, the fundamental frequency mean, the fundamental frequency variance and the flatness characteristic coefficient.

[0018] In an embodiment of the present application, the process of determining the target audio feature includes:

[0019] Acquire a speech data sample, and annotate the speech data sample to obtain a labeled speech data sample, wherein the speech data sample includes a reference speech segment in a calm state and an emotion-expressing speech segment;

[0020] Extracting reference audio features of the reference speech segment, extracting audio features of the emotion expression speech segment, and performing differential calculation based on the reference audio features of the reference speech segment and the audio features of the emotion expression speech segment to obtain differential sample features;

[0021] Classifying the speech data samples based on the labels to obtain multiple emotion data sets;

[0022] Normalizing each difference sample feature in each emotion data set to obtain normalized data of each difference sample feature in each emotion data set;

[0023] The variance of the normalized data of each difference sample feature in each emotion data set is calculated, and the difference sample feature whose variance is less than a preset variance threshold is used as the target difference feature of the corresponding emotion label; and the audio feature corresponding to the target difference feature is used as the target audio feature.

[0024] In one embodiment of the present application, entity extraction is performed on the text based on a pre-built key entity library to obtain key entities, including:

[0025] Performing word segmentation processing on the text to obtain multiple words;

[0026] Converting a plurality of words into word vectors, and matching the word vectors with a request entity template, an inquiry entity template, a symptom entity template, and an emotion entity template in the key entity library, wherein the request entity template, the inquiry entity template, the symptom entity template, and the emotion entity template are all stored in the key entity library in the form of vectors;

[0027] When any word vector matches the request entity template, the inquiry entity template, the symptom entity template or the emotion entity template, the word vector is used as a key entity, and the type of the key entity is determined based on the matching result.

[0028] In one embodiment of the present application, the method for constructing the question and answer template is as follows:

[0029] Obtain hospitalization-related and medical-related knowledge corpus samples;

[0030] Extracting triples from the knowledge corpus sample, wherein the triples include a head entity-a relation-a tail entity;

[0031] Constructing a knowledge graph with the head entity or the tail entity as a node;

[0032] A plurality of triples are extracted from the knowledge graph, and a question template is constructed using two of the triples, and an answer template is constructed based on the other one of the triples, so as to obtain a question-answer template.

[0033] In an embodiment of the present application, determining a sentiment analysis result based on the sentiment prediction result and the key entity includes:

[0034] Determine multiple emotion categories in the emotion prediction result And the confidence of multiple emotion categories ;

[0035] When the key entity does not include an emotion entity, taking the emotion type with the highest confidence as the emotion analysis result;

[0036] The key entity includes an emotion entity, and multiple emotion categories corresponding to the emotion entity are determined based on a pre-constructed emotion entity-emotion category relationship map. And a variety of emotions Probability ; Confidence of the same emotion category and probability The weighted sum is taken to obtain the evaluation score; and the emotion type with the highest evaluation score is taken as the emotion analysis result.

[0037] In one embodiment of the present application, a method for constructing an emotion entity-emotion category relationship graph includes:

[0038] Acquire a text sample with an emotion entity, and annotate the text sample to obtain a text sample with a label, wherein the text sample is extracted from a medical communication record;

[0039] Classify text samples based on emotional entities to obtain sample units of multiple emotional entities;

[0040] Count the emotion category labels in the sample unit of the emotion entity, and calculate the probability of each emotion label in the sample unit;

[0041] An emotion entity-emotion category relationship graph is constructed based on the probabilities of multiple emotion entities and multiple emotion labels of multiple emotion entities.

[0042] In an embodiment of the present application, performing interaction based on the text and a pre-built question-and-answer template, or performing interaction based on the text and a called language macro model, includes:

[0043] extracting keywords from the text;

[0044] Matching the keywords with triples in the knowledge graph;

[0045] When the keyword matches two of the triples in the knowledge graph, the corresponding answer template is output; when the keyword does not match two of any triples in the knowledge graph, the text is handed over to the language model for interaction.

[0046] In one embodiment of the present application, the method for constructing the emotion recognition model includes:

[0047] Constructing a training data set based on the target difference feature and the label of the target difference feature;

[0048] Based on the training data set, the artificial neural network is trained in combination with the gradient descent method to obtain an emotion recognition model.

[0049] The present application also provides a medical information interaction method based on AI analysis, comprising the steps of:

[0050] When receiving a request instruction from a user, collecting request voice information from the user;

[0051] Convert the request voice information into text, extract the target audio features in the request voice information; and perform entity extraction on the text based on a pre-built key entity library to obtain key entities, wherein the key entity library includes request entities, inquiry entities, symptom entities and emotion entities, and the type of the target audio features is predetermined;

[0052] Determine the user's reference audio features based on the user's personal information and a pre-built reference audio feature library, or extract the reference audio features from the user's sampled audio; perform differential calculation based on the reference audio features and the target audio features to obtain differential features; input the differential features into a pre-built emotion recognition model to obtain the user's emotion prediction results; and determine the emotion analysis results based on the emotion prediction results and key entities;

[0053] When the emotion prediction result is abnormal, the request voice information is forwarded to the target object; when the emotion prediction result is not abnormal and the key entity is a request entity or a symptom entity, the request voice information is forwarded to the target object; when the emotion prediction result is not abnormal and the key entity is an inquiry entity, the interaction is performed based on the text and a pre-built question and answer template, or, the interaction is performed based on the text and a called language model.

[0054] The beneficial effect of the present invention is that a medical information interaction terminal and interaction method based on AI analysis of the present invention obtains the request voice information of the inpatient or family members when the inpatient or family members have a request or inquiry demand. By extracting the target audio features and text of the request voice information, and then performing differential calculation on the target audio features and the reference audio features, the emotional recognition influence caused by the personal voice characteristics is eliminated. Then, the AI ​​model is used to perform emotional analysis on the reference audio features to obtain the emotional analysis results. Combined with the emotional entities extracted from the text, accurate emotional recognition results can be obtained. Then, based on the emotional analysis results and the extracted key entities, it is determined which interaction method to adopt. Therefore, when the patient or family members are emotionally stable and just want to ask some questions, a question and answer template or a large language model is used to respond. When the patient or family members are emotionally unstable or make a request, the request voice information is forwarded to the corresponding medical staff. While helping medical staff to filter out some non-critical information, the emotions of patients or family members are also taken into account to avoid stimulating serious medical conflicts. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0056] Figure 1 This is a schematic diagram of a medical information interaction terminal based on AI analysis in one embodiment of the present application;

[0057] Figure 2 It is a structural schematic diagram of a medical information interaction terminal based on AI analysis shown in one embodiment of the present application;

[0058] Figure 3 This is a flow chart of extracting target audio features in one embodiment of the present application;

[0059] Figure 4 A flowchart for extracting key entities in an embodiment of the present application;

[0060] Figure 5 Schematic diagram of a process for determining target audio features in an embodiment of the present application;

[0061] Figure 6 It is a flowchart of a medical information interaction method based on AI analysis shown in one embodiment of the present application. DETAILED DESCRIPTION

[0062] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0063] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show the layers related to the present invention rather than being drawn according to the number, shape and size of the layers in actual implementation. In actual implementation, the type, quantity and proportion of each layer may be changed arbitrarily, and the layer layout may also be more complicated.

[0064] In the following description, numerous details are discussed to provide a more thorough explanation of embodiments of the present invention; however, it is apparent to one skilled in the art that embodiments of the present invention may be practiced without these specific details.

[0065] Figure 1 Schematic diagram of a medical information interaction terminal based on AI analysis in one embodiment of the present application, such as Figure 1 As shown, the interactive terminal in the present application is arranged at the head of the bed, and a microphone 110, a speaker 120 and a touch display screen 130 are provided on the front thereof. When the patient or family member has a request or inquiry, the instruction can be input through the touch display screen 130 to start the microphone 110 to collect audio. The collected audio is feature extracted by the processor in the terminal, and the extracted features can be sent to the background host for AI analysis, or the built-in model can be used to perform emotion analysis. Combining emotional features and intention information, different interaction methods are executed, and the interaction results are output by the speaker 120.

[0066] Figure 2 is a schematic diagram of a medical information interaction terminal based on AI analysis shown in an embodiment of the present application, such as Figure 2 In this embodiment, a medical information interaction terminal based on AI analysis may include the following steps:

[0067] The audio collection module 210 is used to collect the request voice information from the user when receiving the request instruction from the user;

[0068] The user's request instruction is sent through the touch display screen of the touch terminal;

[0069] When a request instruction from a user is received, the microphone is started to collect the user's request voice information.

[0070] The feature extraction module 220 is used to convert the request voice information into text, extract the target audio features in the request voice information; and perform entity extraction on the text based on a pre-built key entity library to obtain key entities, wherein the key entity library includes request entities, inquiry entities, symptom entities and emotion entities, and the type of the target audio features is predetermined;

[0071] In this application, the collected request voice information contains a lot of environmental noise, so it needs to be preprocessed. The main preprocessing methods in this application are noise reduction and pre-emphasis. The purpose of noise reduction is to remove environmental noise, and pre-emphasis is to filter the input signal to increase the amplitude of high-frequency components. This is done to compensate for the natural attenuation produced by human speech, because high-frequency phonemes (such as consonants) in human voices are often weaker than low-frequency phonemes (such as vowels), but they are very important for understanding language.

[0072] The preprocessed speech data highlights the human voice. This application uses the automatic speech recognition (ASR) tool to convert the preprocessed speech data into text and extract the text entity features. The preprocessed speech data is also subjected to feature extraction to provide input for the emotion recognition model.

[0073] In this application, the types of target audio features are determined during the model training process, including speech rate, pause ratio (indicating coherence), fundamental frequency mean (indicating tone), fundamental frequency variance (indicating tone change) and flatness characteristic coefficient;

[0074] The principle is: when a person is emotionally excited or depressed, his or her speaking speed, coherence, tone, tone changes, and the smoothness of his or her speech will change.

[0075] Figure 3 is a flowchart of extracting target audio features in an embodiment of the present application, such as Figure 3 As shown, the advance process of the target audio features is as follows:

[0076] S310, performing noise reduction and pre-emphasis on the request voice information to obtain pre-processed voice information;

[0077] The principle and beneficial effects of pretreatment are as described above and will not be repeated here.

[0078] S320, converting the preprocessed voice information into PCM data, extracting spectrum data of multiple voice frames in the preprocessed voice information, and extracting Mel Frequency Cepstrum (MFCC) coefficients of multiple voice frames in the preprocessed voice information;

[0079] S330, extracting speech segments and non-speech segments from the PCM data, and extracting the timestamp of each word in the speech segment; calculating the speech rate based on the timestamp of each word and the number of words, and calculating the pause ratio based on the speech segment and the non-speech segment;

[0080] Extracting speech and non-speech segments from PCM data requires voice activity detection (VAD) algorithms, while determining the timestamp of each word in PCM data requires calling a trained automatic speech recognition (ASR) model.

[0081] The pronunciation duration of each word can be obtained using the timestamp of each word. , calculate the sum of the pronunciation duration of all words , and then use the number of words extracted from the previous text The speaking speed can be calculated , speech speed The mathematical expression is:

[0082]

[0083] Pause ratio The mathematical expression is:

[0084]

[0085] In the formula, is the duration of the speech segment, is the duration of the non-speech segment.

[0086] S340, extracting fundamental frequencies of the multiple speech frames from the frequency spectrum data of the multiple speech frames, calculating average values ​​and variances of the fundamental frequencies of the multiple speech frames, and obtaining fundamental frequency mean values ​​and fundamental frequency variances;

[0087] The fundamental frequency is mainly used to reflect the user's tone. This application uses spectrum data to extract the fundamental frequency of multiple speech frames. ; Then for multiple base frequencies Calculate the average and variance . This reflects the overall level of the tone and the stability of the tone.

[0088] S350, extracting the first N mel-frequency cepstral values ​​from the mel-frequency cepstral coefficients of the multiple speech frames, and calculating the variance of all mel-frequency cepstral values ​​with the same bit order to obtain a flatness characteristic coefficient;

[0089] Mel-frequency cepstral coefficients (MFCCs) are a set of features that mimic the way the human auditory system perceives sound, especially the difference in sensitivity at different frequencies. These coefficients are commonly used to characterize speech signals and are widely used in tasks such as speech recognition, speaker identification, and other audio analysis.

[0090] This application extracts the first five Mel-frequency cepstrum values, and then extracts 1-5 Mel-frequency cepstrum values ​​from the Mel-frequency cepstrum coefficients corresponding to multiple time frames to form five sequences, and then calculates the variance of each sequence to obtain the smoothness characteristic coefficient.

[0091] Calculating the variance of each MFCC coefficient can help understand the stability of the speech signal in different frequency bands. A lower variance may mean a smoother speaking style. On the contrary, it means that the speaking style of the speech signal in the corresponding frequency band is unstable.

[0092] In addition to extracting the target audio features, it is also necessary to extract key entities in the text, which are keywords used in this application to determine user intent and emotions, including request entities, inquiry entities, symptom entities, and emotion entities.

[0093] This application pre-builds a key entity library containing request entity templates, inquiry entity templates, symptom entity templates, and emotion entity templates. The entity templates are stored in the library in the form of vectors. Then, the key entities in the text are found by matching, and the matching basis is cosine similarity.

[0094] S360, constructing a target audio feature based on the speech rate, the pause ratio, the fundamental frequency mean, the fundamental frequency variance and the flatness characteristic coefficient.

[0095] Figure 4 This is a flowchart of key entity extraction in an embodiment of the present application, such as Figure 4 As shown in Figure 2, the extraction process of key entities is as follows:

[0096] S410, performing word segmentation processing on the text to obtain a plurality of words;

[0097] The present application utilizes a word segmentation algorithm to perform word segmentation processing on text. The word segmentation algorithm may be a forward maximum matching method, a reverse maximum matching method, a bidirectional maximum matching method, a model extraction method, and the like.

[0098] S420, converting a plurality of words into word vectors, and matching the word vectors with request entity templates, inquiry entity templates, symptom entity templates, and emotion entity templates in the key entity library, wherein the request entity template, the inquiry entity template, the symptom entity template, and the emotion entity template are all stored in the key entity library in the form of vectors;

[0099] In this application, words are converted into word vectors based on one-hot encoding. The request entity template, inquiry entity template, symptom entity template and emotion entity template in the key entity library use the same encoding method.

[0100] In this application, entities that may represent requests, inquiries, symptoms, and emotions during the conversation are stored in a library.

[0101] In a specific example, entities representing requests include: "please help me", "I want to find", "call", etc. When these entities appear, it means that the patient or family member is expressing the intention of making a request to the medical staff or caregivers.

[0102] Entities that represent inquiries include: "Why", "How to adjust", "How to turn off", "How to turn on", "What is it for", etc. The appearance of these entities indicates that the patient or family member is expressing their intention to inquire with the medical staff or caregivers. In some specific scenarios, the request entity and the inquiry entity may appear at the same time, for example: "How do I adjust the backrest at the head of the bed? Please help me adjust it." In this scenario where both the request and inquiry entities can be extracted, the intent is set as an inquiry.

[0103] The main structure of the entity representing the symptom is "body part" + "symptom expression", such as "headache", "leg numbness", etc. It can also be a separate symptom expression, such as only expressing "pain" or "ache".

[0104] The entities that represent emotions are mainly some modal particles or onomatopeia. For example, when patients speak, they mix in pain onomatopeia such as "hiss". In addition, there are modal particles such as "ah" and "ah" that represent various emotions. These words will serve as supporting information for the results of the emotion analysis in the subsequent emotion analysis process.

[0105] S430, when any word vector matches the request entity template, the inquiry entity template, the symptom entity template or the emotion entity template, the word vector is used as a key entity, and the type of the key entity is determined based on the matching result.

[0106] This application uses the cosine similarity matching method, and the matching degree , where is the word vector, is an entity template. When the matching degree is greater than the corresponding threshold, the word vector is determined With entity template match.

[0107] Finally, when the matching is completed, the key entities matched with the key entity library can be extracted.

[0108] The emotion analysis module 230 is used to determine the user's reference audio features based on the user's personal information and a pre-built reference audio feature library, or to extract reference audio features from the user's sampled audio; and to perform differential calculation based on the reference audio features and the target audio features to obtain differential features; and to input the differential features into a pre-built emotion recognition model to obtain the user's emotion prediction results; and to determine the emotion analysis results based on the emotion prediction results and key entities;

[0109] In the previous text, the target audio features and emotional entities are extracted. In this application, the target audio features and emotional entities are used to perform emotion analysis.

[0110] In general, this application performs sentiment analysis based on AI analysis. In addition, it introduces sentiment entities to correct the analysis results. It also uses reference audio features and target audio features to perform differential calculations, thereby eliminating sentiment analysis errors caused by different personal speaking habits (such as dialects, male and female tones, speaking tones of different ages, etc.).

[0111] If the user's speech audio sample in a peaceful state can be collected in advance, the target audio features of the speech audio sample can be collected through the consistent process described above as reference audio features. However, in most cases, it is difficult to collect the user's speech audio sample in a peaceful state in advance. Therefore, in this embodiment, the reference audio feature library of different people is constructed by dividing the population, so as to find the corresponding reference audio features according to the personal information of the current user.

[0112] The construction process of the reference audio feature library can be:

[0113] (1) Collect a large number of speech samples and divide them according to the characteristics of different dimensions to obtain sample sets corresponding to multiple groups of people. The division dimensions can be dialect type, age, gender, etc.

[0114] (2) Extract target audio features from all speech samples in the sample sets corresponding to different groups of people. Please refer to the previous article for the extraction process, which will not be repeated here;

[0115] (3) For each audio feature in the sample set corresponding to different groups of people, the variance and mean are calculated to obtain the feature mean and feature variance;

[0116] (4) In order to remove specific samples, in this application, for any audio feature of any sample set, the feature is performed, the sample is removed from the set, and the feature variance of the sample set after removing the specific sample is calculated. When the feature variance is still greater than the preset variance threshold, the specific sample is removed again, and the cycle is repeated until the feature variance of the sample set is no greater than the preset variance threshold. A set with relatively concentrated feature values ​​is obtained.

[0117] For example, a sample set of males aged 25-25 and from a certain dialect region has the above operations performed on its tone (F0 fundamental frequency) features. After removing some samples with special voices, the tone features representing the majority of people in this population can be obtained.

[0118] (5) Finally, calculate the feature mean of the set without the specific samples, and you can get the reference audio feature.

[0119] When a patient is hospitalized, they usually enter their personal information. By finding the corresponding reference audio feature library through the personal information, the corresponding reference audio features can be obtained.

[0120] By performing a difference calculation between the reference audio feature and the target audio feature, a differentiated feature can be obtained. The difference calculation can be a difference calculation or a deviation rate calculation.

[0121] In a specific example, the feature is obtained by difference calculation. , ,in, is the target audio feature, is the reference audio feature;

[0122] In a specific example, the deviation rate is used to calculate the feature , .

[0123] The emotion recognition model in this application is obtained based on differential feature training. The specific training process includes:

[0124] 1. Find the relevant characteristics of different emotions; Figure 5 FIG. 1 is a schematic diagram of a process for determining target audio features in an embodiment of the present application. Figure 5 As shown, the process of determining the target audio features specifically includes:

[0125] S510, acquiring a speech data sample, and annotating the speech data sample to obtain a speech data sample with a label, wherein the speech data sample includes a reference speech segment in a calm state and a speech segment expressing emotion;

[0126] The reference speech segment and the emotion expression speech segment in the speech data sample are speech segments produced by the same person, representing data samples produced by the same person in different emotional states.

[0127] The annotated labels are the emotion types and the execution degrees of the emotion types, and the annotating process can be done manually.

[0128] S520, extracting reference audio features of the reference voice segment, extracting audio features of the emotion expression voice segment, and performing differential calculation based on the reference audio features of the reference voice segment and the audio features of the emotion expression voice segment to obtain differential sample features;

[0129] The process of extracting audio features please refer to the target audio feature extraction process mentioned above. The difference is that more features will be extracted here, such as loudness, loudness change characteristics, zero-crossing rate, etc.

[0130] Please refer to the previous article for the calculation of differentiated sample features, which will not be repeated here.

[0131] S530, classifying the voice data samples based on the labels to obtain multiple emotion data sets;

[0132] Among them, the basis of classification is the emotion label, which is the emotion type. According to different emotion types, multiple emotion data sets are obtained. Emotion labels include: happiness, sadness, pain, anger, doubt, calmness, surprise, etc.

[0133] S540, performing normalization processing on each difference sample feature in each emotion data set to obtain normalized data of each difference sample feature in each emotion data set;

[0134] The mathematical expression of normalized data is:

[0135]

[0136] In the formula, is the normalized differential feature value, Features The maximum eigenvalue of Features The minimum eigenvalue of .

[0137] The normalized data can transform each feature to \left [ {0,1} \right ] In the subsequent calculation of variance, the same variance standard can be used to measure the degree of aggregation of different feature types.

[0138] S550, calculating the variance of the normalized data of each difference sample feature in each emotion data set, and taking the difference sample feature whose variance is less than a preset variance threshold as the target difference feature corresponding to the emotion label; and taking the audio feature corresponding to the target difference feature as the target audio feature.

[0139] When the variance is small, it means that the values ​​of the difference sample features are relatively stable and clustered under the corresponding emotion. This means that the difference sample features are features that are likely to be generated under the corresponding emotion. For example, under the angry emotion, the speech speed will be concentrated in a relatively fast range, which means that under the angry emotion, the speech speed is likely to be accelerated. At this time, the speech speed is the target audio feature of the angry emotion.

[0140] Using label-related parameters to train the model can improve training efficiency and obtain more accurate training results. When the model is subsequently input for recognition, the target features also need to be input into the model for prediction.

[0141] Second, use the differentiated features of related feature types to train the model. The specific training process includes:

[0142] Constructing a training data set based on the target difference feature and the label of the target difference feature;

[0143] Based on the training data set, the artificial neural network is trained in combination with the gradient descent method to obtain an emotion recognition model.

[0144] This application uses RNN convolutional neural network for training. During training, the model parameters are adjusted by calculating the residual between the model prediction result and the label until the training is completed. The training model is a prior art and will not be described in detail here.

[0145] Using the emotion recognition model constructed by the above process and the current differentiation features, the emotion type and the confidence level of the emotion type can be identified. For example, "anger: 80%, doubt: 15%, pain: 5%";

[0146] Finally, the emotion entities extracted in the previous article are used to verify the emotion prediction results, and more accurate emotion recognition results are obtained, including:

[0147] (1) Determining multiple emotion categories in the emotion prediction results And the confidence of multiple emotion categories ; For example, "Anger: 80%, Doubt: 15%, Pain: 5%";

[0148] (2) When the key entity does not contain an emotion entity, the emotion type with the highest confidence is used as the emotion analysis result;

[0149] (3) The key entity includes an emotion entity, and multiple emotion categories corresponding to the emotion entity are determined based on a pre-constructed emotion entity-emotion category relationship map. And a variety of emotions Probability ; Confidence of the same emotion category and probability The weighted sum is taken to obtain the evaluation score; and the emotion type with the highest evaluation score is taken as the emotion analysis result.

[0150] Evaluation score The is: ; The confidence has been normalized in advance and can be regarded as another probability.

[0151] If the key entities include emotional entities, then it is necessary to determine which emotion the emotional entity belongs to, so as to support the emotional prediction results. Generally speaking, emotional entities may not belong to only one emotion, for example, "ah" may be anger or surprise, and "eh" may be sadness or confusion. Therefore, this application uses the constructed emotional entity-emotion type relationship map to determine the emotion type represented by each common emotional entity, as well as the probability of the represented emotion type. In order to support the emotion classification results, more accurate emotion analysis results can be obtained.

[0152] In one embodiment of the present application, a method for constructing an emotion entity-emotion category relationship graph includes:

[0153] (1) obtaining a text sample with an emotion entity and annotating the text sample to obtain a text sample with a label, wherein the text sample is extracted from a medical communication record;

[0154] The text samples extracted from medical communication records are more in line with the scenarios in this application, and the extracted emotional entities are also likely to appear in the scenarios in this application.

[0155] (2) Classify text samples based on emotional entities to obtain sample units of multiple emotional entities;

[0156] (3) Count the emotion category labels in the sample unit of the emotion entity and calculate the probability of each emotion label in the sample unit; the probability of each emotion label in the sample unit represents the probability that the emotion entity represents a certain emotion when the emotion entity appears.

[0157] (4) Construct an emotion entity-emotion category relationship graph based on the probabilities of multiple emotion entities and multiple emotion labels of multiple emotion entities.

[0158] The interaction module 240 is used to forward the request voice information to the target object when the emotion prediction result is abnormal;

[0159] When the emotion prediction result is not abnormal and the key entity is a request entity or a symptom entity, forwarding the request voice information to a target object;

[0160] When the emotion prediction result is not abnormal and the key entity is a query entity, the interaction is performed based on the text and a pre-built question and answer template, or the interaction is performed based on the text and a called language macro model.

[0161] In this application, anger and pain emotions are used as abnormal labels.

[0162] When the user is in an abnormal mood, in order to avoid exacerbating the conflict, regardless of whether the user's intention is to inquire, the call needs to be transferred to the on-duty personnel. When the user is in a normal mood, the interaction needs to be performed based on the user's intention. That is, when the user's intention is to inquire, the interaction can be performed through the question-and-answer template and the call to the language model. If the user's intention is to request, the call needs to be transferred to the on-duty personnel to respond to the user's request.

[0163] The above process can effectively filter out some unnecessary transfers and reduce the workload of medical staff or care workers. It can also avoid the problem of users becoming emotionally agitated and intensifying conflicts when users are in abnormal emotions and interacting with them using question and answer templates and large language models.

[0164] In one embodiment of the present application, it is impossible to use a large language model to cover all questions, because some users' questions are about the ward or the hospital itself, such as "How to adjust the bed backrest" and "Who should I contact to change the quilt". Such questions require setting up corresponding question and answer templates to deal with them.

[0165] The method for constructing the question and answer template is as follows:

[0166] (1) Obtain hospitalization-related and medical-related knowledge corpus samples;

[0167] (2) extracting triples from the knowledge corpus sample, wherein the triples include a head entity-relation-tail entity;

[0168] For example, extract triples from the following residue:

[0169] "In modern medical environments, hospital beds are designed not only to provide a basic resting place, but also to take into account the patient's comfort, safety and convenience of care. A key feature of hospital beds is that their backrest can be adjusted electrically or manually, either at the foot or side of the bed to meet the needs of different patients. This adjustable backrest design allows medical staff to adjust the height of the head of the bed according to the patient's treatment needs and personal preferences, thereby promoting better blood circulation, respiratory and digestive functions, and reducing the discomfort caused by long-term bed rest.

[0170] Through a simple control panel or remote control, medical staff can easily adjust the backrest to multiple preset angles or make fine adjustments to achieve the most suitable position. This function is especially important for patients who need to maintain a semi-recumbent or sitting position for a long time. In addition, some advanced beds are equipped with a memory function that can save specific angle settings so that you can quickly return to the previously used posture.

[0171] In addition to improving the patient experience, an adjustable backrest can also help prevent complications such as bedsores. It allows patients to change their positions without being disturbed, dispersing body pressure points, while facilitating nurses to perform daily care tasks such as changing dressings and turning the bed over. Therefore, how to adjust the bed's backrest has become one of the important factors in hospital purchasing decisions, and also reflects the medical institution's commitment to patient care and service quality. "

[0172] The corresponding extracted triples include:

[0173] (Hospital beds, design considerations, patient comfort)

[0174] (Hospital beds, design considerations, safety)

[0175] (Hospital beds, design considerations, convenience of care)

[0176] (Backrest, yes, electrically adjustable)

[0177] (Backrest, yes, manually adjustable)

[0178] (backrest, adjustable position, foot of bed)

[0179] (backrest, adjustable position, bed side)

[0180] (backrest, satisfaction, therapeutic needs)

[0181] (backrest, satisfaction, personal preference)

[0182] (backrest, promote blood circulation)

[0183] (backrest, boost, breathing function)

[0184] (backrest, promotion, digestive function)

[0185] (backrest, reduction, long-term bed rest discomfort)

[0186] (Control panel, enable, adjust backrest)

[0187] (Remote control, enable, adjust backrest)

[0188] (Bed, allows fine adjustment of backrest angle)

[0189] (bed, support, multiple preset angles)

[0190] (Advanced hospital bed, equipped with memory function)

[0191] (Memory function, save, angle setting)

[0192] (Backrest adjustment, important for patients who maintain semi-recumbent position for a long time)

[0193] (Backrest adjustment, important for patients who sit for a long time)

[0194] (Backrest adjustment, prevention, bedsores)

[0195] (backrest adjustment, distribution, body pressure points)

[0196] (Backrest adjustment, convenient for daily care work)

[0197] (backrest adjustment, reflection, medical care)

[0198] (backrest adjustment, reflection, service quality commitment)

[0199] The above triples have some unnecessary beneficial effect-related content, so they need to be manually removed to keep only the useful information.

[0200] (3) constructing a knowledge graph with the head entity or the tail entity as a node;

[0201] (4) Extracting multiple triples from the knowledge graph, constructing a question template based on two of the triples, and constructing an answer template based on the other one of the triples, to obtain a question-answer template.

[0202] For example, a question-answer template is constructed based on the following triples;

[0203] (Backrest, yes, electrically adjustable)

[0204] (Backrest, yes, manually adjustable)

[0205] (backrest, adjustable position, foot of bed)

[0206] (backrest, adjustable position, bed side)

[0207] Question template: "How do I adjust the angle of the bed backrest?"

[0208] Answer template: "The bed can be adjusted manually or electrically"

[0209] Question template: "Can the backrest of the hospital bed be electrically adjusted?"

[0210] Answer template: "Yes"

[0211] Question template: "Where is the adjustment position for the bed backrest?"

[0212] Answer template: "At the end of the bed or at the side of the bed"

[0213] The above question and answer templates are pre-recorded in the library in the form of vectors. When performing interaction, the corresponding answers can be output by matching the questions.

[0214] The specific interaction process includes:

[0215] (1) extracting keywords from the text;

[0216] (2) matching the keyword with the triples in the knowledge graph;

[0217] (3) When the keyword matches two of the triples in the knowledge graph, output the corresponding answer template; when the keyword does not match two of the triples in any of the knowledge graph, transfer the text to the language model for interaction.

[0218] When matching, in order to improve the matching speed, the matching is still carried out from the knowledge graph. If a match cannot be made, it is transferred to the language model for interaction. This can improve the flexibility of interaction and improve the user experience.

[0219] A medical information interaction terminal based on AI analysis of the present invention obtains the request voice information of the inpatient or family members when the inpatient or family members have a request or inquiry need. By extracting the target audio features and text of the request voice information, and then performing differential calculation on the target audio features and the reference audio features, the emotional recognition influence caused by the personal voice characteristics is eliminated. Then, the AI ​​model is used to perform emotional analysis on the reference audio features to obtain the emotional analysis results. Combined with the emotional entities extracted from the text, accurate emotional recognition results can be obtained. Then, based on the emotional analysis results and the extracted key entities, it is determined which interaction method to adopt. Therefore, when the patient or family members are emotionally stable and just want to ask some questions, a question and answer template or a large language model is used to respond. When the patient or family members are emotionally unstable or make a request, the request voice information is forwarded to the corresponding medical staff. While helping medical staff to filter out some non-critical information, the emotions of patients or family members are also taken into account to avoid stimulating serious medical conflicts.

[0220] like Figure 6 As shown, the present application also provides a medical information interaction method based on AI analysis, comprising the steps of:

[0221] S610, upon receiving a request instruction from a user, collecting request voice information from the user;

[0222] S620, converting the request voice information into text, extracting target audio features from the request voice information; and performing entity extraction on the text based on a pre-built key entity library to obtain key entities, wherein the key entity library includes a request entity, an inquiry entity, a symptom entity, and an emotion entity, and the type of the target audio feature is predetermined;

[0223] S630, determining the user's reference audio features based on the user's personal information and a pre-built reference audio feature library, or extracting the reference audio features from the user's sampled audio; performing differential calculation based on the reference audio features and the target audio features to obtain differential features; and inputting the differential features into a pre-built emotion recognition model to obtain the user's emotion prediction results; and determining the emotion analysis results based on the emotion prediction results and key entities;

[0224] S640, when the emotion prediction result is abnormal, forward the request voice information to the target object; when the emotion prediction result is not abnormal and the key entity is a request entity or a symptom entity, forward the request voice information to the target object; when the emotion prediction result is not abnormal and the key entity is an inquiry entity, perform interaction based on the text and a pre-built question and answer template, or perform interaction based on the text and the called language model.

[0225] A medical information interaction method based on AI analysis of the present invention obtains the request voice information of the inpatient or family members when the inpatient or family members have a request or inquiry need. By extracting the target audio features and text of the request voice information, and then performing differential calculation on the target audio features and the reference audio features, the emotional recognition influence caused by the personal voice characteristics is eliminated. Then, the AI ​​model is used to perform emotional analysis on the reference audio features to obtain the emotional analysis results. Combined with the emotional entities extracted from the text, accurate emotional recognition results can be obtained. Then, based on the emotional analysis results and the extracted key entities, it is determined which interaction method to adopt. Therefore, when the patient or family members are emotionally stable and just want to ask some questions, a question and answer template or a large language model is used to respond. When the patient or family members are emotionally unstable or make a request, the request voice information is forwarded to the corresponding medical staff. While helping medical staff to filter out some non-critical information, the emotions of patients or family members are also taken into account to avoid stimulating serious medical conflicts.

[0226] This embodiment also provides an electronic terminal, including: a processor and a memory;

[0227] The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes any one of the methods in this embodiment.

[0228] The computer-readable storage medium in this embodiment can be understood by ordinary technicians in this field: all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0229] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store computer programs, the communication interface is used to communicate, and the processor and the transceiver are used to run computer programs so that the electronic terminal executes each step of the above method.

[0230] In this embodiment, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0231] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0232] In the above-mentioned embodiments, although the present invention has been described in conjunction with the specific embodiments of the present invention, many replacements, modifications and variations of these embodiments will be apparent to those of ordinary skill in the art based on the foregoing description. The embodiments of the present invention are intended to cover all such replacements, modifications and variations falling within the broad scope of the appended claims.

[0233] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A medical information interaction terminal based on AI analysis, characterized in that: include: The audio collection module is used to collect the request voice information from the user when receiving the request instruction from the user; A feature extraction module, used to convert the request voice information into text and extract target audio features in the request voice information; and performing entity extraction on the text based on a pre-built key entity library to obtain key entities, wherein the key entity library includes request entities, inquiry entities, symptom entities and emotion entities, and the type of the target audio feature is predetermined; The emotion analysis module is used to determine the user's reference audio features based on the user's personal information and a pre-built reference audio feature library, or to extract reference audio features from the user's sampled audio; and to perform differential calculation based on the reference audio features and the target audio features to obtain differential features; and to input the differential features into a pre-built emotion recognition model to obtain the user's emotion prediction results; and to determine the emotion analysis results based on the emotion prediction results and key entities, wherein the reference audio features are features of the user's speaking audio samples in a peaceful state; and to determine the emotion analysis results based on the emotion prediction results and key entities, including: determining multiple emotion categories in the emotion prediction results And the confidence of multiple emotion categories ; When the key entity does not contain an emotion entity, the emotion type with the highest confidence is used as the emotion analysis result; When the key entity contains an emotion entity, multiple emotion types corresponding to the emotion entity are determined based on a pre-constructed emotion entity-emotion type relationship map And a variety of emotions Probability ; Confidence of the same emotion category and probability The weighted sum is taken to obtain the evaluation score; the emotion type with the highest evaluation score is taken as the emotion analysis result; An interaction module is used to forward the request voice information to a target object when the sentiment analysis result is abnormal; forward the request voice information to a target object when the sentiment analysis result is not abnormal and the key entity is a request entity or a symptom entity; and perform interaction based on the text and a pre-built question and answer template when the sentiment analysis result is not abnormal and the key entity is an inquiry entity, or perform interaction based on the text and a called large language model.

2. A medical information interaction terminal based on AI analysis according to claim 1, characterized in that: Extracting target audio features from the request voice information includes: Performing noise reduction and pre-emphasis on the request voice information to obtain pre-processed voice information; Converting the preprocessed voice information into PCM data, extracting spectrum data of a plurality of voice frames in the preprocessed voice information, and extracting Mel-frequency cepstral coefficients of a plurality of voice frames in the preprocessed voice information; Extracting speech segments and non-speech segments from the PCM data, and extracting the timestamp of each word in the speech segment; calculating the speech rate based on the timestamp of each word and the number of words, and calculating the pause ratio based on the speech segment and the non-speech segment; Extracting fundamental frequencies of the plurality of speech frames from the spectrum data of the plurality of speech frames, calculating average values ​​and variances of the fundamental frequencies of the plurality of speech frames, and obtaining fundamental frequency mean values ​​and fundamental frequency variances; Extracting first N mel-frequency cepstral values ​​from the mel-frequency cepstral coefficients of the multiple speech frames, and calculating the variance of all mel-frequency cepstral values ​​with the same bit order to obtain a flatness characteristic coefficient; The target audio feature is constructed based on the speaking rate, the pause ratio, the fundamental frequency mean, the fundamental frequency variance and the flatness characteristic coefficient.

3. A medical information interaction terminal based on AI analysis according to claim 1, characterized in that: The process of determining the type of the target audio feature includes: Acquire a speech data sample, and annotate the speech data sample to obtain a labeled speech data sample, wherein the speech data sample includes a reference speech segment in a calm state and an emotion-expressing speech segment; Extracting reference audio features of the reference speech segment, extracting audio features of the emotion expression speech segment, and performing differential calculation based on the reference audio features of the reference speech segment and the audio features of the emotion expression speech segment to obtain differential sample features; Classifying the speech data samples based on the labels to obtain multiple emotion data sets; Normalizing each difference sample feature in each emotion data set to obtain normalized data of each difference sample feature in each emotion data set; The variance of the normalized data of each difference sample feature in each emotion data set is calculated, and the difference sample feature whose variance is less than a preset variance threshold is used as the target difference feature of the corresponding emotion label; and the audio feature corresponding to the target difference feature is used as the target audio feature.

4. A medical information interaction terminal based on AI analysis according to claim 1, characterized in that: Entity extraction is performed on the text based on a pre-built key entity library to obtain key entities, including: Performing word segmentation processing on the text to obtain multiple words; Converting a plurality of words into word vectors, and matching the word vectors with a request entity template, an inquiry entity template, a symptom entity template, and an emotion entity template in the key entity library, wherein the request entity template, the inquiry entity template, the symptom entity template, and the emotion entity template are all stored in the key entity library in the form of vectors; When any word vector matches the request entity template, the inquiry entity template, the symptom entity template or the emotion entity template, the word vector is used as a key entity, and the type of the key entity is determined based on the matching result.

5. A medical information interaction terminal based on AI analysis according to claim 4, characterized in that: The method for constructing the question and answer template is as follows: Obtain hospitalization-related and medical-related knowledge corpus samples; Extracting triples from the knowledge corpus sample, wherein the triples include a head entity-a relation-a tail entity; Constructing a knowledge graph with the head entity or the tail entity as a node; A plurality of triples are extracted from the knowledge graph, and a question template is constructed using two of the triples, and an answer template is constructed based on the other one of the triples, so as to obtain a question-answer template.

6. A medical information interaction terminal based on AI analysis according to claim 1, characterized in that: The method for constructing the emotion entity-emotion category relationship graph includes: Acquire text samples with emotion entities, and annotate the text samples to obtain text samples with labels, wherein the text samples are extracted from medical communication records; Classify text samples based on emotional entities to obtain sample units of multiple emotional entities; Count the emotion category labels in the sample unit of the emotion entity, and calculate the probability of each emotion label in the sample unit; An emotion entity-emotion category relationship graph is constructed based on the probabilities of multiple emotion entities and multiple emotion labels of multiple emotion entities.

7. The medical information interaction terminal based on AI analysis according to claim 5, characterized in that: Performing interaction based on the text and a pre-built question-answer template, or performing interaction based on the text and a called large language model, including: extracting keywords from the text; Matching the keywords with triples in the knowledge graph; When the keyword matches two of the triples in the knowledge graph, the corresponding answer template is output; when the keyword does not match two of any triples in the knowledge graph, the text is transferred to the large language model for interaction.

8. The medical information interaction terminal based on AI analysis according to claim 3 is characterized in that: The method for constructing the emotion recognition model includes: Constructing a training data set based on the target difference feature and the label of the target difference feature; Based on the training data set, the artificial neural network is trained in combination with the gradient descent method to obtain an emotion recognition model.

9. A medical information interaction method based on AI analysis, characterized in that: Includes steps: When receiving a request instruction from a user, collecting request voice information from the user; Converting the request voice information into text and extracting target audio features from the request voice information; and performing entity extraction on the text based on a pre-built key entity library to obtain key entities, wherein the key entity library includes request entities, inquiry entities, symptom entities and emotion entities, and the type of the target audio feature is predetermined; Determine the user's reference audio features based on the user's personal information and a pre-built reference audio feature library, or extract the reference audio features from the user's sampled audio; perform differential calculation based on the reference audio features and the target audio features to obtain differential features; and input the differential features into a pre-built emotion recognition model to obtain the user's emotion prediction results; and determine the emotion analysis results based on the emotion prediction results and key entities, wherein the reference audio features are features of the user's speaking audio samples in a peaceful state; determine the emotion analysis results based on the emotion prediction results and key entities, including: determining multiple emotion categories in the emotion prediction results And the confidence of multiple emotion categories ; When the key entity does not contain an emotion entity, the emotion type with the highest confidence is used as the emotion analysis result; When the key entity contains an emotion entity, multiple emotion types corresponding to the emotion entity are determined based on a pre-constructed emotion entity-emotion type relationship map And a variety of emotions Probability ; Confidence of the same emotion category and probability The weighted sum is taken to obtain the evaluation score; the emotion type with the highest evaluation score is taken as the emotion analysis result; When the sentiment analysis result is abnormal, the request voice information is forwarded to the target object; when the sentiment analysis result is not abnormal and the key entity is a request entity or a symptom entity, the request voice information is forwarded to the target object; when the sentiment analysis result is not abnormal and the key entity is an inquiry entity, the interaction is performed based on the text and a pre-built question and answer template, or the interaction is performed based on the text and a called large language model.

Images