Emotion adaptive multi-modal dialogue generation method and system based on biological signals

Abstract

The invention provides an emotional adaptive multi-modal dialogue generation method and system based on biological signals, and relates to the technical field of data processing, the method comprises the following steps: step 1, processing collected biological signals and multi-modal interaction information, and extracting video spatial-temporal features, audio time sequence features and text semantic features; step 2, performing cross-modal fusion on the video spatio-temporal features, the audio time sequence features and the text semantic features, obtaining a deep semantic relation among multi-modal data through fusion semantic analysis, and establishing a cross-modal semantic corresponding relation; the emotion consistency evaluation link is used for analyzing the consistency degree of different modes on emotion expression, and comprehensive emotion state expression is obtained. The emotion change of the user is sensed in real time, the dialogue strategy is adaptively adjusted, and the naturalness of dialogue interaction is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to an emotion adaptive multi-modal dialogue generation method and system based on biological signals. BACKGROUND

[0002] The dialogue generation technology based on multi-modal information can combine various input modalities such as text, speech and vision to generate responses, mostly using natural language processing models to analyze the semantic content of user input, and referring to the dialogue context to maintain topic coherence. However, the existing methods still have some limitations in multi-modal emotion perception and dynamic adaptability. For example, in dialogue scenarios involving emotional support, existing systems may mostly rely on explicit emotional cues in text and speech, while the use of biological signals reflecting the user's internal emotional state is relatively limited, which may lead to insufficient comprehensive perception of user emotional changes, especially in cases where emotional expression is not significant or there is cross-modal emotional inconsistency. In addition, although existing natural language generation models have certain context adaptation capabilities, they mostly lack dynamic tracking and adjustment mechanisms for continuous fluctuations in user emotional state during long-term interaction. SUMMARY

[0003] The technical problem to be solved by the present application is to provide an emotion adaptive multi-modal dialogue generation method and system based on biological signals, which can perceive user emotional changes in real time and adaptively adjust dialogue strategies to improve the naturalness of dialogue interaction.

[0004] To solve the above technical problems, the technical solution of the present application is as follows: In a first aspect, an emotion adaptive multi-modal dialogue generation method based on biological signals, the method comprising: Step 1, processing the collected biological signals and multi-modal interaction information to extract video spatio-temporal features, audio temporal features and text semantic features; Step 2, cross-modal fusion of video spatio-temporal features, audio temporal features and text semantic features, obtaining deep semantic relationship between multi-modal data through fusion semantic analysis, and establishing cross-modal semantic correspondence; using emotion consistency evaluation link to analyze the consistency degree of different modalities in emotional expression, and obtaining comprehensive emotional state representation; Step 3, according to the comprehensive emotional state representation, combining the context information of multi-turn dialogue, using natural language generation model to obtain dialogue text content consistent with the current emotional state of the user, and converting to voice output through voice conversion process; Step 4, in the process of voice output, the dynamic changes of the user biological signals and the voice signals are monitored in real time, the newly collected multi-modal data are organized as time sequence data streams, and an emotional state analysis benchmark is constructed; by detecting key turning points and extreme points in the time sequence data streams, key event points representing emotional state boundaries are extracted, and two emotional change modes are identified; Step 5, the two emotional change modes are analyzed, the fluctuation interval of the emotional state is determined in combination with the result of the fusion semantic analysis, and an emotional state reference instance is selected to construct an emotional state change trajectory; emotional state dynamic adjustment parameters are generated based on the emotional state change trajectory, the dialogue strategy and the generated content are adjusted in real time, and an emotional adaptive response is realized.

[0005] In a second aspect, the emotional adaptive multi-modal dialogue generation system based on biological signals comprises: The extraction module is configured to process the collected biological signals and multi-modal interaction information, extract video spatiotemporal features, audio time sequence features and text semantic features, and output the features to the fusion module. The fusion module is configured to perform cross-modal fusion on the video spatiotemporal features, the audio time sequence features and the text semantic features, obtain deep semantic connections between the multi-modal data through fusion semantic analysis, and establish cross-modal semantic correspondence; and analyze the consistency degree of different modalities in emotional expression through the emotional consistency evaluation link, and obtain a comprehensive emotional state representation. The generation module is configured to obtain dialogue text content consistent with the current emotional state of the user by using a natural language generation model according to the comprehensive emotional state representation and in combination with the context information of the multi-round dialogue, and convert the dialogue text content into voice output through a voice conversion process. The recognition module is configured to monitor the dynamic changes of the user biological signals and the voice signals in real time in the process of voice output, organize the newly collected multi-modal data as time sequence data streams, and construct an emotional state analysis benchmark; by detecting key turning points and extreme points in the time sequence data streams, key event points representing emotional state boundaries are extracted, and two emotional change modes are identified. The adjustment module is configured to analyze the two emotional change modes, determine the fluctuation interval of the emotional state in combination with the result of the fusion semantic analysis, and select an emotional state reference instance to construct an emotional state change trajectory; emotional state dynamic adjustment parameters are generated based on the emotional state change trajectory, the dialogue strategy and the generated content are adjusted in real time, and an emotional adaptive response is realized.

[0006] The above scheme of the present application has at least the following beneficial effects: By synchronously processing biological signals and video, audio, text multi-modal interactive information, extracting video space-time features, audio time sequence features and text semantic features, combining the physiological objectivity of biological signals, reducing the emotional misjudgment caused by single dependence on language or behavior characteristics, making the emotion state recognition more in line with the real psychological state of the user, through cross-modal fusion and semantic analysis, the deep semantic corresponding relationship of multi-modal data is established, and a sentiment consistency evaluation link is additionally added, the consistency degree of different modalities in emotional expression is quantified, and the reliability of the comprehensive emotional state representation is ensured, based on the comprehensive emotional state representation and the multi-turn dialogue context information, the natural language generation model is used to generate dialogue text that fits the current emotion of the user, and the output is output through the voice conversion process, so that the dialogue response not only meets the semantic coherence, but also meets the emotional needs of the user. BRIEF DESCRIPTION OF DRAWINGS

[0007] Figure 1 is a flowchart of a method for generating an emotion adaptive multi-modal dialogue based on biological signals provided by an embodiment of the present application.

[0008] Figure 2 is a schematic diagram of a system for generating an emotion adaptive multi-modal dialogue based on biological signals provided by an embodiment of the present application. DETAILED DESCRIPTION

[0009] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be accurately conveyed to those skilled in the art.

[0010] As Figure 1 shown, an embodiment of the present application proposes a method for generating an emotion adaptive multi-modal dialogue based on biological signals, the method comprising the following steps: Step 1, processing the collected biological signals and multi-modal interactive information, extracting video space-time features, audio time sequence features and text semantic features; Step 2, cross-modal fusion of video space-time features, audio time sequence features and text semantic features, obtaining deep semantic relationship between multi-modal data through fusion semantic analysis, and establishing cross-modal semantic corresponding relationship; analyzing the consistency degree of different modalities in emotional expression by using a sentiment consistency evaluation link, obtaining a comprehensive emotional state representation; Step 3, according to the comprehensive emotional state representation, combining the context information of the multi-turn dialogue, using a natural language generation model to obtain dialogue text content consistent with the current emotional state of the user, and converting to voice output through a voice conversion process; Step 4, during the voice output process, the dynamic changes of the user biological signals and voice signals are monitored in real time, the newly collected multi-modal data are organized as time series data flow, and an emotional state analysis benchmark is constructed; by detecting key turning points and extreme points in the time series data flow, key event points representing the boundaries of the emotional state are extracted, and two emotional change patterns are identified; Step 5, the two emotional change patterns are analyzed, the fluctuation interval of the emotional state is determined in combination with the result of the semantic analysis, and an emotional state reference instance is selected to construct an emotional state change trajectory; based on the emotional state change trajectory, emotional state dynamic adjustment parameters are generated to adjust the dialogue strategy and generated content in real time, and an emotional self-adaptive response is realized.

[0011] In the embodiment of the present application, by synchronously processing biological signals and video, audio, text multi-modal interactive information, video space-time features, audio time sequence features and text semantic features are extracted, combined with the physiological objectivity of biological signals, the emotional misjudgment caused by single dependence on language or behavior features is reduced, the emotional state recognition is more in line with the real psychological state of the user, the deep semantic corresponding relationship of multi-modal data is established through cross-modal fusion and semantic analysis, and a sentiment consistency evaluation link is additionally added, the consistency degree of different modalities in emotional expression is quantified, the reliability of the comprehensive emotional state representation is ensured, based on the comprehensive emotional state representation and the multi-round dialogue context information, the dialogue text that meets the current emotion of the user is generated by using the natural language generation model, and the output is realized through the voice conversion process, so that the dialogue response not only meets the semantic coherence, but also meets the emotional needs of the user.

[0012] In a preferred embodiment of the present application, the above step 1, processing the collected biological signals and multi-modal interactive information to extract video space-time features, audio time sequence features and text semantic features, can include: In the embodiment of the present application, non-invasive sensing equipment is used to synchronously collect user brain electrical signals, electrocardio signals and skin electrical response signals, and time stamps are recorded to ensure time alignment with multi-modal information. Each signal processing and feature extraction is as follows: after brain electrical signals are subjected to 5Hz-30Hz band pass filtering, 50 sample point sliding average denoising, 1 second segmentation, segment mean value, variance, alpha wave (8Hz-13Hz) and beta wave (14Hz-30Hz) energy proportion are extracted; electrocardio signals are subjected to R wave peak value identification by threshold method and interference waveform is removed, and after normalization to 0-1 range, heart rate, 1 minute R wave number, R wave interval mean value and standard deviation are calculated; after skin electrical response signals are subjected to 20 sample point median filtering denoising, 1 second segmentation, segment peak value, rising rate and numerical total are extracted; video space-time feature extraction, video is uniformly standardized to 1920x1080 resolution, 30fps frame rate, face processing, 224x224 pixel face region is detected and cropped, and histogram equalization is used to enhance contrast; fuzzy frames (clarity < average 60%) and non-action frames (feature point displacement total < threshold) are removed, and effective frames are retained; static features, 68 face key feature points of each frame are located, 30 key distance parameters, 15 feature vector angle parameters and 100 10x10 pixel block texture parameters (mean value + variance) are calculated, and are integrated into 145 dimensional single frame space feature vector; dynamic features are based on 10 continuous frame space features, 9 inter-frame difference vectors are calculated, and average change rate, change amplitude and cumulative change amount of each parameter are extracted; space-time fusion, single frame space features and dynamic features are spliced, 580 dimensional video space-time feature vector is formed, and static face features and dynamic change information are contained; audio time sequence feature extraction, speech signal is uniformly standardized to 16kHz sampling rate, 16 bit depth single channel, high frequency signal is enhanced by pre-emphasis, is framed according to 20ms frame length and 10ms frame shift, each frame has 320 sample points, Hanning window is applied to reduce spectrum leakage, and mute frames with short time energy < average 10% are removed; feature extraction, time domain features, short time energy, zero crossing rate, fundamental frequency and frequency domain features, 256 frequency point spectrum amplitude and spectrum barycenter are extracted; time sequence integration, effective frame features are arranged in time sequence, and 260 dimensional / frame audio time sequence feature sequence is formed; text semantic feature extraction, text is uniformly coded as UTF-8, special symbols, numbers and punctuation marks are removed, and after word segmentation, part-of-speech tagging is performed, stop words are removed and wrong characters are corrected, and core semantic words are retained; sentiment and semantic features, based on Chinese sentiment dictionary, positive / negative sentiment word number is counted, total sentiment intensity and average sentiment intensity are calculated; context window is constructed, and core word overlap rate is calculated; the above indexes and core word semantic codes are integrated to form feature vectors; biological signal features, video space-time features, audio time sequence features and text semantic features are aligned according to time stamps to form a unified multi-modal feature set, to ensure time synchronization of each modal feature, and to comprehensively cover user internal physiological state, external visual expression, acoustic emotional clues and semantic sentiment tendency.

[0013] In a preferred embodiment of the present application, step 2, the video spatio-temporal features, audio temporal features and text semantic features are cross-modal fused, the deep semantic relationship between the multi-modal data is obtained through the fusion semantic analysis, and the cross-modal semantic corresponding relationship is established; the consistency of different modalities in emotional expression is analyzed by using the emotional consistency evaluation link, and the comprehensive emotional state representation is obtained, which can include: In the embodiment of the present application, step 220, the video spatio-temporal features, audio temporal features and text semantic features are cross-modal semantic aligned, and the interaction correlation strength between different modal features is analyzed through the multi-modal attention process; specifically, first, the video spatio-temporal features, audio temporal features and text semantic features are respectively labeled with basic semantic labels, and the semantic labels of each modality are divided into 5 categories of core semantic dimensions, wherein the semantic label dimensions of the video spatio-temporal features include facial expression, head posture, body movement, eye state and facial muscle movement, each category is subdivided into specific labels, such as facial expression including smile, frown, calm, stick out tongue and surprise; the semantic label dimensions of the audio temporal features include tone change, speech rhythm, volume strength, tone fluctuation and audio rhythm, and the subdivided labels include tone change including rising, falling, smooth, fluctuation, etc.; the semantic label dimensions of the text semantic features include lexical emotion, sentence tone, semantic tendency, expression intention and emotional intensity, and the subdivided labels include lexical emotion including positive, neutral, negative, praise and derogation, etc.

[0014] A cross-modal semantic alignment mapping table is established, taking emotional expression as the core anchor point, and labels with the same or highly similar semantic meanings in the three modalities are paired and grouped to form semantic groups. For example, the first semantic group includes video facial expressions and smiles, head poses and nods, audio tone changes and rising, speech rhythm and light, text lexical emotion and positive, and semantic tendency and positive. The second semantic group includes video facial expressions and frowns, eye states and tension, audio tone changes and low, speech tone and harsh, text lexical emotion and negative, and emotional intensity and strong. In this way, each semantic group is constructed around the same core emotion or semantic meaning, achieving preliminary semantic alignment. For each semantic group, the feature segments corresponding to the three modalities in the group are extracted, and the semantic details contained in each feature segment are analyzed one by one. The number of matching details between each modality feature segment and the core meaning of the semantic group is counted. For example, the core meaning of the first semantic group is positive emotional expression. The video feature segment contains two matching details: smiles and nods. The audio feature segment contains two matching details: rising tone and light speech rhythm. The text feature segment contains two matching details: positive words and positive semantics. The total number of details for this semantic group is 6, which is the sum of the matching details of the three modalities. Then, the local correlation coefficients between the modalities in a single semantic group are calculated. The local correlation coefficient between the video modality and the audio modality is (video matching details + audio matching details) ÷ semantic group total details, i.e., (2+2) ÷ 6 ≈ 0.67. The local correlation coefficient between the video modality and the text modality is (2+2) ÷ 6 ≈ 0.67. The local correlation coefficient between the audio modality and the text modality is (2+2) ÷ 6 ≈ 0.67.

[0015] The local correlation coefficients of all semantic groups are summarized, and the average of the local correlation coefficients between each modality and the other two modalities is calculated, which is the interactive correlation strength between different modality features. Assuming that 10 semantic groups are constructed, the local correlation coefficients of video and audio in the 10 semantic groups are 0.67, 0.7, 0.65, 0.72, 0.68, 0.71, 0.66, 0.69, 0.73, and 0.64. The sum of these 10 values is 6.85, and the interactive correlation strength of video and audio is 0.685 when divided by the total number of semantic groups, which is 10. In the same way, the sum of the 10 local correlation coefficients of video and text is 6.78, and the interactive correlation strength is 0.678. The sum of the 10 local correlation coefficients of audio and text is 6.82, and the interactive correlation strength is 0.682.

[0016] In step 221, the modal features are fused based on the interaction correlation strength to generate a semantic alignment cross-modal joint representation; specifically including: first, determine the weight coefficient of each modal feature, the weight coefficient directly uses the average value of the interaction correlation strength obtained in step 220, that is, the weight coefficient of the video modality = (the interaction correlation strength between video and audio + the interaction correlation strength between video and text) ÷ 2, substituting the numerical calculation is (0.685+0.678) ÷ 2≈0.6815; the weight coefficient of the audio modality = (the interaction correlation strength between video and audio + the interaction correlation strength between audio and text) ÷ 2, that is, (0.685+0.682) ÷ 2≈0.6835; the weight coefficient of the text modality = (the interaction correlation strength between video and text + the interaction correlation strength between audio and text) ÷ 2, that is, (0.678+0.682) ÷ 2=0.68.

[0017] The video spatiotemporal feature, the audio time sequence feature and the text semantic feature are processed in dimension unification. It is assumed that the original dimension of the video spatiotemporal feature is 128, the audio time sequence feature is 64, and the text semantic feature is 256. The maximum dimension of 256 in the three is selected as the unified dimension. The video spatiotemporal feature and the audio time sequence feature are dimensionally supplemented. The video spatiotemporal feature is expanded from 128 to 256 by repeatedly filling the feature dimension with high semantic contribution. The audio time sequence feature is expanded from 64 to 256 by interpolation calculation to supplement the missing dimension of the feature value. The features of the three modalities are ensured to be 256, meeting the dimension requirement of fusion calculation. For each feature dimension, the feature values of the three modalities on the dimension are extracted, for example, in the first dimension, the feature value of the video modality is 0.35, the feature value of the audio modality is 0.42, and the feature value of the text modality is 0.38. In the second dimension, the feature value of the video modality is 0.29, the feature value of the audio modality is 0.36, and the feature value of the text modality is 0.41. Similarly, the feature values of the three modalities on the dimension are extracted. Then, the feature values of each dimension are weighted and calculated. The feature value of the video modality is multiplied by the weight coefficient 0.6815, the feature value of the audio modality is multiplied by the weight coefficient 0.6835, and the feature value of the text modality is multiplied by the weight coefficient 0.68. Taking the first dimension as an example, the weighted feature value of the video is 0.35×0.6815≈0.2385, the weighted feature value of the audio is 0.42×0.6835≈0.2871, and the weighted feature value of the text is 0.38×0.68≈0.2584. Then, the three weighted feature values are added to obtain the joint feature value of the dimension, which is 0.2385+0.2871+0.2584≈0.784. The joint feature values of the 256 feature dimensions are calculated in the above manner, and all the joint feature values are arranged in the order of the first dimension to the 256th dimension to form a 256-dimensional semantic alignment cross-modal joint representation.

[0018] At step 222, based on the semantic alignment cross-modal joint representation, a semantic correlation matrix between modalities is constructed by calculating the semantic similarity between different modal feature segments. Specifically, first, the video spatiotemporal features, audio temporal features and text semantic features are divided into segments according to a fixed time length. The time length of each segment is set to 2 seconds. Assuming that the total time length of the original multi-modal data is 20 seconds, the features of each modality are divided into 10 feature segments, each segment corresponds to a time interval of 2 seconds, and the segment time intervals of the three modalities are completely consistent, such as segment 1 corresponding to 0-2 seconds, segment 2 corresponding to 2-4 seconds, …, segment 10 corresponding to 18-20 seconds, realizing time alignment at the segment level. Taking the semantic group in the cross-modal joint representation as a reference, the core semantic keywords for each feature segment are determined. The specific way is to count the number of semantic group labels contained in each feature segment, select the semantic group with the most labels as the core semantic group of the segment, and then extract 2-3 key labels from the core semantic group as the core semantic keywords. For example, in video segment 3, the number of labels of the positive emotion expression semantic group is the most, so the core semantic keywords are smile, nod and positive; in the corresponding audio segment 3, the number of labels of the positive emotion expression semantic group is the most, and the core semantic keywords are pitch rise and light speed; in the corresponding text segment 3, the number of labels of the positive emotion expression semantic group is the most, and the core semantic keywords are positive vocabulary and positive semantics.

[0019] The semantic similarity of any two different modal feature segments is calculated, and the calculation rule is that the number of same core semantic keywords in the two feature segments is first counted, then the number of similar semantic keywords is counted, the number of same keywords is added to the result of the number of similar semantic keywords multiplied by 0.5, and then divided by the sum of the total number of keywords of the two feature segments, to obtain the semantic similarity of a single segment pair. Taking video segment 3 and audio segment 3 as an example, the number of same keywords is 0; the number of similar semantic keywords is 3; the total number of keywords of video segment 3 is 3, the total number of keywords of audio segment 3 is 2, and the sum of the total number of keywords is 5; then the semantic similarity of the two is (0+3x0.5)÷5=1.5÷5=0.3. Taking video segment 3 and text segment 3 as an example: the number of same keywords is 1; the number of similar semantic keywords is 2; the total number of video keywords is 3, the total number of text keywords is 2, and the sum is 5; the semantic similarity is (1+2x0.5)÷5=(1+1)÷5=0.4; a modal semantic association matrix is constructed, which is divided into three groups, a video and audio semantic association matrix, an audio and text semantic association matrix, and a video and text semantic association matrix. The number of rows and columns of each matrix is equal to the number of feature segments (10 rows and 10 columns), the row index of the matrix represents the sequence number of the feature segment of the first modal (1-10), and the column index represents the sequence number of the feature segment of the second modal (1-10). Each element in the matrix is the semantic similarity of the corresponding two segments, for example, in the video and audio semantic association matrix, the element in the 3rd row and the 3rd column is the semantic similarity of video segment 3 and audio segment 3, which is 0.3, and the element in the 5th row and the 5th column is the semantic similarity of video segment 5 and audio segment 5. Fill in all the matrix elements in turn to complete the construction of the three semantic association matrices.

[0020] Step 223, analyze the semantic association matrix, identify the cross-modal semantic association pattern, and establish the structured cross-modal semantic correspondence; specifically including: first, set the semantic similarity threshold, calculate the average value of all elements in all semantic association matrices, take the average value as the threshold, assume three groups of matrices, each group has 10x10=100 elements, the total sum of semantic similarity values of 300 elements is 120, then the average value = 120÷300 = 0.4, that is, set the semantic similarity threshold to 0.4, traverse each inter-modal semantic association matrix, and filter out elements in the matrix whose values are greater than the threshold 0.4, the feature segment pairs corresponding to these elements are highly correlated segment pairs, for example, in the video and text semantic association matrix, the element in the 2nd row and 2nd column is 0.5, which is greater than 0.4, so video segment 2 and text segment 2 are highly correlated segment pairs; the element in the 7th row and 7th column is 0.6, which is greater than 0.4, so video segment 7 and text segment 7 are highly correlated segment pairs. In this way, all highly correlated segment pairs in the three groups of matrices are summarized; analyze these highly correlated segment pairs, identify the cross-modal semantic association pattern, and set two core association patterns, one is the comprehensive matching type association pattern, the judgment standard is that the semantic similarity corresponding to a segment pair in all semantic groups is greater than the threshold 0.4, and the proportion of the number of semantic groups involved to the total number of semantic groups is not less than 80%, that is, the segment pair is highly matched in multiple dimensions of semantics; the second is the local matching type association pattern, the judgment standard is that the semantic similarity corresponding to a segment pair in only part of the semantic groups is greater than the threshold 0.4, and the proportion of the number of semantic groups involved to the total number of semantic groups is between 30%-80%, that is, the segment pair is highly matched in a specific semantic dimension; for example, the segment pair composed of video segment 5 and audio segment 5, text segment 5 has a similarity greater than 0.4 in 9 of the 10 semantic groups, accounting for 90%, so it is determined to be a comprehensive matching type; the segment pair composed of video segment 8 and audio segment 8 has a similarity greater than 0.4 in 4 of the 10 semantic groups, accounting for 40%, so it is determined to be a local matching type.

[0021] For each association mode, all highly relevant fragment pairs under the mode are summarized, and detailed information of each fragment pair is recorded, including the type of the first modality, the fragment sequence number, the core semantic keyword, the association mode type, the type of the other modality corresponding to the association, and the fragment sequence number, for example, a record under the full matching type is modality type-video, fragment sequence number-5, core semantic keyword-peaceful, relaxing, neutral, association mode-full matching type, corresponding association modality-audio fragment 5, text fragment 5; a record under the partial matching type is modality type-audio, fragment sequence number-8, core semantic keyword-slow speech speed, calm tone, association mode-partial matching type, corresponding association modality-video fragment 8, all records are arranged into a structured cross-modal semantic corresponding relationship table, the table contains fields, modality type, fragment sequence number, core semantic keyword, association mode, corresponding modality 1 type, corresponding modality 1 fragment sequence number, corresponding modality 2 type, and corresponding modality 2 fragment sequence number, through the table, the semantic corresponding relationship of all feature fragments among the three modalities is clearly presented.

[0022] Step 224, based on the cross-modal semantic correspondence relationship, corresponding visual emotional features, acoustic emotional features and semantic emotional features are extracted from the video modality, audio modality and text modality; Specifically, first, the emotion-related semantic groups are screened out from the cross-modal semantic correspondence table, and are divided into three categories according to the emotion category: positive emotion group, containing core semantic keywords such as joy, excitement, satisfaction and comfort; neutral emotion group, containing core semantic keywords such as calm, flat and no obvious emotion; negative emotion group, containing core semantic keywords such as sadness, anger, anxiety and dissatisfaction, and the feature segments corresponding to the three modalities of each emotional semantic group are determined, that is, the segment pairs in the correspondence table with the association mode of full matching type or local matching type; visual emotional features are extracted, and specific features are extracted from four dimensions of facial expression, head posture, eye state and facial muscle movement for the corresponding feature segments of the video modality; the facial expression dimension extracts the mouth corner lifting amplitude, which is divided into 0-5 levels, 0 level is no lifting, 5 level is maximum lifting, eyebrow shape state is classified into furrowed, relaxed and tense; the head posture dimension extracts the head tilt angle, which is classified into horizontal, left tilt, right tilt, low head, high head and nodding frequency, which is calculated in times per second; the eye state dimension extracts the eye brightness, which is divided into 0-5 levels, and the blinking frequency is calculated in times per second; the facial muscle movement dimension extracts the cheek muscle tightness, which is divided into 0-3 levels, and the forehead wrinkle number is classified into none, small amount and more, for example, the video segment of a certain positive emotion group, the extracted visual emotional features are mouth corner lifting amplitude 4 level, eyebrow shape relaxed, head horizontal, nodding frequency 1.5 times per second, eye brightness 4 level, blinking frequency 0.8 times per second, cheek muscle tightness 1 level, and forehead without wrinkles; acoustic emotional features are extracted, and specific parameters are extracted from four dimensions of tone change, speech rhythm, volume strength and tone, for the corresponding feature segments of the audio modality; the tone change dimension extracts the average tone, which is calculated in hertz, the range is 100-500 hertz, and the tone fluctuation range is calculated by the maximum tone and the minimum tone; the speech rhythm dimension extracts the speech rate, which is calculated in words per second, and the inter-sentence pause time is calculated in seconds; the volume strength dimension extracts the average volume, which is calculated in decibels, the range is 30-80 decibels, and the volume fluctuation amplitude is calculated by the maximum volume and the minimum volume; the tone dimension extracts the tone frequency, which is calculated in times per second, and the tone intensity is divided into 0-3 levels, for example, the audio segment of the above-mentioned positive emotion group, the extracted acoustic emotional features are average tone 250 hertz, tone fluctuation range 50 hertz, speech rate 3.2 words per second, inter-sentence pause time 0.2 seconds, average volume 60 decibels, volume fluctuation amplitude 10 decibels, tone frequency 0.5 times per second, and tone intensity 1 level.

[0023] The semantic sentiment features are extracted, and the corresponding feature fragments of the text mode are extracted from four dimensions of lexical sentiment, sentence style, sentiment intensity, and semantic tendency. The lexical sentiment dimension counts the number of positive words, the number of neutral words, and the number of negative words, and calculates the proportion of sentiment words (number of sentiment words / total number of words). The sentence style dimension judges the sentence type and the number of mood auxiliaries. The sentiment intensity dimension is divided into 0-1 levels, 0 level is no obvious emotion, 1 level is strong emotion, and is adjusted combined with the intensity of sentiment words. The semantic tendency dimension judges positive, neutral, and negative. For example, the text fragment of the positive emotion group above, the extracted semantic sentiment features are 2 positive words, 3 neutral words, 0 negative words, and the proportion of sentiment words is 40%. The sentence type is an exclamation sentence, and the number of mood auxiliaries is 1. The sentiment intensity is 0.9 level. The semantic tendency is positive. The extracted visual sentiment features, acoustic sentiment features, and semantic sentiment features are classified and stored according to the emotional semantic group. Three modal emotion feature sub-folders are established under each emotion group. The files in the folder are named in the format of segment number-feature type-feature details, ensuring that the three modal emotion features under each emotion category can be quickly associated and queried.

[0024] Step 225, discriminates the visual emotional features, acoustic emotional features and semantic emotional features to form the emotional tendency description of the video mode, the emotional tendency description of the audio mode and the emotional tendency description of the text mode; Specifically, first, a detailed emotional discrimination standard library is established, and the emotion is divided into three categories: positive, neutral and negative, and a clear multi-dimensional discrimination basis is set for each category of emotion, each basis includes specific numerical range or classification standard, the positive emotional discrimination basis: visual basis, need to meet ≥3 items: mouth corner up amplitude ≥3 levels, eyebrows relaxed, nodding frequency ≥1 times / second, eye brightness ≥4 levels, cheek muscle tightness degree ≤1 level; acoustic basis, need to meet ≥3 items: average tone 200-300 hertz, speech rate ≥2.5 words / second, average volume 50-70 decibels, pause frequency ≤0.8 times / second, pause intensity ≤1 level; text basis, need to meet ≥2 items: positive vocabulary proportion ≥30%, sentence type is exclamation sentence / statement sentence, emotional intensity ≥0.7 level, semantic tendency positive; neutral emotional discrimination basis: visual basis, need to meet ≥3 items: mouth corner up amplitude 2-3 levels, eyebrows are flat, nodding frequency 0.5-1 times / second, eye brightness 2-3 levels, cheek muscle tightness degree 1-2 levels; acoustic basis, need to meet ≥3 items: average tone 150-200 hertz or 300-350 hertz, speech rate 1.5-2.5 words / second, average volume 40-50 decibels or 70-75 decibels, pause frequency 0.8-1.2 times / second, pause intensity 1-2 levels; text basis, need to meet ≥2 items: positive vocabulary proportion 10%-30% and negative vocabulary proportion 10%-30%, sentence type is statement sentence, emotional intensity 0.3-0.7 level, semantic tendency neutral; negative emotional discrimination basis: visual basis, need to meet ≥3 items: mouth corner up amplitude ≤2 levels, eyebrows frown / tighten, nodding frequency ≤0.5 times / second, eye brightness ≤2 levels, cheek muscle tightness degree ≥2 levels; acoustic basis, need to meet ≥3 items, average tone ≤150 hertz or ≥350 hertz, speech rate ≤1.5 words / second, average volume ≤40 decibels or ≥75 decibels, pause frequency ≥1.2 times / second, pause intensity ≥2 levels; text basis, need to meet ≥2 items: negative vocabulary proportion ≥30%, sentence type is statement sentence / question sentence, emotional intensity ≥0.7 level (negative direction), semantic tendency negative.

[0025] Determine the sentiment tendency of the video modality, take out the sentiment features of the video modality in a certain sentiment group, and compare the visual determination basis of positive, neutral, and negative sentiment one by one. For example, the video features include a 4-level mouth upturn, relaxed eyebrows, a horizontal head, 4-level eye brightness, and 1-level cheek muscle tightness. A total of 4 positive sentiment basis, 1 neutral sentiment basis, and 0 negative sentiment basis are met. Since the number of positive sentiment basis is the largest and meets the requirement of ≥3, the sentiment tendency of the video modality is determined to be positive, forming a sentiment tendency description of the video modality, which tends to be positive emotion, determined based on visual features such as 4-level mouth upturn, relaxed eyebrows, 4-level eye brightness, and 1-level cheek muscle tightness. Determine the sentiment tendency of the audio modality in the same way, take out the corresponding sentiment features of the audio modality, compare the acoustic determination basis of the three types of sentiment, and count the number of matches. For example, the audio features include an average tone of 250 Hz, a speech rate of 3.2 words per second, an average volume of 60 decibels, a pause frequency of 0.5 times per second, and a pause intensity of level 1. A total of 5 positive sentiment basis, 1 neutral sentiment basis, and 0 negative sentiment basis are met. The sentiment tendency of the audio modality is determined to be positive, forming a sentiment tendency description of the audio modality, which tends to be positive emotion, determined based on acoustic features such as average tone of 250 Hz, speech rate of 3.2 words per second, average volume of 60 decibels, pause frequency of 0.5 times per second, and pause intensity of level 1. Determine the sentiment tendency of the text modality, take out the sentiment features of the corresponding text modality, compare the text determination basis of the three types of sentiment, and count the number of matches. For example, the text features include a positive vocabulary proportion of 40%, an exclamatory sentence, a sentiment intensity of level 0.9, and a semantic tendency of positive. A total of 4 positive sentiment basis, 1 neutral sentiment basis, and 0 negative sentiment basis are met. The sentiment tendency of the text modality is determined to be positive, forming a sentiment tendency description of the text modality, which tends to be positive emotion, determined based on text features such as positive vocabulary proportion of 40%, exclamatory sentence, sentiment intensity of level 0.9, and semantic tendency of positive. Organize the sentiment tendency description of each modality to ensure that the description clearly includes the sentiment tendency category and the core determination basis.

[0026] Step 226, cross comparison of the emotional tendency descriptions of the video, audio and text modalities is performed to evaluate the internal consistency between the emotional tendency descriptions of different modalities, and according to the internal consistency, a corresponding fusion contribution degree is determined for the emotional tendency description of each modality; specifically including: first, three groups of objects for cross comparison are determined, video modality and audio modality, audio modality and text modality, and text modality and video modality, the fit degree of the emotional tendency description is aggregated for each comparison, referring to the polygon internal angle and calculation logic, the three modalities are regarded as the three vertices of a triangle, the three comparison results correspond to the three internal angles of the triangle, the sum of the three internal angles of the triangle is fixed at 180°, so the total score of the consistency of the three comparisons is fixed at 3 points, each comparison is scored 1 point, the sum of the three internal angle scores is 3 points, which ensures the rigor and fairness of the evaluation logic, at the same time, the consistency scoring rules are formulated, if the emotional tendency descriptions of two groups of modalities are exactly the same, i.e. the emotional categories are consistent and the intensity deviation is ≤0.1, such as both being positive emotion tendency, intensity level 0.9, then this group of comparison is scored 1 point, corresponding to the internal angle of the triangle 60°, the angle is balanced, and the fit degree is the highest; if the emotional tendency descriptions of two groups of modalities are similar, i.e. the emotional categories are consistent but the intensity deviation is 0.1-0.3, or the emotional categories are adjacent and the intensity deviation is ≤0.2, such as one group being positive emotion tendency, intensity level 0.9, and the other group being positive emotion tendency, intensity level 0.7; or one group being neutral positive emotion tendency, intensity level 0.6, and the other group being positive emotion tendency, intensity level 0.7, then this group of comparison is scored 0.5 points, corresponding to the internal angle of the triangle 45°, the angle is relatively balanced, and the fit degree is medium; if the emotional tendency descriptions of two groups of modalities are completely different, i.e. the emotional categories are not adjacent or the intensity deviation is >0.3, such as one group being positive emotion tendency and the other group being negative emotion tendency; or one group being positive emotion tendency, intensity level 0.9, and the other group being positive emotion tendency, intensity level 0.5, then this group of comparison is scored 0 points, corresponding to the internal angle of the triangle 30°, the angle is unbalanced, and the fit degree is the lowest.

[0027] The consistency scores of the three groups of comparisons are calculated in turn according to the above rules, assuming that the emotional tendency of the three modalities of this comparison is described as follows: video modality, positive emotional tendency, intensity level 0.9; audio modality, positive emotional tendency, intensity level 0.8; text modality, positive emotional tendency, intensity level 0.7; the first group of video and audio, consistent in emotional category, intensity deviation 0.1, meets the similar standard, and gets 0.5 points; the second group of audio and text, consistent in emotional category, intensity deviation 0.1, meets the similar standard, and gets 0.5 points; the third group of text and video, consistent in emotional category, intensity deviation 0.2, meets the similar standard, and gets 0.5 points; the total score of the three groups of comparisons = 0.5 + 0.5 + 0.5 = 1.5 points, corresponding to the total sum of the three angles of the triangle 180°, the total score meets the distribution logic of the fixed value 3 points; the fusion contribution degree of each modality is calculated, the core logic of the fusion contribution degree is that the contribution degree of each modality is positively related to the sum of the scores of the two groups of comparisons involving the modality, and each comparison score in the total score involves two modalities, so it needs to be divided by twice the total score to ensure that the sum of the contribution degrees of the three modalities is 1, the specific calculation steps are as follows, the two groups of comparisons involving the video modality, video and audio 0.5 points, text and video 0.5 points, the sum of the scores of the two groups = 0.5 + 0.5 = 1 point; the two groups of comparisons involving the audio modality, video and audio 0.5 points, audio and text 0.5 points, the sum of the scores of the two groups = 0.5 + 0.5 = 1 point; the two groups of comparisons involving the text modality, audio and text 0.5 points, text and video 0.5 points, the sum of the scores of the two groups = 0.5 + 0.5 = 1 point; twice the total score = 1.5 x 2 = 3 points; the fusion contribution degree of the video modality = the sum of the scores of the two groups involving the video ÷ twice the total score = 1 ÷ 3 ≈ 0.333; the fusion contribution degree of the audio modality = the sum of the scores of the two groups involving the audio ÷ twice the total score = 1 ÷ 3 ≈ 0.333; the fusion contribution degree of the text modality = the sum of the scores of the two groups involving the text ÷ twice the total score = 1 ÷ 3 ≈ 0.334; finally, the sum of the fusion contribution degrees of the three modalities is verified, 0.333 + 0.333 + 0.334 = 1, which meets the basic requirement of contribution degree distribution, if there is a slight deviation, the contribution degree of one of the modalities is adjusted slightly to ensure that the total sum of the contribution degrees is strictly 1, which can objectively reflect the reliability and fit degree of the emotional description of each modality.

[0028] Step 227, according to the fusion contribution degree, the sentiment tendency description of each modality is proportionally fused to form a preliminary comprehensive emotion representation, and the preliminary comprehensive emotion representation is normalized to obtain a final comprehensive emotion state representation; Specifically, first, the sentiment tendency description of each modality is converted into a quantifiable sentiment value, and the conversion rule is based on the sentiment category and intensity level. The basic value of positive emotion is 1, the basic value of neutral emotion is 0.5, and the basic value of negative emotion is 0; The intensity level is converted into a correction coefficient according to 0-1 level, that is, the intensity level is directly used as the correction coefficient, such as intensity 0.9 level corresponding to correction coefficient 0.9; The final sentiment value = basic value x correction coefficient, for example, video modality, sentiment category positive basic value 1, intensity 0.9 level, sentiment value = 1 x 0.9 = 0.9; Audio modality, sentiment category positive basic value 1, intensity 0.8 level, sentiment value = 1 x 0.8 = 0.8; Text modality, sentiment category positive basic value 1, intensity 0.7 level, sentiment value = 1 x 0.7 = 0.7; For mixed sentiment tendency such as neutral positive and negative neutral, the basic value is the average of the basic values of the corresponding two types of emotions, and then multiplied by the intensity correction coefficient, for example, the basic value of neutral positive = (1+0.5) ÷ 2 = 0.75, and the intensity 0.6 level is 0.75 x 0.6 = 0.45; According to the fusion contribution degree, the preliminary comprehensive emotion representation is calculated, and the calculation logic is that the sentiment value of each modality is multiplied by the corresponding fusion contribution degree to obtain the weighted sentiment value of the modality; Then add the weighted sentiment values of the three modalities, and the sum is the preliminary comprehensive emotion representation. According to the sentiment value and fusion contribution degree calculated this time, the weighted sentiment value of video modality = video sentiment value x video fusion contribution degree = 0.9 x 0.333 ≈ 0.2997; The weighted sentiment value of audio modality = audio sentiment value x audio fusion contribution degree = 0.8 x 0.333 ≈ 0.2664; The weighted sentiment value of text modality = text sentiment value x text fusion contribution degree = 0.7 x 0.334 ≈ 0.2338; The preliminary comprehensive emotion representation = 0.2997 + 0.2664 + 0.2338 ≈ 0.8.

[0029] Normalization processing is performed, and first, the effective value range of the sentiment value is determined as 0 to 1, 0 corresponds to the extreme negative, 0.5 corresponds to the absolute neutral, and 1 corresponds to the extreme positive. The purpose of normalization is to ensure that the preliminary comprehensive sentiment representation is within the effective range, and to avoid exceeding the range due to calculation deviation or special circumstances. The processing rule is that if the preliminary comprehensive sentiment representation is within the range of 0 to 1, the normalized result = the preliminary comprehensive sentiment representation; if the preliminary comprehensive sentiment representation > 1, the normalized result = 1; if the preliminary comprehensive sentiment representation < 0, the normalized result = 0. The preliminary comprehensive sentiment representation this time is 0.8, which is within the range of 0 to 1, so the normalized result is still 0.8. Combined with the sentiment category corresponding to the sentiment value and the core basis, a structured comprehensive emotional state representation is generated. The representation format needs to include three parts: emotional category, emotional intensity, and core basis mode. For example, the comprehensive emotional state representation this time is that the comprehensive emotional state is positive emotion, the emotional intensity is 0.8 level, and the core basis is that the sentiment tendency of the video, audio and text three modes is highly consistent, among which the positive emotional features of the video mode are the most significant, and the positive emotional features of the audio and text modes are mutually confirmed. The representation not only clearly indicates the category and intensity of the emotion, but also explains the core basis, ensuring the credibility and traceability of the comprehensive emotional state.

[0030] By multi-dimensional semantic label annotation, cross-modal semantic alignment and interactive correlation strength analysis, the explicit emotional clues of the video, audio and text three modes are comprehensively integrated. With the help of strict scoring rules and contribution degree distribution logic in emotional consistency evaluation, the reliability of different modal emotional expressions is effectively distinguished.

[0031] In a preferred embodiment of the present application, the above step 3, according to the comprehensive emotional state representation, combining the context information of the multi-round dialogue, using the natural language generation model to obtain the dialogue text content consistent with the current emotional state of the user, and converting it to voice output through the voice conversion process, can include: In the embodiment of the present application, step 330, the integrated emotional state representation is fused with the vectorized encoded multi-round dialogue context to generate the context semantic representation; specifically comprising: determining the specific form of the two core inputs, the integrated emotional state representation is a fixed dimension numerical vector containing the quantized numerical values of the three core dimensions of valence, arousal, and dominance, as well as the emotional contribution degree numerical values corresponding to the video, audio, and text modalities, and each dimension numerical value is in a preset unified interval; the multi-round dialogue context refers to the historical dialogue content before the current interaction, containing the user's sent sentences and the system's previously generated response sentences, arranged in chronological order to form a dialogue sequence; the multi-round dialogue context is vectorized encoded, first, each sentence in the dialogue sequence is split into independent lexical units, and stop words without actual semantics are removed, leaving core nouns, verbs, adjectives, and other semantic words; each core word is assigned a corresponding semantic weight, which is set according to the importance of the word in the dialogue, such as a core appeal word weight of 3, a descriptive word weight of 1, and a conjunction word weight of 0.5; each word is mapped to a fixed-length basic vector, and then the weighted vector of each word is calculated by the word basic vector x word semantic weight method, and then the weighted vectors of all words in a sentence are added to obtain the initial vector of a single sentence; in chronological order of the dialogue, the initial vectors of all single sentences are sequentially concatenated, and then all single sentence vectors are integrated by the weighted recursive method of previous sentence vector x 0.7 + next sentence vector x 0.3, to finally obtain the vectorized encoding result of the multi-round dialogue context, which is consistent with the vector dimension of the integrated emotional state representation; the context semantic representation is generated by fusion calculation, first, the fusion weights of the integrated emotional state representation and the multi-round dialogue context encoding vector are set, which are preset according to the interaction scene, such as a weight of 0.6 for the integrated emotional state representation and a weight of 0.4 for the multi-round dialogue context encoding vector in the emotional support scene; in the regular consultation scene, the weights of both are 0.5; then the integrated emotional state representation vector x the corresponding fusion weight + the multi-round dialogue context encoding vector x the corresponding fusion weight is calculated to obtain the initial fusion vector; then each dimension value in the initial fusion vector is normalized by dividing each dimension value by the sum of all dimension values, so that the sum of all dimension values is 1, and finally the context semantic representation containing both emotional information and context semantic information is generated.

[0032] Step 331, input the context semantic representation into the pre-trained natural language generation model to generate dialogue text consistent with the current emotional state in semantic content and expression; specifically including: constructing a pre-trained natural language generation model, the specific process is to select a large-scale general dialogue corpus, including daily casual conversation, consultation service, emotional counseling and other dialogue samples and emotional dialogue corpus with emotion labels, each dialogue sample is labeled with corresponding emotion type, such as happy, angry, depressed, anxious, etc. as the data source for natural language generation model training; input the samples in the general dialogue corpus into the basic language, and let the natural language generation model learn semantic coherence, topic relevance and language expression standardization; then classify the samples in the emotional dialogue corpus with emotion labels according to emotion type, input the basic model with learned general semantics one by one, adjust the parameters in the natural language generation model, so that the natural language generation model preferentially activates the vocabulary library and sentence pattern related to the emotion when generating text with corresponding emotion; during the training process, by comparing the semantic similarity and emotional matching degree between the text generated by the natural language generation model and the original samples in the corpus, the parameters of the natural language generation model are constantly corrected until the semantic accuracy and emotional fit degree of the generated text reach the preset standard, and the construction of the pre-trained natural language generation model is completed.

[0033] Input the context semantic representation into the pre-trained natural language generation model, the natural language generation model first analyzes the multi-round dialogue semantic information in the context semantic representation, determines the core topic of the current dialogue, the user's expressed appeal and the historical interaction logic, and ensures that the generated text is consistent with the topic; then extract the emotion information in the context semantic representation to determine the user's current emotion type and emotion intensity; according to the emotion type, match the corresponding emotional vocabulary library, and according to the emotion intensity, adjust the use intensity of the emotional vocabulary; at the same time, combine the context semantics to determine the core function of the generated text, and select the sentence pattern matched with the function and emotion; finally, through the semantic organization logic in the natural language generation model, combine the screened vocabulary and the determined sentence pattern to generate dialogue text that not only fits the dialogue context, but also highly matches the current emotional state of the user.

[0034] Step 332, the dialogue text and the comprehensive emotional state representation are input into the emotional speech synthesizer, and corresponding rhythm, pitch and rhythm parameters are mapped according to the comprehensive emotional state; specifically including: constructing an emotional speech synthesizer, the specific process is to collect a real human voice sample library containing multiple emotional types, each voice sample corresponds to clear text content and emotional label, and the rhythm, pitch, rhythm and other parameter data of the sample are labeled; the text content in the voice sample library is split into syllable units, and a corresponding relationship library of text syllable-voice parameter-emotional label is established; a voice parameter mapping rule system is built to associate the core dimensions of the comprehensive emotional state representation with the voice parameters; the mapping rule system is optimized and improved using the data in the sample library, so that the mapping rule system masters the corresponding rules of text syllable and voice parameters under different emotional states, and by continuously optimizing the associated parameters in the rule system, it can accurately output corresponding voice parameters according to the input emotional state and text content, and complete the construction of the emotional speech synthesizer.

[0035] The parameter mapping is performed, the dialogue text is input into the emotional speech synthesizer, and the text is segmented, punctuated and syllable-labeled; then the comprehensive emotional state representation is input into the synthesizer, the synthesizer first extracts the valence value to determine the positive and negative tendency of the emotion, the valence value higher than the preset intermediate value is positive emotion, and the valence value lower than the preset intermediate value is negative emotion; the arousal value is extracted to determine the intensity of the emotion, the arousal value higher than the preset intermediate value is high-intensity emotion, and the arousal value lower than the preset intermediate value is low-intensity emotion; according to the positive and negative tendency and the intensity of the emotion, the basic parameters are matched from the text syllable-voice parameter-emotional label corresponding relationship library, such as positive high-intensity emotion corresponding to high basic pitch and fast basic rhythm, and negative low-intensity emotion corresponding to low basic pitch and slow basic rhythm; then combined with the modal emotional contribution degree in the comprehensive emotional state representation, such as high audio modal emotional contribution degree, indicating that the user's voice emotion is obvious, and the pitch fluctuation parameter needs to be strengthened; the video modal emotional contribution degree is high, indicating that the user's facial emotion is obvious, and the rhythm pause parameter needs to be strengthened, the basic parameters are modified, such as audio modal emotional contribution degree x 0.3 + basic pitch = modified pitch, and video modal emotional contribution degree x 0.2 + basic pause duration = modified pause duration; finally, the rhythm parameters, pitch parameters and rhythm parameters accurately matched with the comprehensive emotional state are generated.

[0036] In step 333, the dialogue text is processed by voice synthesis based on the rhythm, tone and rhythm parameters to obtain natural voice output carrying corresponding emotional expressiveness; specifically including: preprocessing the dialogue text, according to the annotated syllable units, the dialogue text is disassembled into the smallest pronunciation unit, and the standard pronunciation duration of each pronunciation unit is annotated; combining the pronunciation speed in the rhythm parameter, adjusting the actual pronunciation duration of each pronunciation unit, adjusting the pronunciation characteristics according to the parameter, setting the reference tone of the voice according to the basic frequency in the tone parameter; then adjusting the tone change of the corresponding pronunciation unit according to the frequency fluctuation value of each syllable, ensuring that the tone change meets the emotional expression requirement; according to the pause position and duration in the rhythm parameter, inserting a pause at the corresponding position of the dialogue text to avoid harsh voice expression; performing voice synthesis and optimization, concatenating the adjusted pronunciation units in sequence according to the text to form a continuous voice stream; smoothing the voice stream; referring to the emotional expression rule of real voice, fine-tuning the volume of the voice stream; finally, the voice output is obtained, which not only conforms to the semantic of the dialogue text, but also carries the emotional expressiveness consistent with the current emotional state of the user, and the pronunciation is natural and smooth.

[0037] By deeply fusing the comprehensive emotional state and the context of the multi-round dialogue, it is ensured that the generated dialogue text not only closely fits the semantic logic of the previous text and does not deviate from the discussion topic, but also accurately matches the current real emotional state of the user.

[0038] In a preferred embodiment of the present application, step 4, in the process of voice output, the dynamic changes of user biological signals and voice signals are monitored in real time, the newly collected multi-modal data is organized into time series data stream, and the emotional state analysis benchmark is constructed; by detecting the key turning points and extreme points in the time series data stream, the key event points representing the boundary of the emotional state are extracted, and two emotional change modes are identified, which can include: In the embodiment of the present application, step 440, the continuously collected original biological signals and speech signals are intercepted with a preset length of sliding time window to obtain the biological signal segments and speech signal segments within the current time window; specifically comprising: determining the setting standard of the preset length and the sliding step length, combining the timeliness of the emotional response of the biological signals and the speech signals, determining the preset length of the sliding time window as 2 seconds and the sliding step length as 1 second, determining the type and format standard of the collected original signals, the original biological signals including electroencephalogram signals, electrocardiogram signals, and galvanic skin response signals, the collection format of the three being consistent with step 1; the collection format of the original speech signals is also consistent with step 1, which is a 16 kHz sampling rate, 16-bit depth, single-channel signal, performing the signal interception operation, starting real-time collection from the moment of speech output, taking the current system time as the reference, triggering window interception once every 1 second, when intercepting, taking the current trigger time as the window endpoint, tracing back 2 seconds as the window starting point to determine the time range of the window, intercepting and storing the segments according to the signal type, for the electroencephalogram signals, electrocardiogram signals, and galvanic skin response signals, all the sampling point data within the corresponding time range are intercepted to form three independent biological signal segments; at the same time, the speech signal sampling point data within the corresponding time range is intercepted to form a speech signal segment, each segment is labeled with the corresponding timestamp and stored in the format of signal type-window number-timestamp, ensuring the traceability of the segments, for example, EEG-1-0-2s represents the electroencephalogram signal segment within the 0-2 second time range of the first window.

[0039] Step 441, feature extraction is performed on the biosignal segment and the speech signal segment within the current time window to obtain a real-time feature vector representing the current physiological state and speech state of the user; specifically including: feature extraction is performed on the biosignal segment, the extraction dimension is consistent with the biosignal feature in step 1, ensuring the uniformity of the features, the EEG signal segment feature, the 2-second EEG segment is divided into 2 small segments according to 1 second as a unit, the mean value of each small segment is calculated, the sum of all sample point values in the segment is added, divided by the sample point number of the segment, the variance, the square of the difference between each sample point value and the segment mean value is added, divided by the sample point number, and the alpha wave (8-13 Hz) energy ratio, the beta wave (14-30 Hz) energy ratio (the energy ratio of a certain frequency band = the sum of the squares of all sample point values in the frequency band, divided by the sum of the squares of all sample point values in the small segment); the average value of the features corresponding to the 2 small segments is calculated (the mean value of the 2 small segments is added and divided by 2, the variance and energy ratio are the same), 4 feature parameters of EEG are finally obtained; ECG signal segment feature, the number of R wave peaks in the 2-second ECG segment is counted, multiplied by 30 to obtain the heart rate per minute, because 2 seconds is 1 / 30 minute, the number x 30 is the heart rate per minute; the time interval between consecutive R wave peaks is calculated to obtain a number of R-R interval values, and the mean value (the sum of all R-R interval values is added and divided by the interval number) and the standard deviation (the square of the difference between each R-R interval value and the mean value is added and divided by the interval number) of these values are calculated, 3 feature parameters of ECG are finally obtained; GSR signal segment feature, the peak value of the 2-second GSR segment is calculated, the maximum value of all sample points in the segment, the rising rate (the difference between the peak value and the starting sample point value of the segment, divided by the difference between the peak value occurrence time and the segment starting time), and the total value (the sum of all sample point values in the segment), 3 feature parameters of GSR are finally obtained.

[0040] The 4 features of EEG, 3 features of ECG, and 3 features of GSR are arranged in the order of EEG-ECG-GSR to form a 10-dimensional biological signal feature sub-vector. Next, the speech signal segment is extracted for features, and the extraction dimension is consistent with the audio features in step 1. The time domain features are divided into several frames by 20 ms frame length and 10 ms frame shift for a 2-second speech segment. The silent frames with short-time energy lower than 10% of the average value of all frames are removed. The short-time energy, zero-crossing rate, and pitch frequency of the remaining valid frames are calculated, respectively. The average value of the features corresponding to all valid frames is calculated. The short-time energy of all valid frames is added and divided by the number of valid frames. The zero-crossing rate and pitch frequency are calculated in the same way, and 3 time domain feature parameters are obtained. The frequency domain features are processed by Fourier transform for each valid frame to obtain the spectral amplitude of 256 frequency points. The average value of the spectral amplitude of the frequency points corresponding to all valid frames is calculated. The amplitude values of all valid frames of each frequency point are added and divided by the number of valid frames to form a 256-dimensional spectral amplitude feature. The spectral centroid is calculated. The frequency value of each frequency point is multiplied by the average spectral amplitude of the point. The sum of all products is obtained, and then divided by the sum of the average spectral amplitudes of all frequency points to obtain 1 spectral centroid parameter. The 3 time domain features, 256-dimensional spectral amplitude features, and 1 spectral centroid feature are arranged in order to form a 260-dimensional speech signal feature sub-vector. The 10-dimensional biological signal feature sub-vector and the 260-dimensional speech signal feature sub-vector are spliced in the order of biological signal and speech signal to form a vector with a dimension of 10+260=270. The vector is the real-time feature vector representing the current physiological state and speech state of the user.

[0041] At step 442, the real-time feature vector is spliced and aligned with the preset reference feature sequence to form a unified and continuous time sequence data stream; specifically including: determining the source and composition of the preset reference feature sequence, the reference feature sequence is a sequence composed of multiple rounds of feature vectors collected in the user's calm state before the voice output starts, the collection process is to continuously collect biological signals and voice signals for 5 windows with a 2-second window and a 1-second step, extract 5 real-time feature vectors respectively according to the method of step 441, arrange the 5 vectors in the order of collection time, form a preset reference feature sequence, the sequence length is 5, the dimension of each vector is 270, and each vector is labeled with a corresponding collection timestamp, after the voice output starts, each generated real-time feature vector is labeled with a corresponding window timestamp, the timestamp of the newly generated real-time feature vector is connected with the timestamp of the last vector in the reference feature sequence to ensure that the timestamps of all vectors are continuous without breakpoints, each vector in the reference feature sequence and the newly generated real-time feature vector are 270-dimensional, and the feature types corresponding to each dimension are completely consistent, without dimension adjustment, directly spliced according to the timestamp sequence, the reference feature sequence is used as the initial part of the data stream, then each newly generated real-time feature vector is added to the end of the data stream according to the timestamp sequence, so that the length of the data stream gradually increases over time, each element in the data stream is a 270-dimensional feature vector, and is arranged continuously according to the timestamp, forming a unified and continuous time sequence data stream, which completely records the physiological and voice feature changes of the user from the calm state to the voice interaction process.

[0042] Step 443, the reference feature vectors corresponding to the user calm state interval in the time series data stream are aggregated and analyzed, the central tendency of the reference feature vector numerical distribution is calculated, and the emotional state baseline, i.e. the emotional state analysis baseline, is constructed; specifically including: screening the reference feature vectors of the calm state interval, extracting the initial reference feature sequence from the time series data stream, these vectors correspond to the feature data of the user in the calm state before voice output, without emotional fluctuation interference, determining the reference feature vector set of the calm state interval, respectively aggregating and analyzing each feature dimension, calculating the central tendency, using the mean as the central tendency index, taking into account the calculation simplicity and representativeness, for each dimension in the 270-dimensional feature, the numerical values of the 5 reference feature vectors in the dimension are extracted, the 5 numerical values are added, and then divided by the number of reference vectors 5 to obtain the mean of the dimension. The mean is the central tendency value of the feature dimension in the calm state, for example, the 5 reference values of the 1st dimension (EEG mean) are 0.3, 0.32, 0.28, 0.31, and 0.29, the sum is 1.5, and the mean is 0.3, which is the central tendency value of the 1st dimension; construct the emotional state baseline, arrange the central tendency values of the 270 feature dimensions in the original feature dimension order to form a 270-dimensional baseline vector, which is the emotional state baseline. The value of each dimension in the baseline represents the typical level of the feature of the user in the calm state, for example, the baseline value of the 11th dimension is 0.4, which means that the typical value of the short-time energy of the user's voice in the calm state is 0.4. The deviation degree of the real-time feature vector from the baseline can be compared to judge whether the emotion has changed. The baseline is verified and calibrated, the Euclidean distance of each vector in the 5 reference feature vectors from the baseline is calculated, the square of the difference between the value of each dimension and the corresponding value of the baseline is added, and then the square root of the sum is taken. If the Euclidean distance of all vectors is less than the preset deviation threshold, the baseline is valid; if there is a vector exceeding the threshold, the abnormal reference vector is removed, the mean of each dimension of the remaining 4 vectors is recalculated, and the baseline is updated until the distance of all reference vectors from the baseline meets the requirements, ensuring the reliability of the baseline.

[0043] Step 444, the time series data stream is divided into a fixed time window, the time domain features and the frequency domain features of the biological signal in each window are calculated, and a high-dimensional feature sequence is generated; specifically including: setting the parameters of the fixed time window, combining the update frequency of the time series data stream, setting the time length of the fixed time window as 1 second and the step length as 0.5 second, that is, each window contains 2 continuous data stream vectors, dividing the fixed time window, taking the starting time stamp of the first vector of the time series data stream as the starting point, dividing the window in 1 second time length and 0.5 second step length, for example, the first window time range is 0-1 second, the second is 0.5-1.5 second, the third is 1-2 second, and so on, to ensure that each fixed window corresponds to part of the vector in the data stream and covers the entire time range of the data stream, extracting the biological signal features in each fixed window, time domain feature extraction, for each fixed window, extracting the biological signal feature sub-vector in all data stream vectors covered by the window, for each dimension, calculating the mean value of all numerical values in the dimension in the window, the sum of all numerical values in the dimension in the window divided by the number of numerical values, the variance, the square of the difference between each numerical value and the mean value divided by the number of numerical values, the maximum value, the maximum value of the dimension in the window, the minimum value of the dimension in the window, the minimum value of the dimension in the window, 4 time domain features are obtained for each dimension, and 40 time domain feature parameters are obtained for 10 dimensions; frequency domain feature extraction, Fourier transform processing is performed on the biological signal feature sub-vector in each fixed window according to the dimension, the frequency spectrum energy of each dimension is calculated, the sum of the squares of all frequency point values in the frequency domain, the frequency spectrum center, the frequency value of each frequency point multiplied by the frequency domain value of the point, and the sum of all products divided by the sum of all frequency domain values of the frequency points, 2 frequency domain features are obtained for each dimension, and 20 frequency domain feature parameters are obtained for 10 dimensions; finally, a high-dimensional feature sequence is generated by fusing, 40 time domain feature parameters and 20 frequency domain feature parameters of each fixed window are spliced in the order of time domain and frequency domain to form a 60-dimensional window feature vector; then the 60-dimensional feature vector of each window is spliced with the speech signal feature sub-vector in the data stream vector covered by the window to form a 60+260=320-dimensional high-dimensional feature vector; all high-dimensional feature vectors are arranged in the order of fixed window division to form a high-dimensional feature sequence, the sequence length is consistent with the number of fixed windows, and each vector is labeled with the corresponding window time stamp.

[0044] Step 445, first-order difference calculation is performed on the high-dimensional feature sequence to obtain a feature change rate sequence, zero-crossing points and local extreme points in the feature change rate sequence are detected, and candidate key event points are identified; specifically including: calculating the first-order difference to obtain the feature change rate sequence, the first-order difference calculation is performed on 320 feature dimensions in the high-dimensional feature sequence respectively, for each dimension, the feature value of the next window is subtracted from the feature value of the previous window according to the window order of the high-dimensional feature sequence, to obtain the difference value of this dimension, that is, the feature change rate; the difference values of all windows are arranged in order to form the change rate subsequence of each dimension; the change rate subsequences of 320 dimensions are spliced in the original dimension order to form a 320-dimensional feature change rate sequence, the sequence length is the length of the high-dimensional feature sequence minus 1, for the change rate subsequence of each dimension, the signs of the adjacent two difference values are checked one by one, if the previous difference value is positive and the next difference value is negative, or the previous difference value is negative and the next difference value is positive, it is determined that there is a zero-crossing point between the windows corresponding to the two difference values, the window position and the feature dimension to which the zero-crossing point belongs are recorded, for example, the difference value of the third window of a certain dimension is 0.2, and the difference value of the fourth window is -0.1, there is a zero-crossing point between the third and fourth windows, the window position is recorded as 4, and the dimension is the feature dimension, for the change rate subsequence of each dimension, the size relationship between each difference value and its adjacent two difference values is checked one by one, only one adjacent value is compared with the first and last difference values, if a certain difference value is greater than its adjacent difference values, it is determined to be a local maximum point; if a certain difference value is less than its adjacent difference values, it is determined to be a local minimum point; the window position and the feature dimension to which each local extreme point belongs are recorded, for example, the difference value of the fifth window of a certain dimension is 0.3, the difference value of the fourth window is 0.2, and the difference value of the sixth window is 0.1, the difference value of the fifth window is a local maximum point, the window position is recorded as 5, and the dimension is the feature dimension, the records of all zero-crossing points and local extreme points are summarized, and repeated window positions are removed to obtain a candidate key event point list, each candidate point in the list contains two core information, window position, and feature dimension set involved, these candidate points represent potential time nodes where emotional state changes may occur.

[0045] Step 446, using a preset amplitude threshold and duration threshold to screen the candidate key event points, merging adjacent candidate key event points to form a key event point sequence; specifically including: the amplitude threshold is used to screen out candidate points with significant feature changes, and the calculation method is as follows: for each dimension of the feature change rate sequence, extract all difference values in the calm state interval before the voice output, calculate the mean value of the absolute values of these values, add all absolute values and divide by the number of values, and then multiply the mean value by 2 to obtain the amplitude threshold of this dimension; the amplitude thresholds of 320 dimensions are arranged in dimension order to form a 320-dimensional amplitude threshold vector, for each candidate key event point, check the absolute values of the difference values of all feature dimensions involved, if at least 50% of the dimension difference value absolute values are greater than the amplitude threshold of the corresponding dimension, the candidate point is retained; if it is less than 50%, it is determined as a false candidate point caused by noise interference and is removed; the duration threshold is used to merge adjacent candidate points to avoid repeated counting, and according to the time characteristics of emotional changes, it is set to 0.5 seconds, that is, when the time interval between two candidate points is less than 0.5 seconds, it is determined that they are adjacent and close, the candidate key event points after screening are sorted in ascending order according to the time stamp corresponding to the window position, the time interval between adjacent two candidate points is calculated, if the time interval is less than 0.5 seconds, the two candidate points are merged into one key event point, the time stamp of the merged point is the average of the time stamps of the two points, and the feature dimension set involved is the union set of the dimension sets of the two points; if the time interval is greater than or equal to 0.5 seconds, two independent candidate points are retained without merging, for example, candidate point A has a time stamp of 1.2 seconds, candidate point B has a time stamp of 1.5 seconds, and the interval is 0.3 seconds < 0.5 seconds, which are merged into a key event point with a time stamp of 1.35 seconds, and the dimension set is the result of merging the dimension sets of A and B; all merged key event points are sorted in ascending order according to the time stamp, each point contains the time stamp, the feature dimension set involved, and the maximum absolute value of the corresponding difference value, and are arranged in order to form a key event point sequence, each point in the sequence represents an explicit time node where the emotional state may change, and the feature change is significant and has no repetition and redundancy.

[0046] At step 447, based on the sequence of key event points, the time interval between adjacent key event points and the feature amplitude difference value are calculated; specifically, the time interval between adjacent key event points is calculated, the sequence of key event points is arranged in ascending order of timestamp, and it is assumed that the sequence contains N key event points. Adjacent point pairs are formed by taking the first and second, second and third, and N-1th and Nth points in turn, and N-1 point pairs are formed. For each point pair, the timestamp of the latter point is subtracted from the timestamp of the former point to obtain the time interval of the point pair. For example, the timestamp of the first point is 2.0 seconds, and the timestamp of the second point is 3.5 seconds. The time interval is 3.5 seconds-2.0 seconds=1.5 seconds. The timestamps of the second point and the third point are 3.5 seconds and 5.0 seconds, respectively, and the time interval is 1.5 seconds. The time intervals of all adjacent point pairs are calculated in turn to obtain N-1 time interval values. The feature amplitude difference value between adjacent key event points is calculated. For each adjacent point pair, the high-dimensional feature vectors corresponding to the former point and the latter point are extracted. For each feature dimension, the feature value of the latter point is subtracted from the feature value of the former point to obtain the feature difference value of the dimension, and the absolute value of the difference value is taken. The absolute values of the feature difference values of the 320 dimensions are added together and then divided by the total number of feature dimensions, 320, to obtain the average feature amplitude difference value of the point pair. This value reflects the overall degree of change in the features between the two key event points. For example, the sum of the absolute values of the 320-dimensional feature difference values of a certain point pair is 64, and the average feature amplitude difference value is 64÷320=0.2. The sum of the absolute values of the 320-dimensional feature difference values of another point pair is 96, and the average feature amplitude difference value is 96÷320=0.3. The average feature amplitude difference values of all adjacent point pairs are calculated in turn to obtain N-1 amplitude difference values. The time interval value of each adjacent point pair is bound to the average feature amplitude difference value corresponding to the point pair to form N-1 time interval and amplitude difference value data pairs. Each data pair corresponds to a continuous emotional change process in the sequence of key event points.

[0047] Step 448, compare the time interval and the feature amplitude difference value with the preset threshold value, distinguish the mutation mode and the gradual change mode of the emotional state; Specifically includes: determining the time interval threshold value and the feature amplitude difference value threshold value, both of which are determined based on the feature change law in the calm state, the time interval threshold value is calculated as the average time interval between the key event points corresponding to the calm state interval, half of the average time interval is set as the time interval threshold value, for example, the average time interval in the calm state is 2 seconds, the time interval threshold value = 2 seconds ÷ 2 = 1 second; The feature amplitude difference value threshold value is calculated as the average feature amplitude difference value of all time interval and amplitude difference value data pairs in the calm state interval, and 3 times of the average value is set as the feature amplitude difference value threshold value, for example, the average amplitude difference value in the calm state is 0.1, the feature amplitude difference value threshold value = 0.1 × 3 = 0.3, the mode distinguishing rule is formulated, based on the characteristics of emotional change, the determination criteria of the two modes are determined, the mutation mode corresponds to the sharp change of emotion in a short time, and the gradual change mode corresponds to the slow change of emotion in a long time, the specific rule is that if the time interval and the amplitude difference value data pair, the time interval is less than the time interval threshold value, and the feature amplitude difference value is greater than the feature amplitude difference value threshold value, it is determined that the emotional change process corresponding to the data pair is the mutation mode, for example, the time interval threshold value is 1 second, the amplitude difference value threshold value is 0.3, the time interval of a certain data pair is 0.8 seconds (<1 second), and the amplitude difference value is 0.4 (>0.3), which is determined as the mutation mode, if the time interval of a certain data pair is greater than or equal to the time interval threshold value, and the feature amplitude difference value is less than or equal to the feature amplitude difference value threshold value, it is determined that the emotional change process corresponding to the data pair is the gradual change mode, for example, the time interval of a certain data pair is 1.2 seconds (≥1 second), and the amplitude difference value is 0.2 (≤0.3), which is determined as the gradual change mode, if the data pair does not satisfy the above two rules, the emotional state reference line is used to assist in judgment, the Euclidean distance between the high-dimensional feature vectors of the two key event points corresponding to the data pair and the reference line is calculated, the square of the difference between each dimension value and the corresponding dimension value of the reference line is added, and then the square root is taken, if the distance difference between the two points and the reference line is greater than 2 times the average distance of the reference line, it is determined as the mutation mode; If the distance difference is less than or equal to 2 times the average distance of the reference line, it is determined as the gradual change mode, according to the order of the time interval and the amplitude difference value data pair, the mode corresponding to each data pair is arranged in turn to form the emotional change mode sequence.

[0048] The dynamic capture of the user's physiological and speech signals is realized, the subtle fluctuations of the emotion are ensured to be perceived in time, the response lag to the emotional change is avoided, the interference of individual differences and environmental noise on the emotional recognition is reduced, and the accuracy of emotional judgment is improved. In a preferred embodiment of the present application, step 5 analyzes the two emotional change patterns, determines the fluctuation range of the emotional state in combination with the result of the semantic analysis, selects the reference instances of the emotional state to construct the emotional state change trajectory, generates the emotional state dynamic adjustment parameter based on the emotional state change trajectory, and adjusts the dialogue strategy and generated content in real time to realize the emotional adaptive response, which can include: In the embodiment of the present application, step 550 calculates the emotional state fluctuation time span and the characteristic amplitude range corresponding to each mode based on the mutation mode and the gradual change mode to form the preliminary fluctuation range parameters; specifically including: determining the key event point sequence corresponding to the mutation mode and the gradual change mode respectively, for the mutation mode, first filtering out all adjacent key event points under this mode, calculating the time difference between each pair of adjacent key event points, that is, the time stamp of the latter key event point minus the time stamp of the former key event point, then adding all these time difference values, the sum obtained is the emotional state fluctuation time span corresponding to the mutation mode; at the same time, extracting the real-time feature vector value corresponding to each key event point under the mutation mode, finding the maximum and minimum values in these feature vector values, subtracting the minimum value from the maximum value obtained, the result is the emotional state characteristic amplitude range corresponding to the mutation mode, for the gradual change mode, using the same calculation method as the mutation mode, first calculating all the time difference values between adjacent key event points under this mode and summing them up to obtain the emotional state fluctuation time span of the gradual change mode; then extracting the real-time feature vector value corresponding to all key event points under this mode, finding the maximum and minimum values and calculating the difference to obtain the emotional state characteristic amplitude range of the gradual change mode, the fluctuation time span and the characteristic amplitude range of the mutation mode and the fluctuation time span and the characteristic amplitude range of the gradual change mode are respectively taken as the preliminary fluctuation range parameters of the two modes.

[0049] Step 551, align the preliminary fluctuation interval parameters with the emotional semantic information obtained by cross-modal fusion semantic analysis, correct and calibrate the boundaries of the fluctuation interval by using the emotional semantic information to determine the final emotional state fluctuation interval; Specifically, it includes: obtaining the emotional semantic information obtained by cross-modal fusion semantic analysis, which contains the emotional semantic labels corresponding to the multi-modal data, the time segments corresponding to the emotional semantics and the emotional semantic intensity description, corresponding the time range of the fluctuation time span in the preliminary fluctuation interval parameters with the time segments in the emotional semantic information one by one, determining all emotional semantic labels contained in the preliminary fluctuation time range and the corresponding intensity description; At the same time, the feature amplitude range in the preliminary fluctuation interval parameter is associated with the emotional semantic intensity description, and then the boundary is corrected. If the start time of the preliminary fluctuation time span is earlier than the start time of the corresponding core emotional semantic in the emotional semantic information, the start time of the preliminary fluctuation time span is adjusted to the start time of the core emotional semantic. If the end time of the preliminary fluctuation time span is later than the end time of the corresponding core emotional semantic in the emotional semantic information, the end time of the preliminary fluctuation time span is adjusted to the end time of the core emotional semantic. If the upper limit of the feature amplitude range of the preliminary fluctuation is lower than the upper limit of the regular feature amplitude corresponding to the emotional semantic intensity description, the upper limit of the feature amplitude range is corrected to the regular amplitude upper limit. If the lower limit of the feature amplitude range of the preliminary fluctuation is higher than the lower limit of the regular feature amplitude corresponding to the emotional semantic intensity description, the lower limit of the feature amplitude range is corrected to the regular amplitude lower limit. After the calibration of the time boundary and the amplitude boundary, the final emotional state fluctuation interval is obtained.

[0050] Step 552, based on the emotional state fluctuation interval, retrieve the samples matching the emotional characteristics and emotional semantic labels in the current fluctuation interval in the emotional state instance library, and select the emotional state reference instance; Specifically, it includes: determining the core information contained in the sample data stored in the emotional state instance library, each sample corresponding to the emotional fluctuation time span, the emotional feature amplitude range, the emotional feature vector set, the emotional semantic label and the emotional change trend description, then taking the final determined emotional state fluctuation interval as the retrieval condition, first step: screening out the samples in the instance library whose emotional fluctuation time span difference with the current fluctuation time span is within the pre-set reasonable range and the overlap ratio of emotional feature amplitude range with the current feature amplitude range reaches the pre-set threshold; Second step: further comparing the emotional semantic labels from the samples screened out in the first step, keeping the samples whose emotional semantic labels are completely consistent or highly similar with the current cross-modal fusion semantic analysis; Third step: extracting the emotional feature vector set within the current fluctuation interval, calculating the similarity of these feature vectors with the emotional feature vector set of the samples screened in the second step, selecting a number of samples with the highest similarity by comparing the fitting degree of corresponding dimension feature values; Finally, these selected samples are determined as the emotional state reference instance.

[0051] Step 553, the mood state reference instances are organized in chronological order, and combined with the determined mood state fluctuation interval, a mood state change trajectory reflecting the continuous change process of the mood state is constructed; specifically including: extracting the time sequence information corresponding to each mood state reference instance and the mood feature value and mood semantic label corresponding to each time node, then, taking the time axis of the current human-computer interaction process as the benchmark, uniformly mapping the time sequences of all mood state reference instances, ensuring that the time order of the reference instances is consistent with the time order of the current user mood change, then, combining the final mood state fluctuation interval, marking the mood feature value and mood semantic label corresponding to each time node on the corresponding position of the current time axis; for the blank time period between adjacent time nodes, according to the mood feature value change trend of the front and back two nodes, supplementary transitional mood feature values are added to make the mood feature value change form a continuous curve, finally, the continuous curve marked with time nodes, mood feature values and mood semantic labels is integrated to form a mood state change trajectory that can completely reflect the continuous change process of the user from the mood fluctuation starting point to the current time point, and contains the future potential change trend.

[0052] Step 554, analyzing the mood state change trajectory to extract the mood intensity change rate and the mood type transfer mode to form the mood evolution trend feature; specifically including: extracting the mood intensity value corresponding to each time node from the mood state change trajectory, which is obtained by synthesizing the multi-modal fused emotional features and reflects the intensity of the emotion, then calculating the mood intensity change rate, selecting two adjacent time nodes in the trajectory, subtracting the mood intensity value of the former time node from the mood intensity value of the latter time node to obtain the mood intensity difference, and then dividing the difference by the time interval between the two time nodes, i.e. the time stamp of the latter time node minus the time stamp of the former time node, to obtain the mood intensity change rate between the two nodes; in the above manner, the mood intensity change rates between all adjacent time nodes in the trajectory are calculated to form a complete mood intensity change rate sequence, then the mood type transfer mode is analyzed, the mood semantic label corresponding to each time node in the trajectory is observed, and whether the label changes, the starting time and ending time of the change, and the emotion types before and after the change are recorded; if the mood semantic label remains unchanged within a plurality of continuous time nodes, it is determined as a mood type stable mode; if the label switches, it is determined as a mood type transfer mode, and the direction and speed of the transfer are specified, finally, the obtained mood intensity change rate sequence and mood type transfer mode are integrated to form the mood evolution trend feature.

[0053] At step 555, according to the emotion evolution trend feature, the context information of the multi-round dialogue is combined to generate dialogue strategy parameters and content generation parameters through a preset mapping relationship. The dialogue strategy parameters include response timing and dialogue initiative degree, and the content generation parameters include emotional vocabulary density and sentence complexity. Specifically, the context information of the multi-round dialogue is obtained, including the topic content of the historical dialogue, the core demand expressed by the user before, the generated dialogue response content, and the feedback performance of the user to the historical response, then the emotion evolution trend feature is associated and analyzed with the context information to determine the relevance of the current emotion evolution and the historical dialogue, then the preset mapping relationship is called, if the emotion evolution trend is rapid rise of emotion intensity, transfer of negative emotion type, and context showing that the user is not satisfied because the demand is not responded, then the parameter combination of early response timing, low dialogue initiative degree, low emotional vocabulary density, and simple sentence complexity is mapped; if the emotion evolution trend is slow rise of emotion intensity, stable positive emotion type, and context showing that the user is interested in the current topic, then the parameter combination of normal response timing, high dialogue initiative degree, high emotional vocabulary density, and medium sentence complexity is mapped; if the emotion evolution trend is slow decline of emotion intensity, stable negative emotion type, and context showing that the user is still pouring out, then the parameter combination of delayed response timing, medium dialogue initiative degree, medium emotional vocabulary density, and simple sentence complexity is mapped. Through the above mapping process, the corresponding dialogue strategy parameters and content generation parameters are finally generated.

[0054] At step 556, the dialogue strategy parameters are applied to the dialogue management process to determine the timing of the response and the guiding mode of the dialogue. Specifically, the response timing parameter in the dialogue strategy parameter is analyzed, if the parameter is marked as early, then after the current expression of the user is finished, the regular response waiting interval is not executed, and the response preparation process is immediately started; if the parameter is marked as normal, then the response preparation is started according to the preset regular waiting interval; if the parameter is marked as delayed, then the response preparation is started after the preset time period is extended on the basis of the regular waiting interval, ensuring that the user has sufficient expression space, then the dialogue initiative degree parameter is analyzed, if the parameter is marked as high, then the expansion guide related to the current user emotion and the dialogue topic is added to the response content; if the parameter is marked as medium, then the response content only echoes the core content expressed by the user, without adding additional expansion topics, and only maintaining the current dialogue direction; if the parameter is marked as low, then the response content only directly responds to the specific problem or appeal expressed by the user, without containing any guiding sentences, avoiding interfering with the subsequent expression of the user. Through the landing execution of the two parameters, the response timing and the dialogue guiding mode of the current interaction are finally determined.

[0055] Step 557, input the content generation parameters into the natural language generation process to control the emotional expression intensity and language complexity of the generated text; specifically including: parsing the emotional vocabulary density parameter in the content generation parameter, if the parameter is marked as high, preferentially select emotional vocabulary matching the current user emotional state in the natural language generation process, and the proportion of emotional vocabulary in the generated text is not less than a preset proportion, for example, when the user's emotion is happy, use happy, great, really comforting, inspiring, etc. If the parameter is marked as medium, use emotional vocabulary moderately, and the proportion is controlled within a preset intermediate proportion, taking into account the accuracy of semantic expression and the appropriateness of emotional transmission; if the parameter is marked as low, try to avoid using subjective emotional vocabulary in the generated text, and mainly use objective and simple descriptive vocabulary to convey core semantic information, then parse the sentence complexity parameter, if the parameter is marked as simple, the generated text mainly uses short sentences and simple subject-predicate-object structures, avoiding complex clauses and multiple modifiers, and the length of a single sentence is controlled within a preset short length range; if the parameter is marked as medium, combine short sentences and simple long sentences, allowing the use of a small amount of simple clauses, and the length of a single sentence is controlled within a preset medium range; if the parameter is marked as complex, multiple modifiers and complex clauses can be used appropriately to enrich the hierarchical structure of the sentence, and the length of a single sentence can be appropriately extended according to the needs of semantic expression. Through the above parameter control, the emotional expression intensity and language complexity of the generated text are accurately adapted to the current emotional state of the user.

[0056] Step 558, based on the dialogue management process and the natural language generation process, form a voice response consistent with the emotional state to achieve emotion adaptive response; specifically including: integrating response timing and dialogue guidance mode, as well as text content with adapted emotional expression intensity and language complexity, clearly defining the core elements of the voice response, inputting the integrated text content into the emotional speech synthesis process, and simultaneously calling the corresponding rhythm, tone and rhythm parameters of the comprehensive emotional state representation obtained through cross-modal fusion, starting the speech synthesis at the specified time point according to the requirements of the response timing: if the response timing is early, the synthesis is performed immediately after the text is generated; if it is normal or delayed, the synthesis is performed according to the corresponding time node, in the synthesis process, the emotional intensity of the text is combined to adjust the emotional concentration of the voice, and the language complexity is combined to adjust the pause position and duration of the voice, finally, the synthesized voice is delivered to the user through the voice output device. The voice not only meets the response timing and guidance logic required by the dialogue strategy, but also highly matches the current emotional state of the user in terms of text content, emotional intensity and language rhythm, thereby achieving emotion adaptive response.

[0057] By calculating the fluctuation parameters of the two emotional change modes respectively and combining semantic information calibration, the determination of the emotional state fluctuation interval is more comprehensive and accurate, and the ability to capture the user's real emotional state is improved.

[0058] AsFigure 2 As shown, the embodiment of the present application also provides an emotion adaptive multi-modal dialogue generation system based on biological signals, comprising: An extraction module is configured to process the collected biological signals and multi-modal interaction information, extract video spatiotemporal features, audio timing features and text semantic features; A fusion module is configured to perform cross-modal fusion on the video spatiotemporal features, audio timing features and text semantic features, obtain deep semantic connections between multi-modal data through fusion semantic analysis, and establish cross-modal semantic correspondence; utilize a sentiment consistency evaluation link to analyze the consistency degree of different modalities in emotional expression, and obtain a comprehensive emotional state representation; A generation module is configured to obtain dialogue text content consistent with the current emotional state of the user according to the comprehensive emotional state representation, in combination with the context information of multi-round dialogue, utilize a natural language generation model, and convert the dialogue text content into voice output through a voice conversion process; A recognition module is configured to monitor the dynamic changes of user biological signals and voice signals in real time during voice output, organize newly collected multi-modal data into a timing data stream, and construct an emotional state analysis benchmark; through detection of key turning points and extreme points in the timing data stream, extract key event points representing the boundaries of emotional states, and identify two emotional change patterns; An adjustment module is configured to analyze the two emotional change patterns, determine the fluctuation range of the emotional state in combination with the results of fusion semantic analysis, select emotional state reference instances to construct an emotional state change trajectory, generate emotional state dynamic adjustment parameters based on the emotional state change trajectory, and real-time adjust dialogue strategies and generated content, so as to realize emotion adaptive response.

[0059] It should be noted that the system is a system corresponding to the above method, and all implementation manners in the above method embodiment are applicable to this embodiment and can achieve the same technical effects.

[0060] The above is the preferred embodiment of the present application. It should be noted that for ordinary skilled persons in the technical field, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered within the protection scope of the present application.

Claims

1. A method for emotion-adaptive multimodal dialogue generation based on biosignals, characterized in that, The method includes: Step 1: Process the collected biological signals and multimodal interaction information to extract video spatiotemporal features, audio temporal features, and text semantic features; Step 2 involves cross-modal fusion of video spatiotemporal features, audio temporal features, and text semantic features. Through fusion semantic analysis, deep semantic connections between multimodal data are obtained, and cross-modal semantic correspondences are established. The emotional consistency assessment step is used to analyze the degree of consistency in emotional expression across different modalities, resulting in a comprehensive emotional state representation. Step 3: Based on the comprehensive emotional state representation and combined with the contextual information of the multi-turn dialogue, use a natural language generation model to obtain dialogue text content that matches the user's current emotional state, and convert it into speech output through a speech conversion process. Step 4: During the speech output process, monitor the dynamic changes of the user's biosignals and speech signals in real time, organize the newly acquired multimodal data into a time-series data stream, and construct a baseline for emotion state analysis. By detecting key turning points and extreme points in the time-series data stream, extract key event points representing the boundaries of the emotion state and identify two emotion change patterns. Specifically, this includes: dividing the time-series data stream into fixed-duration windows, calculating the time-domain and frequency-domain features of the biosignals within each window, and generating a high-dimensional feature sequence; performing first-order difference calculation on the high-dimensional feature sequence to obtain a feature change rate sequence, and detecting feature changes. Zero-crossing points and local extrema in the rate sequence are used to identify candidate key event points. These candidate key event points are then filtered using preset amplitude and duration thresholds, merging closely adjacent candidate key event points to form a key event point sequence. Based on this sequence, the time interval and characteristic amplitude difference between adjacent key event points are calculated. The time interval and characteristic amplitude difference are compared with preset thresholds to distinguish between abrupt and gradual change patterns in emotional states. Abrupt change pattern is defined as a short time interval with a large amplitude change, while a gradual change pattern is defined as a uniform time interval with a small amplitude change. Step 5: Analyze the two emotion change patterns, determine the fluctuation range of the emotion state by combining the results of the fusion semantic analysis, and select reference instances of the emotion state to construct the emotion state change trajectory; generate dynamic adjustment parameters of the emotion state based on the emotion state change trajectory, and adjust the dialogue strategy and generated content in real time to achieve adaptive emotion response.

2. The emotion-adaptive multimodal dialogue generation method based on biosignals according to claim 1, characterized in that, This approach involves cross-modal fusion of video spatiotemporal features, audio temporal features, and text semantic features. Through fused semantic analysis, deep semantic connections between multimodal data are obtained, and cross-modal semantic correspondences are established, including: Cross-modal semantic alignment is performed on video spatiotemporal features, audio temporal features, and text semantic features. The interaction and correlation strength between different modal features is analyzed through a multimodal attention process. Based on the interaction correlation strength, the features of each modality are fused to generate a semantically aligned cross-modal joint representation; The cross-modal joint representation based on semantic alignment constructs a semantic association matrix between modalities by calculating the semantic similarity between feature fragments of different modalities. The semantic association matrix is ​​analyzed to identify cross-modal semantic association patterns and establish structured cross-modal semantic correspondences.

3. The emotion-adaptive multimodal dialogue generation method based on biosignals according to claim 2, characterized in that, The consistency of emotion expression across different modalities is analyzed using an affective consistency assessment process to obtain a comprehensive representation of emotional state, including: Based on cross-modal semantic correspondence, corresponding visual sentiment features, acoustic sentiment features, and semantic sentiment features are extracted from video modality, audio modality, and text modality; Visual emotional features, acoustic emotional features, and semantic emotional features are discriminated to form emotional tendency descriptions for video modality, audio modality, and text modality. The sentiment tendency descriptions of video, audio, and text modalities are cross-compared to evaluate the intrinsic consistency among the sentiment tendency descriptions of different modalities. Based on the intrinsic consistency, a corresponding fusion contribution is determined for the sentiment tendency description of each modality. Based on the fusion contribution, the emotional tendency descriptions of each modality are proportionally fused to form a preliminary comprehensive emotional representation. The preliminary comprehensive emotional representation is then normalized to obtain the final comprehensive emotional state representation.

4. The emotion-adaptive multimodal dialogue generation method based on biosignals according to claim 3, characterized in that, Based on the comprehensive emotional state representation and combined with the contextual information from multiple rounds of dialogue, a natural language generation model is used to obtain dialogue text content that matches the user's current emotional state. This text is then converted into speech output through a speech-to-speech process, including: By fusing a comprehensive emotional state representation with a vectorized, multi-turn dialogue context, a contextual semantic representation is generated. Inputting the contextual semantic representation into a pre-trained natural language generation model generates dialogue text that matches the current emotional state in terms of semantic content and expression. The dialogue text and the comprehensive emotional state representation are input into the emotional speech synthesizer, and the corresponding prosody, pitch and rhythm parameters are mapped out according to the comprehensive emotional state. Based on prosody, pitch, and rhythm parameters, the dialogue text is processed by speech synthesis to obtain natural speech output that carries corresponding emotional expression.

5. The emotion-adaptive multimodal dialogue generation method based on biosignals according to claim 4, characterized in that, During voice output, the dynamic changes of the user's biosignals and voice signals are monitored in real time. The newly acquired multimodal data is organized into a time-series data stream to construct a baseline for emotion state analysis, including: The raw biological signals and speech signals collected continuously are extracted using a sliding time window of preset duration to obtain biological signal segments and speech signal segments within the current time window; Feature extraction is performed on biosignal and speech signal segments within the current time window to obtain real-time feature vectors representing the user's current physiological and speech states; The real-time feature vectors are concatenated and aligned with the preset reference feature sequence to form a unified and continuous time-series data stream; Aggregate analysis is performed on the reference feature vectors corresponding to the user's calm state interval in the time series data stream, the central trend of the numerical distribution of the reference feature vectors is calculated, and an emotional state baseline is constructed, i.e., the emotional state analysis baseline.

6. The emotion-adaptive multimodal dialogue generation method based on biosignals according to claim 5, characterized in that, Two emotion change patterns are analyzed, and the fluctuation range of emotion states is determined by combining the results of fusion semantic analysis. Emotion state change trajectories are constructed using reference instances of emotion states, including: Based on the mutation mode and the gradual mode, the time span and characteristic amplitude range of emotional state fluctuations corresponding to each mode are calculated to form preliminary fluctuation range parameters. The initial fluctuation range parameters are aligned with the emotional semantic information obtained from cross-modal fusion semantic analysis. The emotional semantic information is then used to correct and calibrate the boundaries of the fluctuation range to determine the final emotional state fluctuation range. Based on the emotional state fluctuation range, samples that match the emotional features and emotional semantic labels within the current fluctuation range are retrieved from the emotional state instance library, and reference emotional state instances are selected. By organizing reference examples of emotional states in chronological order and combining them with the established range of emotional state fluctuations, an emotional state change trajectory reflecting the continuous process of emotional state change is constructed.

7. The emotion-adaptive multimodal dialogue generation method based on biosignals according to claim 6, characterized in that, Based on the trajectory of emotional state changes, dynamic adjustment parameters for emotional state are generated to adjust the dialogue strategy and generated content in real time, achieving adaptive emotional response, including: By analyzing the trajectory of changes in emotional state, the rate of change in emotional intensity and the pattern of emotional type transition are extracted to form the characteristics of emotional evolution trends. Based on the characteristics of emotional evolution trends and combined with the contextual information of multi-turn dialogues, dialogue strategy parameters and content generation parameters are generated through preset mapping relationships. The dialogue strategy parameters include response timing and the degree of dialogue initiative, while the content generation parameters include emotional vocabulary density and sentence complexity. Apply dialogue strategy parameters to the dialogue management process to determine the timing of responses and the way the dialogue is guided. Input the content generation parameters into the natural language generation process to control the intensity of emotional expression and the complexity of language in the generated text; Based on the dialogue management process and the natural language generation process, a voice response that matches the emotional state is formed, thus achieving an emotion-adaptive response.

8. A biosignal-based emotion-adaptive multimodal dialogue generation system, wherein the system implements the method as described in any one of claims 1 to 7, characterized in that, include: The extraction module is used to process the collected biological signals and multimodal interaction information to extract video spatiotemporal features, audio temporal features, and text semantic features. The fusion module is used to perform cross-modal fusion of video spatiotemporal features, audio temporal features, and text semantic features. Through fusion semantic analysis, it obtains deep semantic connections between multimodal data and establishes cross-modal semantic correspondences. The sentiment consistency assessment step is used to analyze the degree of consistency in sentiment expression across different modalities to obtain a comprehensive emotional state representation. The generation module is used to obtain dialogue text content that matches the user's current emotional state based on the comprehensive emotional state representation and the context information of the multi-turn dialogue, and then convert it into speech output through the speech conversion process. The recognition module is used to monitor the dynamic changes of the user's biosignals and speech signals in real time during the speech output process, organize the newly acquired multimodal data into a time-series data stream, and construct a benchmark for emotion state analysis; by detecting key turning points and extreme points in the time-series data stream, it extracts key event points that represent the boundaries of the emotion state and identifies two emotion change patterns. The adjustment module analyzes two emotion change patterns, determines the fluctuation range of the emotion state by combining the results of fusion semantic analysis, and constructs the emotion state change trajectory by selecting reference instances of the emotion state. Based on the emotion state change trajectory, it generates dynamic adjustment parameters for the emotion state, and adjusts the dialogue strategy and generated content in real time to achieve adaptive emotion response.

Images