Voice emotion interaction method and system based on active questioning strategy

Abstract

This application discloses a voice-based emotional interaction method and system based on a proactive questioning strategy, relating to the fields of artificial intelligence and mental health technology. The method includes acquiring the user's input voice signal for the current round, processing it to obtain acoustic feature vectors and text feature vectors; concatenating the acoustic and text feature vectors, and calculating a single-round emotional state vector using a neural network; generating a dialogue context vector based on historical emotional state vectors from multiple rounds; calculating an action probability distribution vector based on the single-round emotional state vector and the dialogue context vector using a pre-trained policy network; selecting the action with the highest probability as the action for the current round based on the action for the current round, and generating and outputting the corresponding follow-up question voice content based on the action for the current round. This solves the problem in existing emotional interaction systems that cannot adjust based on real-time emotions and dialogue history, and can select an appropriate follow-up question strategy based on the user's emotional state and the progress of the dialogue.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and mental health technology, specifically to a voice-based emotional interaction method and system based on a proactive questioning strategy. Background Technology

[0002] In the fields of artificial intelligence and mental health assistance technology, emotion recognition and intervention through voice interaction are often applied in scenarios such as mental health support, intelligent customer service, and emotional companionship robots. The core objective is to identify the user's emotional state and make appropriate responses to guide the conversation to go deeper.

[0003] Existing technical solutions typically follow a process of extracting acoustic / textual features from speech signal processing, followed by emotion recognition, and finally generating a response. Current speech emotion recognition and interaction methods usually first process the user's speech signal to extract features representing emotions, then combine this with text information obtained from automatic speech recognition technology to comprehensively determine the user's current emotional state. Subsequently, the system generates or selects predefined response content based on certain rules or models, using historical dialogue information and the currently identified emotion.

[0004] However, when making follow-up questioning strategy decisions, this technical method mainly relies on the abstract summary of historical dialogue sequences to predict the next action. It fails to effectively align and utilize the emotional state vector that deeply integrates acoustic and textual information in the current round with the historical context. This results in insufficient matching between the follow-up questioning strategy selected by the system and the emotional cues in the user's current expression, leading to insufficient accuracy of the follow-up questioning strategy. It is difficult to guide users to make deeper emotional expressions and explorations at key moments, thus affecting the support effect of the interaction. Summary of the Invention

[0005] The purpose of this application is to provide a voice emotion interaction method and system based on an active questioning strategy to solve the problems mentioned in the background art.

[0006] In a first aspect, one embodiment of this application provides a voice emotion interaction method, which includes: acquiring the current round voice signal input by the user, and processing the current round voice signal to obtain an acoustic feature vector and a text feature vector; concatenating the acoustic feature vector and the text feature vector, and calculating a single-round emotion state vector through a neural network; generating a dialogue context vector based on the emotion state vectors of previous rounds, the dialogue context vector being used to represent the dialogue evolution process; calculating an action probability distribution vector based on the single-round emotion state vector and the dialogue context vector through a pre-trained policy network; taking the action with the highest probability as the action of the current round based on the action of the current round, generating and outputting corresponding follow-up question voice content based on the action of the current round, the action of the current round being used to represent an active follow-up question strategy to guide the user to further express emotions.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, based on the single-round emotion state vector and context vector, an action probability distribution vector is calculated through a pre-trained policy network. This includes: concatenating the current emotion state vector, context vector, and action from the previous round to obtain a state representation vector; mapping the state representation vector through a non-linear activation function to obtain a decision state vector; inputting the decision state vector into the policy network, where multiple hidden layers of the policy network perform feature transformation to obtain hidden layer output vectors; passing the last hidden layer output vector through the linear output layer of the policy network to generate an action score for each action; and calculating the action probability distribution vector based on the action scores.

[0008] In conjunction with the first aspect, in certain implementations of the first aspect, a dialogue context vector is generated based on the historical emotional state vectors from multiple rounds. This includes: concatenating the historical emotional state vectors and corresponding actions from multiple rounds according to round to form a historical feature sequence; passing the historical feature sequence through the forgetting gate of a Long Short-Term Memory (LSTM) network to calculate the forgetting gate vector; passing the historical feature sequence through the input gate of the LSTM network to calculate the input gate vector and candidate cell state vector; updating the cell state based on the forgetting gate vector, input gate vector, and candidate cell state vector; calculating the output gate vector and hidden state based on the updated cell state through the output gate of the LSTM network; using the hidden state of the previous layer as the historical feature sequence of the next layer, and through multi-layer cyclic calculation of the LSTM network, using the last hidden state of the last layer as the dialogue context vector.

[0009] In conjunction with the first aspect, in certain implementations of the first aspect, the current round of speech signal input by the user is acquired, and the current round of speech signal is processed to obtain acoustic feature vectors and text feature vectors, including: acquiring the current round of speech signal input by the user to obtain a pulse code modulation digital sequence; performing filtering and framing operations on the pulse code modulation digital sequence to obtain a frame sequence; applying a fast Fourier transform to the data at each frequency point of each frame based on the frame sequence to obtain a frequency domain complex sequence; calculating the square of the modulus of the frequency domain complex sequence to obtain the power spectrum at each frequency point; constructing a Mel filter on the power spectrum to calculate the energy sequence; calculating multiple acoustic feature values ​​based on the power spectrum and the energy sequence, including fundamental frequency, fundamental frequency probability, perceived loudness, and spectral flux; calculating the statistics for each acoustic feature value based on the acoustic feature values, and constructing an acoustic feature vector from the acoustic feature values ​​and statistics, including the mean, standard deviation, quantiles, and extreme values; generating a text sequence based on the energy sequence, and processing the text sequence through a multi-layer encoder to obtain a text feature vector.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, a text sequence is generated based on the energy sequence, and the text sequence is processed by a multi-layer encoder to obtain a text feature vector, including: generating a text sequence based on the energy sequence using an ASR encoder and an ASR decoder; segmenting the text sequence into sub-word tags, adding special tags to the beginning and end of the sub-word tags, the special tags being used to represent the instruction signals for the system to understand sentence structure and semantics; calculating the input vector of each sub-word tag based on the sub-word tags to form a tag input sequence; processing the tag input sequence through a multi-head self-attention layer of a multi-layer encoder to obtain the output vector of each attention head; concatenating the output vectors of multiple attention heads, inputting them again into the multi-head self-attention layer, and after processing through different layers, obtaining the final context-related representation of each sub-word tag, and using the final layer output vector corresponding to the special tag as the text feature vector.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the corresponding follow-up question voice content is generated and output according to the action of the current round, including: randomly selecting a text template from the follow-up question template library of the corresponding action according to the action of the current round, the text template including placeholders; filling the placeholders in the text template with the dialogue context vector, and outputting the text template.

[0012] In conjunction with the first aspect, some implementations of the first aspect also include: recording the user's next round of single-round emotional state vector and action after executing the policy network; evaluating the immediate reward of the policy network based on the next round of single-round emotional state vector and action; and updating the weight matrix and bias vector of the policy network based on the immediate reward.

[0013] Secondly, this application provides a voice emotion interaction system, comprising: a signal processing module for acquiring the current round voice signal input by the user and processing the current round voice signal to obtain an acoustic feature vector and a text feature vector; a feature processing module for concatenating the acoustic feature vector and the text feature vector and calculating a single-round emotion state vector through a neural network; a context generation module for generating a dialogue context vector based on the emotion state vectors of multiple historical rounds, the dialogue context vector being used to represent the dialogue evolution process; a strategy decision module for calculating an action probability distribution vector based on the single-round emotion state vector and the dialogue context vector through a pre-trained strategy network; and a voice interaction module for selecting the action with the highest probability as the action of the current round based on the action probability distribution vector, generating and outputting corresponding follow-up question voice content based on the action of the current round, the action of the current round being used to represent an active follow-up question strategy to guide the user to further express emotions.

[0014] Thirdly, this application provides an electronic device, including: a processor; and a memory storing computer program instructions, which, when executed by the processor, implement the steps of the voice emotion interaction method mentioned in the first aspect above.

[0015] Fourthly, this application provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor, cause the processor to perform the steps in the voice emotion interaction method mentioned in the first aspect.

[0016] The voice-based emotional interaction method provided in this application acquires and processes user voice signals to obtain a single-turn emotional state vector that integrates acoustic and textual features. Combined with a context vector generated based on historical dialogue, a pre-trained policy network dynamically determines the optimal proactive questioning strategy. This solves the problem in existing emotional interaction systems where questioning strategies are fixed and formulaic, failing to provide personalized and context-aware guidance based on real-time emotions and dialogue history. It achieves the ability to adaptively select the most suitable questioning strategy based on the user's current subtle emotional state and the overall dialogue progress, thereby more effectively guiding users to express emotions more deeply and improving interaction depth and support effectiveness. Attached Figure Description

[0017] Figure 1 The diagram shown is a flowchart of a voice emotion interaction method provided in an exemplary embodiment of this application.

[0018] Figure 2 The diagram shown is a flowchart of a voice emotion interaction method provided in another exemplary embodiment two of this application.

[0019] Figure 3The diagram shown is a flowchart of a voice emotion interaction method provided in another exemplary embodiment three of this application.

[0020] Figure 4 The diagram shown is a flowchart of a voice emotion interaction method provided in another exemplary embodiment four of this application.

[0021] Figure 5 The diagram shown is a flowchart of a voice emotion interaction method provided in another exemplary embodiment five of this application.

[0022] Figure 6 The diagram shown is a flowchart of a voice emotion interaction method provided in another exemplary embodiment six of this application.

[0023] Figure 7 The diagram shown is a flowchart of a voice emotion interaction method provided in another exemplary embodiment seven of this application.

[0024] Figure 8 The diagram shown is an architectural schematic of a voice emotion interaction system provided in an exemplary embodiment of this application.

[0025] Figure 9 The diagram shown is a structural schematic of an electronic device provided in an exemplary embodiment of this application. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] The following is combined Figures 1 to 7 This application provides a detailed description of a voice-emotion interaction method.

[0028] Figure 1 The diagram shown is a flowchart illustrating an exemplary embodiment of this application regarding a voice-based emotional interaction method. Figure 1 As shown in Embodiment 1 of this application, the voice emotion interaction method includes the following steps.

[0029] Step 100: Obtain the current round of speech signal input by the user, and process the current round of speech signal to obtain acoustic feature vector and text feature vector;

[0030] For example, the current round of voice signal refers to the raw voice signal collected from the audio input stream interface after the user interacts with the system in this round.

[0031] For example, an acoustic feature vector is a statistical report that describes the acoustic characteristics of an entire speech. It no longer focuses on the specific value of the sound at a certain moment, but rather summarizes the overall pattern and range of variation of the entire speech in terms of pitch, loudness, timbre, etc.

[0032] For example, a text feature vector refers to a dense vector representation containing rich semantic and contextual information extracted from a text sequence converted from user speech recognition through a multi-layer encoder.

[0033] Step 102: Concatenate the acoustic feature vector and the text feature vector, and calculate the single-round emotional state vector through a neural network;

[0034] For example, a neural network refers to the nonlinear fusion and dimensionality reduction of concatenated vectors.

[0035] For example, a single-round emotion state vector refers to a unified high-dimensional real vector generated by deeply fusing and encoding the acoustic feature vector and text feature vector of the user in the current round through the aforementioned neural network.

[0036] Step 104: Generate a dialogue context vector based on the emotional state vectors from multiple historical rounds. The dialogue context vector is used to represent the dialogue evolution process.

[0037] For example, the dialogue context vector refers to a vector extracted by using a long short-term memory network model to perform temporal modeling on the single-turn emotional state vectors and system action sequences generated in historical multi-turn dialogues.

[0038] Step 106: Based on the single-turn emotion state vector and the dialogue context vector, the action probability distribution vector is calculated through a pre-trained policy network;

[0039] For example, a policy network refers to an Actor network based on the Actor-Critic architecture, which takes a decision state vector as input and outputs an action probability distribution vector.

[0040] For example, the action probability distribution vector refers to the probability of each action type occurring, and the sum of all action probability distribution vectors is 1.

[0041] Step 108: Based on the action probability distribution vector, the action with the highest probability is taken as the action of the current round. Based on the action of the current round, the corresponding follow-up question voice content is generated and output. The action of the current round is used to represent the proactive follow-up question strategy to guide the user to further express emotions.

[0042] For example, an action refers to a type of proactive questioning strategy output by the policy network to guide the user's emotional expression, and is a discrete category label.

[0043] The voice-based emotional interaction method provided in this application acquires and processes user voice signals to obtain a single-turn emotional state vector that integrates acoustic and textual features. Combined with a context vector generated based on historical dialogue, a pre-trained policy network dynamically determines the optimal proactive questioning strategy. This solves the problem in existing emotional interaction systems where questioning strategies are fixed and formulaic, failing to provide personalized and context-aware guidance based on real-time emotions and dialogue history. It achieves the ability to adaptively select the most suitable questioning strategy based on the user's current subtle emotional state and the overall dialogue progress, thereby more effectively guiding users to express emotions more deeply and improving interaction depth and support effectiveness.

[0044] Figure 2 The diagram shown is a flowchart illustrating a voice emotion interaction method provided in another exemplary embodiment two of this application. Figure 1 This application extends from the embodiments shown. Figure 2 The illustrated embodiment will be described in detail below. Figure 2 The illustrated embodiments and Figure 1 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0045] like Figure 2 As shown, in the voice emotion interaction method provided in Embodiment 2 of this application, the action probability distribution vector is calculated based on the single-round emotion state vector and the context vector through a pre-trained policy network, including the following steps.

[0046] Step 200: Concatenate the current emotion state vector, the context vector, and the action from the previous round to obtain the state representation vector;

[0047] Specifically, the emotional state vector V of the current round t Dialogue context vector C t And the action A from the previous round t-1 Concatenate to form a state representation vector Concat(V) t C t A t-1 ).

[0048] Step 202: Based on the state representation vector, obtain the decision state vector through mapping using a nonlinear activation function;

[0049] Specifically, a fully connected layer maps the concatenated state representation vector to the representation space required for decision-making, resulting in the decision state vector S. t .

[0050] ;

[0051] Here, the tanh function represents the hyperbolic tangent activation function, expressed as tanh(x) = (e^(x-1) / x) x -e -x ) / (e x +e -x );w S b represents the weight matrix of the fully connected layer. S This represents the bias vector of the fully connected layer.

[0052] Step 204: Input the decision state vector into the policy network, and perform feature transformation by multiple hidden layers of the policy network to obtain the hidden layer output vector;

[0053] Specifically, first, a policy network is preset, with its action space size set to K and the number of hidden layers set to N. h .

[0054] For example, a policy network refers to an Actor network based on the Actor-Critic architecture, which takes a decision state vector as input and outputs an action probability distribution vector.

[0055] For example, the action space refers to the types of actions taken by the system in the policy network. In this embodiment, the action space is set to K=4, including empathy, clarification, exploring bodily sensations, and facilitating expression.

[0056] Next, based on the decision state vector S t Feature transformation is performed through the hidden layer of the policy network to obtain the hidden layer output vector HA. i .

[0057] ;

[0058] Where i represents the index number of the hidden layer, i = {1, 2, ..., N} h The ReLU function represents a non-linear activation function, expressed as ReLU(x) = max(0, x); w i b i Let represent the weight matrix and bias vector of the i-th hidden layer, respectively.

[0059] Step 206: Pass the output vector of the last hidden layer through the linear output layer of the policy network to generate an action score for each action;

[0060] Specifically, the output vector HA of the last hidden layer Nh A linear output layer is used to generate an action score Z[k] for each action.

[0061] Z[k] = w z ×HA Nh+b z ;

[0062] Where k represents the index number of the action space, k={1, 2, ..., K}; w z b z These represent the weight matrix and bias vector of the linear output layer, respectively.

[0063] Step 208: Calculate the action probability distribution vector based on the action scores.

[0064] Specifically, based on the action score Z[k], the Softmax function is used to convert the action score Z[k] into an action probability distribution vector P. t [k].

[0065] ;

[0066] For example, P t [k] represents the probability distribution vector of the k-th action; the probability distribution vector of an action refers to the probability of each action type occurring, and the sum of the probability distribution vectors of the k actions is 1.

[0067] For example, the policy network receives a decision state vector of current emotion and dialogue history, evaluates each potential probing strategy through a nonlinear mapping, calculates the action score in the current context, and transforms it into a probability distribution using Softmax. The system then selects the action A with the highest probability based on the probability distribution. t This generates the next follow-up question. The weight matrix and bias vector of the policy network are continuously optimized during training using the policy gradient method, employing different follow-up strategies to address various states.

[0068] The voice emotion interaction method provided in this application concatenates the current emotion state vector, context vector, and previous action to obtain a state representation vector, and then obtains the action probability distribution through nonlinear mapping and multi-layer transformation of the policy network, further defining the specific decision-making process of the policy network. This technique solves the problem of the policy network decision-making mechanism being too general, enabling the decision-making process to more finely integrate multi-dimensional information such as the immediate state, historical context, and executed actions, thereby improving the accuracy and contextual relevance of the follow-up strategy decision.

[0069] Figure 3 The diagram shown is a flowchart illustrating a voice emotion interaction method provided in another exemplary embodiment three of this application. Figure 1 This application extends from the embodiments shown. Figure 3 The illustrated embodiment will be described in detail below. Figure 3 The illustrated embodiments and Figure 1 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0070] like Figure 3 As shown, in the voice emotion interaction method provided in Embodiment 3 of this application, a dialogue context vector is generated based on the emotion state vectors from multiple historical rounds, including the following steps.

[0071] Step 300: Based on the emotional state vectors from multiple historical rounds and the actions in the corresponding rounds, the vectors are concatenated by round to form a historical feature sequence;

[0072] Specifically, firstly, based on the acoustic feature vector V1[M][7] and the text feature vector V2, they are projected onto the same dimensional space through two independent linear transformations to obtain the transformed acoustic feature vector V1' and the transformed text feature vector V2'.

[0073] V1' = w a ×V1[M][7]+b a ;

[0074] V2' = w t ×V2+b t ;

[0075] Among them, w a w t b represents the weight matrix of acoustic features and the weight matrix of text features, respectively; a b t These represent the bias vectors for acoustic features and text features, respectively.

[0076] Next, the transformed text feature vector V2' is used as the query vector Q, and the transformed acoustic feature vector V1' is used as the key vector K and value vector V. For each dimension position of the transformed text feature vector, the similarity score Score between it and all dimensions positions of the transformed acoustic feature vector is calculated.

[0077] ;

[0078] Among them, w Q w K d represents the weight matrix of the query vector and the weight matrix of the key vector, respectively; K This represents the dimension of the key vector.

[0079] Secondly, similarity scores are used as weights to perform a weighted summation of the transformed acoustic feature vector values ​​to obtain the text query-based acoustic context vector C. audio .

[0080] C audio = Scores×(V1'×w V );

[0081] Next, the transformed acoustic feature vector V1' is compared with the calculated acoustic context vector C. audio The concatenation is performed to obtain the concatenated fused feature vector Concat(V1', C audio The feature vector will be fused using Concat(V1', C). audio Inputting a feedforward neural network, performing layer normalization and residual connections, yields a single-round emotional state vector V. t .

[0082] Finally, obtain the single-round sentiment state vector {V} for the most recent k rounds. t-k V t-(k+1) , ..., V t-2 V t-1} and the actions taken by the system {A t-k A t-(k+1) , ..., A t-2 A t-1}, concatenate the single-round emotional state vectors of the most recent k rounds and the actions taken by the system according to round, forming a historical feature sequence VA = {(V t-k A t-k ), (V t-(k+1) A t-(k+1) ), ...,(V t-2 A t-2 ), (V t-1 A t-1 )}.

[0083] For example, k represents the number of times the user interacts with the system. In this embodiment, k=5. The action taken by the system refers to the follow-up questioning strategy selected by the system in each interaction. For example, if the user answers "I feel a little anxious", the system analyzes the emotions and follow-up questioning strategies in the historical dialogue and gives different types of responses. Follow-up questioning strategies include empathy, clarification, exploring bodily feelings and promoting expression.

[0084] Step 302: Pass the historical feature sequence through the forgetting gate of the Long Short-Term Memory network to calculate the forgetting gate vector;

[0085] Specifically, based on the historical feature sequence VA, the forgetting gate vector F is calculated using the forgetting gate of the Long Short-Term Memory (LSTM) network model. t .

[0086] ;

[0087] Among them, F t This represents the forgetting gate vector for the current round, with a value between 0 and 1; the Sigmoid function represents the activation function; wF H represents the weight matrix of the forget gate; t-1 Indicates the hidden state of the previous round; VA t Represents the historical feature sequence of the current round; b F This represents the bias vector of the forget gate.

[0088] Step 304: Based on the historical feature sequence, calculate the input gate vector and candidate cell state vector through the input gate of the long short-term memory network;

[0089] Specifically, based on the historical feature sequence VA, the input gate vector I is calculated through the input gate of the LSTM model. t And the candidate cell state vector Tilde(c) t .

[0090] ;

[0091] ;

[0092] Among them, I t This represents the input gate vector for the current round; Tilde(c) t The vector represents the candidate cell state vector for the current round; the tanh function represents the hyperbolic tangent activation function; w I w c b represents the weight matrices of the input gate and the candidate state, respectively; I b c These represent the bias vectors of the input gate and the candidate state, respectively.

[0093] Step 306: Update the cell state based on the forget gate vector, the input gate vector, and the candidate cell state vector;

[0094] Specifically, according to the forgetting gate vector F t Input gate vector I t And the candidate cell state vector Tilde(c) t Update cell state c t .

[0095] ;

[0096] Among them, c t-1 This indicates the cell state in the previous round; This indicates element-wise multiplication.

[0097] Step 308: Based on the updated cell state, calculate the output gate vector and hidden state through the output gate of the Long Short-Term Memory network;

[0098] Specifically, based on cell state c tThe output gate vector O is calculated through the output gate of the LSTM model. t and hidden state H t .

[0099] ;

[0100] ;

[0101] Among them, O t H represents the output gate vector for the current round; t Indicates the hidden state in the current round; w O b represents the weight matrix of the output gate; O This represents the bias vector of the output gate.

[0102] Step 310: The hidden state of the previous layer is used as the historical feature sequence of the next layer. Through multi-layer cyclic calculation of the Long Short-Term Memory Network, the last hidden state of the last layer is used as the dialogue context vector.

[0103] Specifically, the number of layers in the LSTM model is set to NL, and the hidden states of the previous layer are H. t As the historical feature sequence of the next layer, the calculation formulas for the forget gate, input gate, candidate cell state, and output gate are executed iteratively for each layer, and the last hidden state H of the last layer is calculated. t As the final dialogue context vector C t .

[0104] For example, through the gating mechanism of the LSTM model, important historical feature sequences are selectively remembered while irrelevant information is forgotten, and the current emotional state vector is incorporated into the memory. Therefore, the dialogue context vector is a dynamic and condensed summary of the dialogue's emotional history.

[0105] The voice emotion interaction method provided in this application concatenates the emotion state vectors from multiple historical rounds with the corresponding actions in turn to form a sequence, and uses the gating mechanism of Long Short-Term Memory (LSTM) network to process the sequence cyclically to generate a dialogue context vector. This concretizes the method for generating context vectors and solves the problem of how to effectively model and compress the complex temporal dependencies between emotions and actions in multi-turn dialogues. By using LSTM to selectively remember important historical information and forget irrelevant information, it achieves the effect of generating a condensed context vector that can accurately represent the dynamic evolution of dialogue emotions.

[0106] Figure 4 The diagram shown is a flowchart illustrating a voice emotion interaction method provided in another exemplary embodiment four of this application. Figure 1 This application extends from the embodiments shown. Figure 4 The illustrated embodiment will be described in detail below. Figure 4 The illustrated embodiments and Figure 1 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0107] like Figure 4 As shown in Embodiment 4 of this application, the voice emotion interaction method obtains the current round voice signal input by the user and processes the current round voice signal to obtain acoustic feature vector and text feature vector, including the following steps.

[0108] Step 400: Obtain the current round of voice signal input by the user to obtain a pulse code modulation digital sequence; perform filtering and framing operations on the pulse code modulation digital sequence to obtain a frame sequence.

[0109] Specifically, firstly, an analog voltage signal is continuously read from the audio input stream interface. This continuous analog voltage signal is then sampled and quantized by an analog-to-digital converter at a fixed sampling rate K to obtain the pulse-code modulation digital sequence A1[n]=[(T0,a0), (T1,a1), ..., (T...]. n-1 ,a n-1 )];

[0110] Here, the pulse code modulation digital sequence A1[n] is a one-dimensional array, and each element of the array includes a sampling point T. i and the amplitude value a of the sampling point i The amplitude value of each sampling point is quantized as a 16-bit integer.

[0111] For example, the sound waves in the real world are converted into a digital form that a computer can store and process. If the sampling rate is 16kHz, it means that 16,000 points are collected per second. n=16,000 is used to characterize the highest frequency that can be captured, and the quantization bits are used to characterize the dynamic range of the amplitude value.

[0112] Next, a first-order high-pass digital filter is applied to the pulse code modulation digital sequence A1[n] to perform a filtering operation, and the filtered amplitude value a is obtained. i '.

[0113] a i '=a i -α×a i-1 ;

[0114] Among them, a i ' represents the filtered amplitude value of the current sampling point i, i=(0, 1, 2, ..., n-1); a i α represents the amplitude value of the current sampling point i; α represents the filtering coefficient, and in this embodiment, α=0.97 is set.

[0115] For example, the energy of the high-frequency part of a speech signal is usually lower than that of the low-frequency part. High-pass filtering is used to balance the spectrum, making the high-frequency features more prominent and improving the performance of subsequent acoustic feature extraction and speech recognition.

[0116] Secondly, the n sampling points {T0, T1, ..., T... n-1} and the corresponding filtered amplitude values ​​{a0', a1',..., a n-1 The filtered digital sequence A1'[n] = [(T0, a0'), (T1, a1'), ..., (T...] n-1 , a n-1 ')].

[0117] Next, based on the filtered digital sequence A1'[n], a framing operation is performed, dividing it into multiple frames A2[N] with frame length N of 20~40ms and frame shift of 10~15ms. N-1 , a N-1 ')].

[0118] For example, the frame length represents the number of sampling points contained in the frame. If the frame length N is set to 25ms and the sampling rate K = 16kHz, then there are 400 sampling points. The frame shift represents the time difference between the start points of two adjacent frames. If the frame shift is 10ms and the sampling rate is 16kHz, then there are 160 sampling points between the start points of two adjacent frames.

[0119] Finally, multiple frames A2[N] are combined into a frame sequence A2[M][N]; where each row represents the data of one frame and each column represents the number of frames.

[0120] For example, M represents the number of frames. If the frame length N = 400 sampling points and the sampling rate K = 16kHz, the calculation process is M = floor((16000-400) / 160)+1 = floor(15600 / 160)+1 = 97+1 = 98, where the function floor(x) represents rounding x.

[0121] It should be understood that speech signals are short-term stationary, meaning their characteristics remain essentially unchanged within tens of milliseconds. Framing transforms non-stationary long-term signals into multiple short-term stationary signal segments for processing, which is an important process in speech processing. Frame shift is less than frame length, indicating that there is overlap between each short frame, which is used to avoid the loss of frame edge information and ensure smooth transitions between frames.

[0122] Step 402: Based on the frame sequence, apply Fast Fourier Transform to the data at each frequency point of each frame to obtain a frequency domain complex sequence.

[0123] Specifically, firstly, based on the frame length N, a window function is set for each sampling point j in each frame to obtain the window function coefficients w[j], j={0, 1, 2, ..., N-1}.

[0124] ;

[0125] Where w[j] represents the coefficient of the window function at the j-th sampling point.

[0126] For example, the window function set in this application is a Hamming window, which has the characteristic of a curve that is smooth at both ends and convex in the middle.

[0127] Next, based on the frame sequence A2[M][N] and the window function coefficient w[j], a windowing operation is performed on the sampling points of each frame in the frame sequence, and all the windowed frames are combined into a windowed frame sequence A2'[M][N].

[0128] ;

[0129] Where A2'[i][j] represents the amplitude value of the j-th windowed sampling point of the i-th frame in the frame sequence, i={0, 1,2, ..., M-1}, j={0, 1, 2, ..., N-1}.

[0130] For example, directly truncating a frame of signal will introduce high-frequency spurious spectrum. Multiplying by a window function, which is smooth at both ends and convex in the middle, can gradually reduce the signal amplitude at both ends of the frame to zero, making the signal within the frame continuous at the boundary, thereby reducing spectral leakage and making the subsequent frequency domain analysis results more accurate.

[0131] Secondly, based on the windowed frame sequence A2'[M][N], apply Fast Fourier Transform to the sampling points of each frame in the windowed frame sequence to obtain the frequency domain complex number A3[m][k] of the m-th frame and the k-th frequency.

[0132] ;

[0133] Where A3[m][k] represents the complex number of the m-th frame and k-th frequency component obtained after transformation, m={0, 1, 2,..., M-1}, k = {0, 1, ..., N-1}; j represents the imaginary unit; Represents the complex sinusoidal basis functions, characterizing the components at different frequencies;

[0134] It should be understood that frequency k and sampling rate K are not the same frequency. Frequency k is an index number, an integer from 0 to N-1, representing the position in the Fast Fourier Transform output array; it has no physical unit. Sampling rate K is a parameter used when digitizing the signal, with the unit being Hertz (Hz), for example, 16000 Hz, which represents the number of points sampled per second for the analog signal. Each frequency k corresponds to a specific physical frequency value f. k Its calculation formula is f k =k×(K / N).

[0135] For example, the Fast Fourier Transform (FFT) refers to converting a time-domain discrete signal (windowed frame sequence A2'[M][N]) with frame length N and sampling rate K into a frequency-domain complex sequence of length N.

[0136] Finally, the frequency domain complex numbers A3[m][k] of multiple frames are combined into a frequency domain complex number sequence A3[M][N].

[0137] For example, the data stored in the frequency domain complex sequence represents the spectrum of the corresponding frame signal, the magnitude (amplitude) of A3[m][k] is used to characterize the intensity of the sampling point corresponding to frequency k, and the argument is used to characterize the phase of the sampling point corresponding to frequency k.

[0138] Step 404: Calculate the square of the modulus of the complex sequence in the frequency domain to obtain the power spectrum at each frequency point, construct a Mel filter on the power spectrum, and calculate the energy sequence.

[0139] Specifically, firstly, based on the frequency domain complex sequence A3[M][N], for each frame m and each frequency k, the square of the amplitude of the frequency domain complex A3[m][k] is calculated to obtain the power spectrum P[m][k] at that frequency point.

[0140] ;

[0141] Where real(A3[m][k]) represents the real part of the frequency domain complex number A3[m][k]; imag(A3[m][k]) represents the imaginary part of the frequency domain complex number A3[m][k].

[0142] For example, typically only the first (N / 2+1) sampling points are taken because the spectrum of a real signal is conjugate symmetric.

[0143] Next, based on the power P[m][k], a square root operation is performed to obtain the modulus |A3[m][k]| of the complex number in the frequency domain. The modulus |A3[m][k]| of the complex number in the frequency domain of each frame m and each frequency k is used to form an amplitude spectrum sequence B[M][N].

[0144] For example, the modulus of a complex number in the frequency domain, also called the amplitude spectrum, is the square root of the power spectrum. It is the first step in converting the complex spectrum into a real number representation, preparing for subsequent Mel-scale filtering.

[0145] Secondly, in the frequency axis k={0, 1, 2, ..., N / 2+1}, X Mel filters are preset. The specific Mel filter construction process includes the following steps.

[0146] For example, the frequency axis refers to a linear scale starting from 0Hz and ending at half the sampling rate. Combined with the frequency k set in the Fast Fourier Transform, N complex frequency points are obtained. The first (N / 2+1) frequency points correspond to the positive frequency components starting from 0Hz and ending at half the sampling rate. Each frequency k corresponds to a specific physical frequency value f. k Its calculation formula is f k =k×(K / N).

[0147] For example, the number of Mel filters, X, is a parameter preset based on experience, task requirements, and computing resources. The smaller the value of X, the coarser the Mel band, the less computation required, but the lower the frequency resolution. The larger the value of X, the higher the frequency resolution, the more spectral details can be preserved, but the computational cost increases, and the feature dimension is higher. The value of X ranges from 20 to 40, and in this embodiment, X=40 is set.

[0148] Step 1: Within the frequency axis k = {0, 1, 2, ..., N / 2+1}, use the Mel frequency calculation formula to calculate the physical frequency value f corresponding to frequency k. k Convert to Mel frequency mel(f) k ).

[0149] ;

[0150] For example, the physical frequency value f corresponding to frequency k k Arranged on the coordinate axes, these are multiple discrete points; therefore, the converted Mel frequency mel(f) k ( ) is also a discrete point.

[0151] Step 2: Combine multiple discrete Mel frequencies mel(f) k After arranging them on the coordinate axes, select (X+2) Mel frequencies evenly.

[0152] For example, X Mel filters require (X+2) boundary points.

[0153] Step 3: Convert the (X+2) Mel frequencies back to the frequency axis using the inverse formula of Mel frequency to obtain (X+2) frequencies f(X).

[0154] Step 4: Based on (X+2) frequencies f(X) and the physical frequency value f corresponding to frequency k. k Calculate the response function H of the x-th Mel filter. x [k].

[0155] ;

[0156] Where x = {1, 2, ..., X}; f(x-1) represents the frequency corresponding to the frequency axis when the Mel frequency (x-1) is converted back to the frequency axis using the inverse formula of the Mel frequency.

[0157] Step 5: Multiply the amplitude spectrum |A3[m][k]| of each frame m and each frequency k by the response function H of each Mel filter x at frequency k. x [k], and sum over all frequencies k to obtain the energy E[x] of the x-th Mel filter channel.

[0158] ;

[0159] For example, linear frequencies are converted to Mel frequencies and band smoothing is performed, reducing the data dimension from (N / 2+1) frequency points to X filter channels, while highlighting low-frequency information that is more important for speech perception.

[0160] It should be understood that the characteristics of Mel frequency are compression in the low-frequency region and stretching in the high-frequency region. When points evenly distributed on the Mel frequency are mapped back to the linear frequency axis, the result is that the filter is dense in the low-frequency region and sparse in the high-frequency region on the linear frequency axis, perfectly simulating the physiological characteristics of the human ear that is more sensitive to low-frequency changes and less sensitive to high-frequency changes.

[0161] Next, based on the energy E[x] of the x-th Mel filter channel, take the logarithm to the base 10 of the energy value of each Mel filter to obtain the compressed energy E'[x], and then construct an energy sequence E from the compressed energies E'[x] of all Mel filters. r [X]={E'[1], E'[2], ..., E'[X]}.

[0162] E'[x] = log10(E[x]+ε);

[0163] Wherein, ε represents a preset constant used to prevent taking the logarithm of 0. In this embodiment, ε is set to 1e-6.

[0164] For example, the human ear perceives sound intensity in an approximate logarithmic manner. Taking the logarithm is used to simulate this characteristic, compressing the dynamic range of energy values ​​and making the feature distribution closer to a Gaussian distribution, which is beneficial for the training and convergence of subsequent neural network models.

[0165] Finally, the energy sequence E of all frames r [X], arranged in chronological order, forms a two-dimensional energy matrix E. t [M][X].

[0166] Step 406: Calculate multiple acoustic characteristic values ​​based on the power spectrum and energy sequence. The acoustic characteristic values ​​include fundamental frequency, fundamental frequency probability, perceived loudness, and spectral flux.

[0167] Specifically, firstly, on the frequency axis k = {0, 1, 2, ..., N / 2+1}, the basic frequency estimation algorithm is used to find the frequency of the signal waveform that is most similar to the waveform corresponding to frequency k, and the time delay difference between the two is set to T. c According to the delay time difference T c Calculate the fundamental frequency F0 = 1 / T c And the fundamental frequency probability p(F0).

[0168] For example, the fundamental frequency probability is used to characterize the probability value of the reliability of the fundamental frequency estimated by the fundamental frequency estimation algorithm.

[0169] Next, based on the power spectrum P[m][k] of each frame m and each frequency k, the perceived loudness L is calculated using a psychoacoustic model. oud .

[0170] ;

[0171] Where band(i) represents the frequency after the frequency axis is mapped to the Barker frequency, and i is the frequency index number; w2(k) represents the psychoacoustic weight at frequency k.

[0172] For example, a psychoacoustic model refers to mapping the power spectrum P[m][k] to the Barker frequency, which simulates the nonlinearity of human ear frequency resolution. A frequency weighting function that simulates the human ear's hearing threshold and equal loudness curve is applied, and the weighted frequency band energy is integrated and cubed to obtain the perceived loudness.

[0173] Next, based on the power spectrum P[m][k] of each frame m and each frequency k, the sum of squares of the energy differences between the corresponding frequency points of the current frame power spectrum and the previous frame power spectrum is calculated to obtain the spectral flux SF[m][k].

[0174] ;

[0175] Secondly, based on the compressed energy E'[x] of the Mel filter, x={1, 2, ..., X}, the discrete cosine transform is applied, and the first 5 coefficients are taken to obtain the cepstral coefficients M[i] of the Mel frequency.

[0176] ;

[0177] Where i represents the index number of the cepstral coefficients of the Mel frequency, i={2, 3, 4, 5}, and the first coefficient is discarded because it represents the DC component.

[0178] Secondly, based on the compressed energy E'[x] of the Mel filter, x={1, 2, ..., X}, the ratio of the energy in the 0~2kHz frequency band to the energy in the 2~5kHz frequency band is calculated to obtain the Hammarberg exponent H. b .

[0179] ;

[0180] For example, the Hammarberg index is an acoustic indicator used to characterize the difference in energy distribution between low and high frequencies in the speech spectrum.

[0181] Finally, the fundamental frequency F0, fundamental frequency probability p(F0), and perceived loudness L of all frames are calculated. oud Spectral flux SF[m][k], cepstral coefficients M[i], and Hammarberg exponent H b This constitutes a two-dimensional acoustic descriptor sequence V1[M-1][5] = {F0[m], p(F0)[m], L oud [m], SF[m][k], M[i][m], H b [m]}, where m represents the frame index number, m={0, 1, 2,..., M-1}.

[0182] For example, various acoustic parameters can be calculated from the filtered spectrum, just as a doctor measures physiological parameters such as heart rate and blood pressure. Each feature value describes a certain attribute of the sound from a specific perspective.

[0183] Step 408: Calculate the statistics for each acoustic feature value based on the acoustic feature value, and construct an acoustic feature vector by combining the acoustic feature value and the statistics. The statistics include the mean, standard deviation, quantiles and extreme values.

[0184] Specifically, for each descriptor in the acoustic descriptor sequence V1, the mean (V1[m][n]) and standard deviation (V1[m][n]) of the corresponding descriptor are calculated, where n represents the descriptor index, n={0, 1, 2, 3, 4, 5}.

[0185] ;

[0186] ;

[0187] Next, based on the acoustic descriptor sequence V1, the descriptors of all frames are sorted from smallest to largest, and the values ​​at the 20, 50, and 80 percentile positions are found and set as quantiles Q1(V1[n]), Q2(V1[n]), and Q3(V1[n]), respectively. The maximum and minimum values ​​are found and set as extreme values ​​Q. max (V1[n]), Q min (V1[n]).

[0188] For example, the fundamental frequency F0[m] of the acoustic descriptor sequence V1 is sorted by the fundamental frequencies of all frames from smallest to largest, resulting in {80.3Hz, 86.4Hz, 91.01Hz, 98.8Hz, 102.3Hz, 114.23Hz, 119.8Hz, 127.6Hz, 135.5Hz, 147.4Hz}. Then, the quantile at the 20th percentile position is Q1(V1[1]) = 86.4Hz, the quantile at the 50th percentile position is Q2(V1[1]) = 102.3Hz, the quantile at the 80th percentile position is Q3(V1[1]) = 135.5Hz, and the maximum value is Q max (V1[1]) = 147.4Hz, with a minimum value of Q. min (V1[1]) = 80.3Hz.

[0189] Step 410: Generate a text sequence based on the energy sequence, and process the text sequence through a multi-layer encoder to obtain the text feature vector.

[0190] Specifically, based on the acoustic descriptor sequence V1[M-1][5], mean(V1[m][n]), standard deviation std(V1[m][n]), quantiles Q1(V1[n]), Q2(V1[n]), Q3(V1[n]), and maximum value Q max (V1[n]), minimum value Q min (V1[n]) constitutes a two-dimensional acoustic feature vector V1[M][7].

[0191] It should be understood that the Mth row of the acoustic feature vector is used to store statistics such as mean and standard deviation, and the 7th column of the first (M-1) rows is set to 0.

[0192] For example, an acoustic feature vector is a statistical report that describes the acoustic characteristics of an entire speech. It no longer focuses on the specific value of the sound at a certain moment, but rather summarizes the overall pattern and range of variation of the entire speech in terms of pitch, loudness, timbre, etc.

[0193] The voice emotion interaction method provided in this application involves a complete acoustic processing flow on the original speech signal, from filtering, framing, windowing, and fast Fourier transform to calculating the power spectrum, constructing a Mel filter, and extracting the energy sequence. Based on this, it calculates various acoustic descriptors and their statistics, such as fundamental frequency, perceived loudness, and spectral flux, to form an acoustic feature vector. Simultaneously, it generates text from the energy sequence and extracts text feature vectors. This process solves the problem of how to comprehensively and multi-dimensionally extract fusion features from the original speech that reflect both the physical characteristics of sound and the semantic content, providing robustness for subsequent emotion recognition and strategy decision-making, and enriching the feature input of information.

[0194] Figure 5 The diagram shown is a flowchart illustrating a voice emotion interaction method provided in another exemplary embodiment five of this application. In this application... Figure 1 This application extends from the embodiments shown. Figure 5 The illustrated embodiment will be described in detail below. Figure 5 The illustrated embodiments and Figure 1 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0195] like Figure 5 As shown, in the voice emotion interaction method provided in Embodiment 5 of this application, a text sequence is generated based on the energy sequence, and the text sequence is processed by a multi-layer encoder to obtain a text feature vector, including the following steps.

[0196] Step 500: Based on the energy sequence, generate a text sequence using an ASR encoder and an ASR decoder;

[0197] Specifically, according to the two-dimensional energy matrix E t [M][X] generates a text sequence Text_t through an ASR encoder and an ASR decoder.

[0198] For example, the ASR encoder is a deep learning model that uses a self-attention mechanism and a feedforward network to convert a two-dimensional energy matrix into an acoustic feature sequence containing rich phonetic and contextual information; each frame of the acoustic feature sequence corresponds to a segment of the original audio, including phonemes, syllables, and pronunciation habits.

[0199] For example, the ASR decoder is an autoregressive deep learning model that takes the acoustic feature sequence input from the ASR encoder, calculates the probability distribution of all candidate next tokens in the vocabulary through a cross-attention mechanism, selects the token with the highest probability as the next token, and outputs all tokens as a text sequence after all frames have been traversed in a loop.

[0200] Step 502: Segment the text sequence into sub-word tags, and add special tags at the beginning and end of the sub-word tags. The special tags are used to represent the instruction signals for the system to understand the sentence structure and semantics.

[0201] Specifically, based on the text sequence Text_t, the text is segmented into sub-tokens using a tokenizer, and special tags [CLS] and [SEP] are added to the beginning and end of the sub-tokens to obtain the token sequence Text_token.

[0202] For example, the text sequence Text_t = “I feel somewhat anxious” is segmented into [“I”, “feel”, “some”, “anxious”, “worried”], and each word after segmentation corresponds to a sub-word token, which is then marked with a special token as Text_token = [[CLS], “I”, “feel”, “some”, “anxious”, “worried”, [SEP]].

[0203] Step 504: Calculate the input vector for each sub-word tag based on the sub-word tags to form a tag input sequence;

[0204] Specifically, based on the token sequence Text_token, for each sub-word token tokens[j], the vector E corresponding to each sub-word token is retrieved from the embedding table. token [j], assign a fixed vector E to each position in the labeled sequence. position [j], add the two vectors to obtain the input vector V for the j-th sub-word tag. token [j].

[0205] V token [j] = E token [j]+E position [j];

[0206] Where j represents the index of the sub-word tag.

[0207] For example, a word segmenter is an algorithmic module for text preprocessing used to segment an input natural language sentence into a series of continuous, machine-processable sub-word units.

[0208] For example, the special symbols [CLS] and [SEP] are predefined special symbols used to control the model's understanding of sentence structure and semantics. [CLS] represents the classification signal, and [SEP] represents the separation signal.

[0209] For example, an embedding table refers to a query table where each row corresponds to a unique sub-word tag, and each column represents a semantic or syntactic feature dimension. When the word segmenter generates a sub-word tag, the system uses this tag as a key to look up its corresponding value in the embedding table, resulting in a fixed-length dense vector, i.e., vector E. token [j].

[0210] For example, vector E position [j] refers to the injection sequence information, the value of which is generated by a predefined sine function.

[0211] Next, the input vector V, which contains multiple sub-word tags, is... token [j] constitutes a labeled input sequence T token = {V token [0], V token [1], ..., V token [J], V token [J+1]}.

[0212] For example, J represents the number of sub-word tokens into which the text sequence Text_t is segmented, and there are (J+2) index numbers because of the special tokens [CLS] and [SEP].

[0213] Step 506: Based on the labeled input sequence, the multi-head self-attention layer of a multi-layer encoder is used to process the input sequence to obtain the output vector of each attention head;

[0214] Specifically, the input sequence T will be labeled token The input is a multi-layer Transformer encoder. Through the multi-head self-attention layer of the Transformer encoder, the long-distance dependency of the word tags and the correlation between the preceding and following word tags are captured, and the output vector Attention(Q, K, V) of each attention head is obtained.

[0215] ;

[0216] Where Q, K, and V represent the query vector, key vector, and value vector, respectively, and are the labeled input sequence T. token Using three different weight matrices w respectively Q w K w V The projection obtained by multiplication; d K The dimension of the key vector is represented by the softmax function, which represents the attention each sub-word tag receives to all sub-word tags in the sequence. The result of the softmax function is then used as a weight to perform a weighted summation on the value vector V.

[0217] For example, the number of layers of the Transformer encoder needs to be preset. In this embodiment, the number of layers of the Transformer encoder is set to 12. The number of attention heads needs to be preset. In this embodiment, the number of attention heads is set to 5.

[0218] Step 508: Concatenate the output vectors of multiple attention heads and input them again into the multi-head self-attention layer. After processing through different layers, obtain the final context-related representation of each sub-word tag. Use the final layer output vector corresponding to the special tag as the text feature vector.

[0219] Specifically, the outputs of multiple attention heads are concatenated, and the concatenated output vector is then calculated again using the Attention(Q, K, V) formula to obtain the output of the corresponding multi-head self-attention layer.

[0220] Next, after processing through multiple multi-head self-attention layers, the final context-related representation of each sub-word tag is obtained, and the final layer output vector corresponding to the sub-word tag [CLS] is used as the text feature vector V2.

[0221] The speech emotion interaction method provided in this application generates a text sequence from an acoustic energy sequence using an ASR encoder and decoder, then performs word segmentation and adds special tags to the text, and processes it through a multi-head self-attention mechanism of a multi-layer Transformer encoder. Finally, the output corresponding to the special tags is used as the text feature vector, which further limits the text feature vector extraction process. This method solves the problem of how to capture deep semantic and contextual information from speech-converted text, and uses the self-attention mechanism to model long-distance dependencies, thereby achieving the goal of extracting high-quality text semantic feature vectors rich in contextual information.

[0222] Figure 6 The diagram shown is a flowchart illustrating a voice emotion interaction method provided in another exemplary embodiment six of this application. Figure 1 This application extends from the embodiments shown. Figure 6 The illustrated embodiment will be described in detail below. Figure 6 The illustrated embodiments and Figure 1 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0223] like Figure 6 As shown in Embodiment Six of this application, the voice emotion interaction method generates and outputs corresponding follow-up question voice content based on the action of the current round, including the following steps.

[0224] Step 600: Based on the action of the current round, randomly select a text template from the question template library corresponding to the action. The text template includes placeholders.

[0225] Specifically, based on the action probability distribution vector P t [k], select the action with the highest probability to obtain the action A for the current round. t .

[0226] ;

[0227] Next, each action is pre-set with a follow-up question template library, based on the action A of the current round. t A text template is randomly selected from the question template library corresponding to the action. The text template includes placeholders.

[0228] For example, the follow-up question template library refers to commonly used templates of follow-up question texts designed according to different actions. For instance, if the action in the current round is empathy, then the text template in the follow-up question template library would be "You just mentioned feeling [emotional word], that must be very difficult. Can you tell me more about that feeling?"

[0229] For example, a placeholder refers to a marker for an emotion word, and the system will use the dialogue context vector C t Fill in the placeholders in the text template and output the text template.

[0230] Step 602: Fill the placeholders in the text template with the dialogue context vector and output the text template.

[0231] Specifically, based on the dialogue context vector C t The dialogue context vector C t Fill in the placeholders in the template and output the text template.

[0232] The voice emotion interaction method provided in this application solves the problem of how to quickly and flexibly transform abstract questioning strategy decisions into context-appropriate natural language questions by pre-setting a text template library containing placeholders for each action, randomly selecting templates according to the decided action, and then filling the placeholders with the dialogue context vector to generate the final follow-up question content. By combining templateization with context information filling, it can ensure the relevance and consistency of the generated content, and introduce a certain degree of randomness to avoid rigid responses.

[0233] Figure 7 The diagram shown is a flowchart illustrating a voice emotion interaction method provided in another exemplary embodiment seven of this application. In this application... Figure 1 This application extends from the embodiments shown. Figure 7 The illustrated embodiment will be described in detail below. Figure 7 The illustrated embodiments and Figure 1 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0234] like Figure 7 As shown, in the voice emotion interaction method provided in Embodiment 7 of this application, after generating and outputting the corresponding follow-up question voice content based on the action of the current round, it further includes:

[0235] Step 700: Record the user's emotional state vector and actions in the next round of feedback after the execution of the strategy network;

[0236] Specifically, after the dialogue ends, obtain the single-round emotional state vector {V} from the most recent (k+1) rounds. t-k V t-(k+1) ,..., V t-2 V t-1 V t} and the actions taken by the system {A t-k A t-(k+1) , ..., A t-2 A t-1 A t} and the next round of text sequence Text_t+1.

[0237] Step 702: Evaluate the immediate reward of the policy network based on the emotional state vector and action of the next round;

[0238] For each round of action decision, an immediate reward r is calculated based on the preset reward function R. t .

[0239] ;

[0240] Among them, D(V) t V t-k:t-1 The function Len(S) measures the difference between the current user's sentiment and historical sentiment. t Pen(A) represents the length of the text used to measure the next round of user responses. t ) represents the penalty for the user prematurely ending the conversation after the current round of follow-up questioning is issued; α, β, and γ represent the weight coefficients, respectively.

[0241] Step 704: Update the weight matrix and bias vector of the policy network based on the immediate reward.

[0242] Specifically, based on the instant reward r t Update the weight matrix and bias vector of the policy network.

[0243] The voice emotion interaction method provided in this application introduces an online learning mechanism by calculating an immediate reward based on the user's feedback on the next round's emotional state vector and action after each dialogue round, using a preset reward function, and then updating the parameters of the policy network with this reward. This method solves the problem that policy networks may fail to adapt to new users or scenarios after deployment, leading to rigid policy decisions. It enables the system to learn and optimize its follow-up questioning strategies from continuous interactive feedback, achieving continuous improvement in personalization and adaptability.

[0244] Figure 8 The diagram shown is an architectural schematic of a voice-emotion interaction system provided in an exemplary embodiment of this application. Figure 8 As shown, the voice emotion interaction system provided in this application embodiment includes: a signal processing module 800, a feature processing module 802, a context generation module 804, a strategy decision module 806, and a voice interaction module 808.

[0245] The signal processing module 800 acquires the current round of speech signal input by the user and processes it to obtain acoustic feature vectors and text feature vectors. The feature processing module 802 concatenates the acoustic feature vectors and text feature vectors and calculates a single-round emotion state vector through a neural network. The context generation module 804 generates a dialogue context vector based on the emotion state vectors from multiple historical rounds. The dialogue context vector is used to represent the dialogue evolution process. The strategy decision module 806 calculates an action probability distribution vector based on the single-round emotion state vector and the dialogue context vector through a pre-trained strategy network. The voice interaction module 808 selects the action with the highest probability as the action for the current round based on the action probability distribution vector, and generates and outputs the corresponding follow-up question voice content based on the action for the current round. The action for the current round is used to represent the proactive follow-up question strategy to guide the user to further express emotions.

[0246] It should be understood that the operation and functions of the relevant modules mentioned in the voice emotion interaction system can be referenced above. Figures 1 to 7 The provided voice emotion interaction methods will not be elaborated upon here to avoid repetition.

[0247] Figure 9 The diagram shown is a structural schematic of an electronic device provided in an exemplary embodiment of this application. Figure 9 As shown, the electronic device 90 includes one or more processors 901 and memory 902.

[0248] The processor 901 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 90 to perform desired functions.

[0249] The memory 902 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 901 may execute the program instructions to implement the power spectrum, energy sequence, and / or other desired functions of the various embodiments of this application described above. Various contents such as weight matrices and bias vectors may also be stored in the computer-readable storage medium.

[0250] In one example, the electronic device 90 may also include an input device 903 and an output device 904, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0251] The input device 903 may include, for example, a keyboard, a mouse, etc.

[0252] The output device 904 can output various information to the outside, including text templates. The output device 904 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0253] Of course, for the sake of simplicity, Figure 9 Only some of the components of the electronic device 90 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 90 may include any other suitable components depending on the specific application.

[0254] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the voice emotion interaction methods according to the various embodiments of this application described above.

[0255] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0256] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the voice emotion interaction method according to various embodiments of this application described above.

[0257] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0258] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0259] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0260] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0261] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0262] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A voice-based emotional interaction method, characterized in that, include: The system acquires the current round of speech signal input by the user and processes the current round of speech signal to obtain acoustic feature vector and text feature vector; The acoustic feature vector and the text feature vector are concatenated, and a single-round emotional state vector is obtained by calculating through a neural network. Based on the emotional state vectors from multiple historical rounds, a dialogue context vector is generated, which is used to represent the dialogue evolution process. The single-round emotion state vector, the dialogue context vector, and the action from the previous round are concatenated to obtain a state representation vector. Based on this state representation vector, a decision state vector is obtained through mapping using a non-linear activation function. This decision state vector is then input into a pre-trained policy network, where multiple hidden layers perform feature transformations to obtain hidden layer output vectors. The last hidden layer output vector is passed through the linear output layer of the policy network to generate an action score for each action. Based on these action scores, an action probability distribution vector is calculated. Based on the action probability distribution vector, the action with the highest probability is taken as the action of the current round. Based on the action of the current round, the corresponding follow-up question voice content is generated and output. The action of the current round is used to represent the proactive follow-up question strategy to guide the user to further express emotions. The user's feedback on the next round's single-round emotional state vector and action after executing the strategy network is recorded. Based on the next round's single-round emotional state vector and action, the immediate reward of the strategy network is evaluated. Based on the immediate reward, the weight matrix and bias vector of the policy network are updated. The action refers to the type of proactive questioning strategy output by the policy network to guide the user's emotional expression, including empathy, clarification, exploring bodily sensations, and promoting expression.

2. The voice emotion interaction method according to claim 1, characterized in that, The step of generating a dialogue context vector based on historical multi-turn emotion state vectors includes: Based on the emotional state vectors from multiple historical rounds and the actions in the corresponding rounds, the vectors are concatenated by round to form a historical feature sequence. The forgetting gate vector is calculated by passing the historical feature sequence through the forgetting gate of the long short-term memory network; Based on the historical feature sequence, the input gate vector and the candidate cell state vector are calculated through the input gate of the long short-term memory network. Update the cell state based on the forget gate vector, the input gate vector, and the candidate cell state vector; Based on the updated cell state, the output gate vector and hidden state are calculated through the output gate of the Long Short-Term Memory network. The hidden state of the previous layer is used as the historical feature sequence of the next layer. Through multi-layer cyclic calculation of the Long Short-Term Memory network, the last hidden state of the last layer is used as the dialogue context vector.

3. A voice emotion interaction method according to any one of claims 1-2, characterized in that, The process of acquiring the current round of speech signal input by the user and processing the current round of speech signal to obtain acoustic feature vectors and text feature vectors includes: The current round of voice signal input by the user is obtained to obtain a pulse code modulation digital sequence. The pulse code modulation digital sequence is then filtered and framed to obtain a frame sequence. Based on the frame sequence, a fast Fourier transform is applied to the data at each frequency point of each frame to obtain a frequency domain complex sequence. The power spectrum at each frequency point is obtained by squaring the modulus of the complex number sequence in the frequency domain, constructing a Mel filter on the power spectrum, and calculating the energy sequence. Based on the power spectrum and the energy sequence, multiple acoustic characteristic values ​​are calculated, including fundamental frequency, fundamental frequency probability, perceived loudness, and spectral flux. Calculate the statistics for each acoustic feature value based on the acoustic feature value, and construct the acoustic feature vector by combining the acoustic feature value and the statistics. The statistics include the mean, standard deviation, quantiles, and extreme values. Based on the energy sequence, a text sequence is generated, and the text sequence is processed by a multi-layer encoder to obtain the text feature vector.

4. The voice emotion interaction method according to claim 3, characterized in that, The step of generating a text sequence based on the energy sequence, and processing the text sequence through a multi-layer encoder to obtain the text feature vector includes: Based on the energy sequence, the text sequence is generated using an ASR encoder and an ASR decoder; The text sequence is segmented into sub-word tags, and special tags are added to the beginning and end of the sub-word tags. The special tags are used to represent the instruction signals for the system to understand the sentence structure and semantics. Based on the sub-word tags, calculate the input vector for each sub-word tag to form a tag input sequence; Based on the labeled input sequence, the output vector of each attention head is obtained by processing it through a multi-head self-attention layer of a multi-layer encoder; The output vectors of multiple attention heads are concatenated and then input into the multi-head self-attention layer. After processing through different layers, the final context-related representation of each sub-word tag is obtained, and the final layer output vector corresponding to the special tag is used as the text feature vector.

5. A voice emotion interaction method according to any one of claims 1-2, characterized in that, The step of generating and outputting corresponding follow-up questioning voice content based on the action of the current round includes: Based on the action of the current round, a text template is randomly selected from the question template library corresponding to the action, and the text template includes a placeholder; The dialogue context vector is used to fill the placeholders in the text template, and the text template is output.

6. A voice-emotion interaction system, characterized in that, include: The signal processing module acquires the current round of speech signal input by the user and processes the current round of speech signal to obtain acoustic feature vector and text feature vector; The feature processing module concatenates the acoustic feature vector and the text feature vector, and calculates a single-round emotion state vector through a neural network. The context generation module generates a dialogue context vector based on the emotional state vectors from multiple historical rounds. The dialogue context vector is used to represent the dialogue evolution process. The strategy decision-making module concatenates the single-round emotion state vector, the dialogue context vector, and the action from the previous round to obtain a state representation vector; based on the state representation vector, a decision state vector is obtained through mapping using a non-linear activation function; the decision state vector is input into a pre-trained policy network, and feature transformation is performed by multiple hidden layers of the policy network to obtain hidden layer output vectors; the last hidden layer output vector is passed through the linear output layer of the policy network to generate an action score for each action; based on the action scores, an action probability distribution vector is calculated. The voice interaction module, based on the action probability distribution vector, selects the action with the highest probability as the action for the current round, generates and outputs corresponding follow-up question voice content based on the action for the current round, and the action for the current round is used to represent the proactive follow-up question strategy to guide the user to further express emotions; records the user's feedback on the next round's single-round emotional state vector and action after executing the strategy network; and evaluates the immediate reward of the strategy network based on the next round's single-round emotional state vector and action. Based on the immediate reward, the weight matrix and bias vector of the policy network are updated. The action refers to the type of proactive questioning strategy output by the policy network to guide the user's emotional expression, including empathy, clarification, exploring bodily sensations, and promoting expression.

7. An electronic device, characterized in that, include: Processor; and A memory that stores computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores computer program instructions that, when executed by a processor, cause the processor to perform the steps of the method as described in any one of claims 1 to 5.

Images