Emotion dynamic decomposition and response method based on multi-modal model

Abstract

The invention discloses an emotion dynamic decomposition and response method based on a multi-modal model. The method comprises the following steps: acquiring and preprocessing a voice digital signal of a user; inputting the voice digital signals into a voice recognition model, and outputting voice characters; inputting the voice characters into a large language model to generate reply characters; and inputting the response emotion label and the reply character into a voice emotion model to generate reply voice with emotion. According to the method, the problem that in the prior art, a man-machine interaction mode mainly based on text or simple voice instructions is limited in user emotion perception is solved, complex intentions and emotion requirements behind user emotions can be deeply understood, and the limitation of a traditional single-mode emotion recognition technology is effectively solved; by fusing multi-modal information and fully utilizing the advantages of different modals such as voice and images, respective defects are effectively made up, the emotion of the user can be accurately recognized in various complex environments, and the recognition accuracy is greatly improved.

Description

Technical Field

[0001] The present invention belongs to the intersection of artificial intelligence and multimodal information processing, and in particular relates to an emotion dynamic decomposition and response method based on a multimodal model. Background Art

[0002] With the rapid development of artificial intelligence (AI), the intelligent level of human-computer interaction has become a research hotspot. Existing technologies primarily rely on text or simple voice commands for human-computer interaction. This interaction model has extremely limited perception of the user's emotional state and is unable to understand the complex intentions and emotional needs behind their emotions. This results in a lack of emotional resonance and an unnatural and unsmooth experience.

[0003] When it comes to emotion recognition, traditional single-modality emotion recognition technology has numerous limitations. Emotion recognition based solely on speech is susceptible to interference from ambient noise. In noisy environments, the characteristics of the speech signal are obscured by the noise, resulting in a significant drop in recognition accuracy. While image-based emotion recognition can capture facial expressions, it is severely impacted when the user's face is partially obscured or in poor lighting, making it impossible to effectively determine the user's emotions.

[0004] From the perspective of emotional responsiveness, existing systems often employ fixed responses and lack dynamic adaptability to changing user emotions. In intelligent customer service scenarios, when responding to user inquiries and complaints, the system fails to adjust its response strategy based on the intensity and type of the user's emotions, effectively failing to soothe or resolve user issues, making it difficult to improve user satisfaction. In the realm of virtual social interaction, when communicating with users, virtual characters fail to respond appropriately to user emotions, failing to create a realistic and emotionally rich communication atmosphere, severely limiting the immersiveness and enjoyment of the virtual social experience.

[0005] In terms of multimodal fusion technology, most existing multimodal fusion methods simply combine features from different modalities, failing to fully tap into the deep connections and complementary information between them. This makes it difficult to accurately analyze and effectively process complex emotions. This leads to biased understanding of emotions in practical applications, making it impossible to provide users with interactive services that meet their emotional needs. Summary of the Invention

[0006] In order to solve the above technical problems existing in the prior art, the present invention proposes a method for dynamic emotional decomposition and response based on a multimodal model, comprising the following steps:

[0007] Step S110: collecting and preprocessing the user's voice digital signal;

[0008] Step S120: Input the voice digital signal into the voice recognition model and output the voice text;

[0009] Step S230: Input the voice text into the large language model to generate a reply text;

[0010] Step S240: Input the response emotion label and reply text into the speech emotion model to generate a reply speech with emotion.

[0011] Furthermore, the response emotion label is generated in the following manner:

[0012] Step S140: Input the voice digital signal collected and pre-processed in step S110 and the voice text generated in step 120 into a multimodal context feature extraction model, and output a context feature vector sequence;

[0013] Step S150: Input the context feature vector sequence into the graph neural network and output the relationship vector sequence;

[0014] Step S160: normalize the relationship vector sequence and output a standard relationship vector sequence;

[0015] Step S170: input the standard relation vector sequence into the emotion network model and output the speech emotion label;

[0016] Step S190: Perform mixed labeling based on the voice emotion label and the visual emotion label, and output the final emotion label;

[0017] Step S220: Using the large language model, select an appropriate label from the emotion category according to the final emotion label as the response emotion label.

[0018] Furthermore, the step S160 specifically includes:

[0019] Calculate the mean and standard deviation of the relationship vector sequence;

[0020] Subtract the calculated mean from each relationship vector, then divide it by the standard deviation, use the scaling factor and bias to adjust the scale and offset of the relationship vector, and output the standard relationship vector.

[0021] Furthermore, the visual emotion label is generated by the following method:

[0022] Step S130: Obtain the user's expression sequence;

[0023] Step S180: Input the expression sequence into the expression recognition network and output the visual emotion label.

[0024] Furthermore, step S130 includes:

[0025] Step S130.1, collecting a sequence of user images;

[0026] Step S130.2: Process the collected user image sequence based on the YOLO algorithm to detect the position and key points of the face, determine the face region based on the position and key points, crop the face region from the user image sequence, and output a face sequence;

[0027] Step S130.3: Use the pre-trained VGG-Face model to extract features from the face sequence, use the extracted feature vectors as expression features, arrange the expression features in chronological order, and use the moving average filtering method to average the expression features of several adjacent frames to obtain a smoothed expression sequence.

[0028] Furthermore, the expression recognition network includes an expression feature extraction module and an expression feature classification module, and step S180 includes:

[0029] The expression feature extraction module performs cube embedding processing on the expression sequence, and the expression sequence is converted into a series of tokens. These tokens are converted into 3D tokens through transposition and dimension expansion. These 3D tokens are then divided into multiple non-overlapping local spatiotemporal tokens to form a local spatiotemporal token sequence, where each token size is 2×5×10;

[0030] A learnable representative token is added to the end of the local spatiotemporal token sequence, and the added sequence is used as the spatiotemporal token sequence. The spatiotemporal token sequence is used to perform local area self-attention processing, global inter-area self-attention processing, and interactive self-attention processing respectively. The local area self-attention processing is to perform multi-head self-attention calculation on the representative token and the remaining tokens of the spatiotemporal token sequence, and combine with the layer normalization operation to output the local area sequence; the global inter-area self-attention processing is to aggregate all tokens of the sequence in the local area and output the global area sequence; the interactive self-attention processing is to interact between the local area sequence and the global area sequence through a multi-head cross-attention mechanism, and combine with the feedforward network for feature mapping to output the expression feature sequence;

[0031] The expression feature classification module inputs the expression feature sequence into multiple deconvolution layers, wherein the first deconvolution layer uses a convolution kernel of size 4×4, the step size is set to 2, and the padding is set to 1; each deconvolution layer generates feature maps of different scales, and then these feature maps of different scales are connected in the channel dimension by splicing to form a feature representation; the feature representation is input into a fully connected layer, the number of neurons in the fully connected layer is set to 1 / 16 of the number of channels of the feature map, and the Sigmoid activation function is used to map each component of the feature vector output by the fully connected layer to the range of 0 to 1 to obtain the attention weight of each component, and the obtained attention weight is multiplied element by element with the original component to output the channel feature sequence; the channel feature sequence is input into the feature classifier, which is stacked by multiple fully connected layers, wherein the output dimension of the first fully connected layer is set to 512, and the output dimension of the last fully connected layer is set to 5, and then after Softmax activation, the predicted visual emotion label is output.

[0032] Furthermore, step S110 includes:

[0033] Step S110.1: Record the user's voice with a microphone, use a microphone array to process the signals received by different microphones, and output a simulated voice signal after locating the sound source;

[0034] Step S110.2: convert the analog voice signal into a digital signal through the device driver of the Linux system;

[0035] Step S110.3: Use a low-pass filter, a high-pass filter, and a band-pass filter to remove high-frequency and low-frequency noise in the digital signal, then adjust the amplitude to a fixed range to achieve signal amplitude normalization, and output the normalized digital signal.

[0036] Furthermore, step S120 includes:

[0037] Step S120.1: Load the speech recognition model into the Linux system and initialize the speech recognition model;

[0038] Step S120.2: resample the digital signal so that the sampling rate is adjusted to the sampling rate required by the speech recognition model, then perform frame segmentation and Hamming windowing on the resampled digital signal to output speech features, and extract Mel-spectrogram features from the speech features;

[0039] Step S120.3: Input the Mel-spectrogram features into a speech recognition model for inference, output multiple text sequences, and use a beam search algorithm to select the optimal text sequence from the multiple text sequences;

[0040] Step S120.4: Correct the text sequence separately, including removing redundant spaces, correcting punctuation marks, and converting uppercase and lowercase letters, and outputting the corrected text sequence.

[0041] Furthermore, the multimodal context feature extraction model in step S140 includes a speech digital signal processing module, a speech text processing module, a feature fusion module and a feature processing module, wherein:

[0042] The input of the speech digital signal processing module is a speech digital signal. The input speech digital signal is framed, and then each frame is windowed and Fourier transformed to obtain spectrum information. The spectrum information is converted to the Mel frequency domain using a Mel filter bank to obtain a Mel spectrum. The Mel spectrum is subjected to discrete cosine transform to output Mel frequency cepstral coefficients.

[0043] The input of the speech and text processing module is speech and text, which is segmented and then converted into corresponding word vectors using a pre-trained word vector model. Multiple word vectors are concatenated into a word vector sequence, which is then processed based on a bidirectional long short-term memory network and an attention mechanism to output a word vector sequence.

[0044] The input of the feature fusion module is the Mel-frequency cepstral coefficient and the word vector sequence. The Mel-frequency cepstral coefficient and the word vector sequence are spliced in the feature dimension to output a fused feature vector. The spliced fused feature vector is then nonlinearly transformed through a fully connected layer and a ReLU activation function to output a context fused vector sequence.

[0045] The input of the feature processing module is a sequence of context fusion vectors. Each vector is projected into a high-dimensional space through a linear projection layer to form a fused embedding vector.

[0046] Add position encoding to each fused embedding vector and output a sequence of position-encoded context feature vectors.

[0047] Furthermore, the graph neural network in step S150 includes a multimodal relationship graph construction module and a network operation module, wherein:

[0048] The multimodal relationship graph construction module determines each vector in the context feature vector sequence as a node vector in the graph, all node vectors form a node set, sets an edge threshold, determines the edge between nodes by cosine distance, and adds an edge between corresponding nodes if the distance between two vectors is less than the set threshold. All edges form an edge set, and the node set and edge set determine the multimodal relationship graph;

[0049] The network operation module uses the attention mechanism to perform vector operations on the multimodal relationship graph. First, each node vector and the adjacent node vector are transformed through a learnable weight matrix. Then, the transformed features of the two are concatenated and input into a linear layer with a LeakyReLU activation function. The result is subjected to exponential operation to obtain the attention coefficient of each node.

[0050] Based on the calculated attention coefficient, the features of adjacent nodes after weight matrix transformation are weighted and aggregated, and then processed by the LeakyReLU activation function to output the fused node vector. All fused node vectors are spliced from large to small according to the attention coefficient to obtain a relationship vector sequence.

[0051] Furthermore, the emotion network model in step S170 concatenates the standard relationship vector sequence to obtain an emotion semantic vector. The emotion semantic vector is transmitted to the processing unit. The emotion semantic vector passes through a series of linear transformation layers in sequence. Each linear transformation layer transforms the emotion semantic vector according to a specific weight matrix. The emotion semantic vector after the final linear transformation is sent to the ReLU activation function layer for nonlinear transformation. The emotion semantic vector after the ReLU activation function is then mapped to a continuous 5-dimensional vector space.

[0052] The 5-dimensional vector is mapped through the softmax function so that each element is converted into a probability value between 0 and 1, the probability value is converted into a 0 / 1 binary vector according to the threshold, and the speech emotion label is output according to the mapping relationship table between the binary vector and the emotion label.

[0053] Furthermore, step S190 includes:

[0054] A large language model is used to make consistency judgments. If the speech emotion label and the visual emotion label are consistent, one of the two consistent labels is randomly selected as the final emotion label.

[0055] If there is a discrepancy between the speech emotion label and the visual emotion label, the large language model maps the pre-defined emotion labels into a semantic space, where each emotion label is converted into a specific vector. The distance between vectors reflects the similarity of the emotional semantics represented by the labels. The model then traverses each label vector in the semantic space.

[0056] For each label vector, the large language model calculates its cosine similarity with the speech emotion label vector and the visual emotion label vector, and then takes the average of these two similarity values. This average is used as an indicator to measure the comprehensive distance between the pre-defined emotion label and the speech emotion label and the visual emotion label.

[0057] All emotion labels in the semantic space are calculated, and finally the emotion label with the smallest distance is selected and determined as the final emotion label.

[0058] Furthermore, step S240 includes:

[0059] After calling the speech emotion model and inputting the response emotion label and reply text, the speech emotion model determines the emotion feature parameters based on the response emotion label, and then adjusts the basic parameters of the speech based on the semantic information of the reply text and outputs the speech intonation parameters;

[0060] The WaveNet model is used to convert the reply text into an acoustic feature sequence, which is then combined with the voice intonation parameters through a vocoder to convert it into an actual speech waveform, outputting an emotional reply voice.

[0061] The present invention solves the problem of limited perception of user emotions in the existing human-computer interaction mode based on text or simple voice commands. It can deeply understand the complex intentions and emotional needs behind user emotions, and effectively solve the limitations of traditional single-modality emotion recognition technology; by fusing multimodal information, fully utilizing the advantages of different modalities such as voice and images, and effectively making up for their respective shortcomings, it can accurately identify user emotions in various complex environments and greatly improve the recognition accuracy; different from the fixed-pattern replies of existing systems, the present invention can also adjust the reply strategy in real time according to the intensity and type of the user's emotions, creating a real and emotional communication atmosphere, greatly enhancing the immersion and fun of the social experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is a flow chart of the present invention;

[0063] Figure 2 This is a schematic diagram of the emotion panel;

[0064] Figure 3 This is a diagram of the mood fluctuation curve. DETAILED DESCRIPTION

[0065] To better understand the objectives, technical solutions, and functions of the present invention, the present invention is described in further detail below with reference to the accompanying drawings. However, the present invention may be implemented in a variety of different ways as defined and covered by the claims. The accompanying drawings, which constitute a part of this invention, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute undue limitations of the present invention.

[0066] like Figure 1 As shown, in a specific embodiment of the present invention, the process includes: step S110, determining the user's voice digital signal, which includes the following steps in detail:

[0067] Step S110.1: Select an appropriate microphone for recording based on the user's environment. Generally, a condenser microphone is selected, but if the surrounding environment is noisy, a dynamic microphone is selected. A microphone array is used to process the signals received by different microphones, and after locating the sound source, an analog voice signal is output.

[0068] Step S110.2: convert the analog voice signal into a digital signal through the device driver of the Linux system;

[0069] Step S110.3: Use a low-pass filter, a high-pass filter, and a band-pass filter to remove high-frequency and low-frequency noise in the digital signal, then adjust the amplitude to a fixed range to achieve signal amplitude normalization, and output the normalized digital signal.

[0070] Step S120: The voice digital signal is output as voice text through the voice recognition model;

[0071] This step specifically includes the following steps:

[0072] Step S120.1: Load the speech recognition model into the system, initialize the model, check whether the dependent libraries and environment required for the model to run are configured correctly, and perform exception handling if a runtime error occurs.

[0073] Step S120.2: resample the digital signal so that the sampling rate is adjusted to the sampling rate required by the model, then frame the signal and perform Hamming windowing to output speech features, and extract Mel spectrum features from the speech features;

[0074] Step S120.3: Input the Mel-spectrogram features into the speech recognition model for inference, output multiple text sequences, and use the beam search algorithm to select the optimal text sequence from the multiple text sequences.

[0075] Step S120.4: Correct the text sequence separately, including removing redundant spaces, correcting punctuation marks, and converting uppercase and lowercase characters, and output the corrected text sequence.

[0076] Step S130: Obtain the user's expression sequence;

[0077] This step specifically includes the following steps:

[0078] Step S130.1: Point a high-definition hyperspectral camera directly at the user's face, approximately at eye level. Set the camera resolution to 1920 × 1080 and the frame rate to 30 fps. Adjust the camera's focal length and aperture to ensure the facial image is clearly focused and has moderate brightness. Finally, capture the user's image sequence.

[0079] Step S130.2: Process the user's image sequence based on the YOLO algorithm to detect the position and key points of the face, determine the face region based on the position and key points, crop the face region from the user's image sequence, and output a face sequence;

[0080] Step S130.3: Use the pre-trained VGG-Face model to extract features from the face sequence, use the extracted feature vectors as expression features, arrange the expression features in chronological order, and use the moving average filtering method to average the expression features of several adjacent frames (such as 3 frames) to obtain a smoothed expression sequence.

[0081] Step S140: The speech digital signal and the speech text are extracted by a multimodal context feature extraction model to output a context feature vector sequence.

[0082] Among them, the multimodal context feature extraction model includes a speech digital signal processing module, a speech text processing module, a feature fusion module and a feature processing module.

[0083] The input of the speech digital signal processing module is the speech digital signal. The input speech digital signal is framed, with each frame usually being 20 to 30 ms. Each frame is then windowed and Fourier transformed to obtain spectral information. The spectral information is converted to the Mel frequency domain using a Mel filter bank to obtain a Mel spectrum. The Mel spectrum is then subjected to discrete cosine transform to output Mel-frequency cepstral coefficients.

[0084] The input of the speech and text processing module is speech and text. The speech and text are segmented, and then each word is converted into a corresponding word vector using a pre-trained word vector model. Multiple word vectors are spliced into a word vector sequence. The word vector sequence is processed based on a bidirectional long short-term memory network and an attention mechanism, and the word vector sequence is output.

[0085] The input of the feature fusion module is the Mel-frequency cepstral coefficients and the word vector sequence. The Mel-frequency cepstral coefficients and the word vector sequence are concatenated in the feature dimension to output the fused feature vector. The concatenated fused feature vector is then further nonlinearly transformed through a fully connected layer and a ReLU activation function to output the context fusion vector sequence.

[0086] The input to the feature processing module is a sequence of contextual fusion vectors. Each vector is projected into the model's high-dimensional space through a linear projection layer to form an initial fused embedding vector. A positional encoding is added to each fused embedding vector, and the output is a sequence of position-encoded contextual feature vectors.

[0087] Step S150: Output the context feature vector sequence into a relationship vector sequence through a graph neural network.

[0088] Among them, the graph neural network includes a multimodal relationship graph construction module and a network operation module.

[0089] The relationship graph construction module determines each vector in the context feature vector sequence as a node vector in the graph. All node vectors form a node set, set the edge threshold, and determine the edges between nodes through cosine distance. If the distance between two vectors is less than the set threshold, an edge is added between the corresponding nodes. All edges constitute an edge set. The node set and the edge set determine the multimodal relationship graph.

[0090] The network operation module uses an attention mechanism to perform vector operations on multimodal relationship graphs. Specifically, each node vector and its adjacent node vectors are first transformed using a learnable weight matrix. The transformed features of the two are then concatenated and input into a linear layer with a LeakyReLU activation function. The result is then subjected to exponential operation to obtain the attention coefficient for each node. Based on the calculated attention coefficient, the features of adjacent nodes, transformed using the weight matrix, are weighted and aggregated. Then, the LeakyReLU activation function is used to process the resulting fused node vector, which is then concatenated from largest to smallest according to the attention coefficient to form a sequence of relationship vectors.

[0091] Step S160: Normalize the relationship vector sequence to output a standard relationship vector sequence.

[0092] First, the mean and standard deviation of the relationship vector sequence are calculated, the calculated mean is subtracted from each relationship vector, and then divided by the standard deviation. The scale and offset of the relationship vector are adjusted using the scaling factor and bias, and the standard relationship vector is output.

[0093] Step S170: The standard relation vector sequence outputs a speech emotion label through the emotion network model.

[0094] The emotion network model concatenates a sequence of standard relational vectors to generate an emotion semantic vector. This vector is then fed into a processing unit comprised of multiple submodules. Within this unit, the vector passes through a series of linear transformation layers, each of which transforms the vector according to a specific weight matrix. The final linearly transformed vector is then fed into a ReLU activation function layer for nonlinear transformation. After the activation function, the vector is then mapped into a continuous 5-dimensional vector space. This 5-dimensional vector is mapped using a softmax function, converting each element into a probability value between 0 and 1. Based on a threshold, the probability values are converted into a 0 / 1 binary vector. The speech emotion label is then output based on a table mapping binary vectors to emotion labels.

[0095] Step S180: The expression sequence outputs a visual emotion label through the expression recognition network.

[0096] Among them, the expression recognition network includes an expression feature extraction module and an expression feature classification module.

[0097] The expression feature extraction module first processes the expression sequence through a cube embedding process, converting it into a series of tokens. These tokens are then converted into 3D tokens through transposition and dimensionality expansion. These 3D tokens are then divided into multiple non-overlapping local spatiotemporal tokens, forming a local spatiotemporal token sequence, where each token is 2×5×10 in size. A learnable representative token is appended to the end of the local spatiotemporal token sequence to facilitate local feature learning and aggregate local information into the representative token. The appended sequence is then used as the spatiotemporal token sequence. The spatiotemporal token sequence undergoes local self-attention, global inter-region self-attention, and interactive self-attention. The local self-attention performs multi-head self-attention on the representative token and the remaining tokens in the spatiotemporal token sequence, combined with layer normalization, to output a local sequence. The global inter-region self-attention aggregates all tokens in the local sequence to output a global sequence. The interactive self-attention uses a multi-head cross-attention mechanism to interact between the local sequence and the global sequence, and is combined with a feedforward network for feature mapping to output an expression feature sequence.

[0098] The expression feature classification module inputs the expression feature sequence into multiple deconvolution layers, each with a specific kernel size, stride, and padding. The first deconvolution layer uniformly uses a 4×4 kernel with a stride of 2 and padding of 1. Each deconvolution layer generates feature maps of different scales, which are then concatenated along the channel dimension to form a more comprehensive feature representation. The feature representation is then input into a fully connected layer with 1 / 16 the number of neurons in the feature map. A sigmoid activation function is used to map each component of the vector to a range of 0 to 1. An attention weight is then assigned to each component and element-wise multiplied with the original component to output a channel feature sequence. This channel feature sequence is then input into a feature classifier, which consists of multiple stacked fully connected layers. The output dimension of the first fully connected layer is 512, and the output dimension of the last fully connected layer is 5. After softmax activation, the predicted visual emotion label is output.

[0099] Step S190: Perform mixed labeling based on the voice emotion label and the visual emotion label, and output the final emotion label.

[0100] Generally speaking, humans can produce 32 kinds of emotions, namely: anxiety, hatred, anger, impatience, excitement, desire, praise, pride, grief, despair, fear, disgust, happiness, surprise, expectation, happiness, sadness, loss, doubt, helplessness, relaxation, gratitude, relief, satisfaction, irritability, disgust, dissatisfaction, resistance, approval, curiosity, gratitude, and trust. Use a panel to display the above 32 emotions, and the panel title is "32 Emotions". Figure 2 shown.

[0101] Emotions are classified and described through the semantics and colors of five dimensions, including the following five dimensions:

[0102] Motivation: This accurately measures the impact of an emotion on individual motivation. Positive motivation emotions, represented by darker colors, strongly drive individual behavior. Negative motivation emotions, which tend to encourage rest and relaxation, are represented by lighter colors, indicating a relatively weaker motivation for behavior.

[0103] Length: This measures the duration of an emotion. Emotions with negative length, or shorter duration, are represented by bright colors, signifying their fleeting nature. Positive length, or longer duration, are represented by dark colors, signifying their enduring nature within an individual's psychological state.

[0104] Unexpectedness: This measures whether an emotion was triggered by an unexpected event. Emotions triggered by unexpected events can be assigned brighter, more eye-catching colors to represent their unexpectedness and strong emotional impact. Emotions triggered by expected events can be assigned softer, more stable colors to represent their calm and stable nature.

[0105] Intensity: This is used to measure the intensity of an emotion. Positive emotions, those with higher intensity, are depicted in cool colors to convey their impact. Negative emotions, those with lower intensity, are depicted in warm colors to convey their relative calmness.

[0106] Positivity: This indicator measures whether an emotion expresses positive or negative sentiment. Positive emotions are represented by center colors, signifying their central and positive role in an individual's psychological state. Negative emotions are represented by edge colors, indicating their marginal and negative impact on an individual's psychological state.

[0107] First, use the large language model to make a preliminary consistency judgment:

[0108] If the speech emotion label and the visual emotion label are consistent, that is, the emotion categories indicated by the two are exactly the same, then one of the two consistent labels is randomly selected as the final emotion label.

[0109] If there is a discrepancy between the spoken and visual emotion labels, the large language model first maps the 32 predefined emotion labels into a semantic space. In this space, each emotion label is converted into a specific vector, and the distance between the vectors reflects the similarity of the emotional semantics represented by the labels. The model then traverses each label vector in the semantic space. For each label vector, the large language model calculates its cosine similarity with the spoken emotion label vector and the visual emotion label vector, and then averages these two similarity values. This average is used as an indicator to measure the combined distance between the spatial label and the spoken and visual emotion labels. By performing this calculation on all 32 emotion labels in the semantic space, the emotion label with the smallest distance is finally selected and determined as the final emotion label.

[0110] In a specific embodiment of the present invention, drawing an emotion tag includes step S200 , drawing an emotion panel.

[0111] First, define the emotion panel. There are 32 emotions on the emotion panel, namely: anxiety, hatred, anger, impatience, excitement, desire, praise, pride, grief, despair, fear, disgust, happiness, surprise, expectation, happiness, sadness, loss, doubt, helplessness, relaxation, gratitude, relief, satisfaction, irritability, disgust, dissatisfaction, resistance, approval, curiosity, gratitude, and belief. The title of the panel is "32 Emotions". Figure 2 shown.

[0112] Next, the semantics and colors of the five dimensions are defined, including the following five dimensions:

[0113] Motivation: This accurately measures the impact of an emotion on individual motivation. Positive motivation emotions, represented by darker colors, strongly drive individual behavior. Negative motivation emotions, which tend to encourage rest and relaxation, are represented by lighter colors, indicating a relatively weaker motivation for behavior.

[0114] Length: This measures the duration of an emotion. Emotions with negative length, or shorter duration, are represented by bright colors, signifying their fleeting nature. Positive length, or longer duration, are represented by dark colors, signifying their enduring nature within an individual's psychological state.

[0115] Unexpectedness: This measures whether an emotion was triggered by an unexpected event. Emotions triggered by unexpected events can be assigned brighter, more eye-catching colors to represent their unexpectedness and strong emotional impact. Emotions triggered by expected events can be assigned softer, more stable colors to represent their calm and stable nature.

[0116] Intensity: This is used to measure the intensity of an emotion. Positive emotions, those with higher intensity, are depicted in cool colors to convey their impact. Negative emotions, those with lower intensity, are depicted in warm colors to convey their relative calmness.

[0117] Positivity: This indicator measures whether an emotion expresses positive or negative sentiment. Positive emotions are represented by center colors, signifying their central and positive role in an individual's psychological state. Negative emotions are represented by edge colors, indicating their marginal and negative impact on an individual's psychological state.

[0118] In a specific embodiment of the present invention, the method further includes step S210 of visualizing the emotion fluctuation on the emotion panel and outputting an emotion fluctuation curve graph.

[0119] First, during the data recording and storage phase, the final emotion labels obtained through the previous steps are accurately recorded each time. These recorded emotion labels are stored in the database in an orderly manner. Then, the corresponding emotion labels are accurately located on the emotion panel. Matplotlib is used to define the coordinate axis, scale, curve style, and color and draw the emotion fluctuation curve. Among them, the previous emotion label will point to the next emotion label. Finally, the curve chart is placed on the Visdom platform to dynamically visualize the emotion fluctuations, such as Figure 3 shown.

[0120] Step S220: Using the large language model, select an appropriate label as the response emotion label from the existing 32 emotion categories according to the final emotion label.

[0121] Step S230: Generate reply text based on the voice text using the large language model.

[0122] Step S240: calling a speech emotion model according to the response emotion tag and the reply text to generate a reply speech with emotion.

[0123] After invoking the speech emotion model and inputting the response emotion label and reply text, the model determines the emotional feature parameters based on the response emotion label. Then, based on the semantic information of the reply text, it adjusts the basic speech parameters and outputs the speech intonation parameters. The WaveNet model converts the reply text into an acoustic feature sequence. The vocoder then combines the acoustic feature sequence with the speech intonation parameters, converting it into an actual speech waveform and outputting the emotionally charged reply.

[0124] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for dynamic emotional decomposition and response based on a multimodal model, characterized in that: The steps include: Step S110: collecting and preprocessing the user's voice digital signal; Step S120: Input the voice digital signal into the voice recognition model and output the voice text; Step S230: Input the voice text into the large language model to generate a reply text; Step S240: Input the response emotion label and reply text into the speech emotion model to generate a reply speech with emotion.

2. The method according to claim 1, characterized in that The response emotion label is generated in the following way: Step S140: Input the voice digital signal collected and pre-processed in step S110 and the voice text generated in step 120 into a multimodal context feature extraction model, and output a context feature vector sequence; Step S150: Input the context feature vector sequence into the graph neural network and output the relationship vector sequence; Step S160: normalize the relationship vector sequence and output a standard relationship vector sequence; Step S170: input the standard relation vector sequence into the emotion network model and output the speech emotion label; Step S190: Perform mixed labeling based on the voice emotion label and the visual emotion label, and output the final emotion label; Step S220: Using the large language model, select an appropriate label from the emotion category according to the final emotion label as the response emotion label.

3. The method according to claim 2, characterized in that The step S160 specifically includes: Calculate the mean and standard deviation of the relationship vector sequence; Subtract the calculated mean from each relationship vector, then divide it by the standard deviation, use the scaling factor and bias to adjust the scale and offset of the relationship vector, and output the standard relationship vector.

4. The method according to claim 2, characterized in that The visual emotion label is generated by the following method: Step S130: Obtain the user's expression sequence; Step S180: Input the expression sequence into the expression recognition network and output the visual emotion label.

5. The method according to claim 4, characterized in that Step S130 includes: Step S130.1, collecting a sequence of user images; Step S130.2: Process the collected user image sequence based on the YOLO algorithm to detect the position and key points of the face, determine the face region based on the position and key points, crop the face region from the user image sequence, and output a face sequence; Step S130.3: Use the pre-trained VGG-Face model to extract features from the face sequence, use the extracted feature vectors as expression features, arrange the expression features in chronological order, and use the moving average filtering method to average the expression features of several adjacent frames to obtain a smoothed expression sequence.

6. The method according to claim 5, characterized in that The expression recognition network includes an expression feature extraction module and an expression feature classification module. Step S180 includes: The expression feature extraction module performs cube embedding processing on the expression sequence, and the expression sequence is converted into a series of tokens. These tokens are converted into 3D tokens through transposition and dimension expansion. These 3D tokens are then divided into multiple non-overlapping local spatiotemporal tokens to form a local spatiotemporal token sequence, where each token size is 2×5×10; A learnable representative token is added to the end of the local spatiotemporal token sequence, and the added sequence is used as the spatiotemporal token sequence. The spatiotemporal token sequence is used to perform local area self-attention processing, global inter-area self-attention processing, and interactive self-attention processing respectively. The local area self-attention processing is to perform multi-head self-attention calculation on the representative token and the remaining tokens of the spatiotemporal token sequence, and combine with the layer normalization operation to output the local area sequence; the global inter-area self-attention processing is to aggregate all tokens of the sequence in the local area and output the global area sequence; the interactive self-attention processing is to interact between the local area sequence and the global area sequence through a multi-head cross-attention mechanism, and combine with the feedforward network for feature mapping to output the expression feature sequence; The expression feature classification module inputs the expression feature sequence into multiple deconvolution layers, wherein the first deconvolution layer uses a convolution kernel of size 4×4, the step size is set to 2, and the padding is set to 1; each deconvolution layer generates feature maps of different scales, and then these feature maps of different scales are connected in the channel dimension by splicing to form a feature representation; the feature representation is input into a fully connected layer, the number of neurons in the fully connected layer is set to 1 / 16 of the number of channels of the feature map, and the Sigmoid activation function is used to map each component of the feature vector output by the fully connected layer to the range of 0 to 1 to obtain the attention weight of each component, and the obtained attention weight is multiplied element by element with the original component to output the channel feature sequence; the channel feature sequence is input into the feature classifier, which is stacked by multiple fully connected layers, wherein the output dimension of the first fully connected layer is set to 512, and the output dimension of the last fully connected layer is set to 5, and then after Softmax activation, the predicted visual emotion label is output.

7. The method according to claim 1, characterized in that Step S110 includes: Step S110.1: Record the user's voice with a microphone, use a microphone array to process the signals received by different microphones, and output a simulated voice signal after locating the sound source; Step S110.2: convert the analog voice signal into a digital signal through the device driver of the Linux system; Step S110.3: Use a low-pass filter, a high-pass filter, and a band-pass filter to remove high-frequency and low-frequency noise in the digital signal, then adjust the amplitude to a fixed range to achieve signal amplitude normalization, and output the normalized digital signal.

8. The method according to claim 7, characterized in that Step S120 includes: Step S120.1: Load the speech recognition model into the Linux system and initialize the speech recognition model; Step S120.2: resample the digital signal so that the sampling rate is adjusted to the sampling rate required by the speech recognition model, then perform frame segmentation and Hamming windowing on the resampled digital signal to output speech features, and extract Mel-spectrogram features from the speech features; Step S120.3: Input the Mel-spectrogram features into a speech recognition model for inference, output multiple text sequences, and use a beam search algorithm to select the optimal text sequence from the multiple text sequences; Step S120.4: Correct the text sequence separately, including removing redundant spaces, correcting punctuation marks, and converting uppercase and lowercase letters, and outputting the corrected text sequence.

9. The method according to claim 2, characterized in that The multimodal context feature extraction model in step S140 includes a speech digital signal processing module, a speech text processing module, a feature fusion module and a feature processing module, wherein: The input of the speech digital signal processing module is a speech digital signal. The input speech digital signal is framed, and then each frame is windowed and Fourier transformed to obtain spectrum information. The spectrum information is converted to the Mel frequency domain using a Mel filter bank to obtain a Mel spectrum. The Mel spectrum is subjected to discrete cosine transform to output Mel frequency cepstral coefficients. The input of the speech and text processing module is speech and text, which is segmented and then converted into corresponding word vectors using a pre-trained word vector model. Multiple word vectors are concatenated into a word vector sequence, which is then processed based on a bidirectional long short-term memory network and an attention mechanism to output a word vector sequence. The input of the feature fusion module is the Mel-frequency cepstral coefficient and the word vector sequence. The Mel-frequency cepstral coefficient and the word vector sequence are spliced in the feature dimension to output a fused feature vector. The spliced fused feature vector is then nonlinearly transformed through a fully connected layer and a ReLU activation function to output a context fused vector sequence. The input of the feature processing module is a sequence of context fusion vectors. Each vector is projected into a high-dimensional space through a linear projection layer to form a fused embedding vector. Add position encoding to each fused embedding vector and output a sequence of position-encoded context feature vectors.

10. The method according to claim 9, characterized in that The graph neural network in step S150 includes a multimodal relationship graph construction module and a network operation module, wherein: The multimodal relationship graph construction module determines each vector in the context feature vector sequence as a node vector in the graph, all node vectors form a node set, sets an edge threshold, determines the edge between nodes by cosine distance, and adds an edge between corresponding nodes if the distance between two vectors is less than the set threshold. All edges form an edge set, and the node set and edge set determine the multimodal relationship graph; The network operation module uses the attention mechanism to perform vector operations on the multimodal relationship graph. First, each node vector and the adjacent node vector are transformed through a learnable weight matrix. Then, the transformed features of the two are concatenated and input into a linear layer with a LeakyReLU activation function. The result is subjected to exponential operation to obtain the attention coefficient of each node. Based on the calculated attention coefficient, the features of adjacent nodes after weight matrix transformation are weighted and aggregated, and then processed by the LeakyReLU activation function to output the fused node vector. All fused node vectors are spliced from large to small according to the attention coefficient to obtain a relationship vector sequence.

11. The method according to claim 10, characterized in that In step S170, the emotion network model concatenates the standard relationship vector sequence to obtain an emotion semantic vector. The emotion semantic vector is transmitted to the processing unit. The emotion semantic vector passes through a series of linear transformation layers in sequence. Each linear transformation layer transforms the emotion semantic vector according to a specific weight matrix. The emotion semantic vector after the final linear transformation is sent to the ReLU activation function layer for nonlinear transformation. After the ReLU activation function is processed, the emotion semantic vector is mapped to a continuous 5-dimensional vector space. The 5-dimensional vector is mapped through the softmax function so that each element is converted into a probability value between 0 and 1, the probability value is converted into a 0 / 1 binary vector according to the threshold, and the speech emotion label is output according to the mapping relationship table between the binary vector and the emotion label.

12. The method according to claim 11, characterized in that Step S190 includes: A large language model is used to make consistency judgments. If the speech emotion label and the visual emotion label are consistent, one of the two consistent labels is randomly selected as the final emotion label. If there is a discrepancy between the speech emotion label and the visual emotion label, the large language model maps the pre-defined emotion labels into a semantic space, where each emotion label is converted into a specific vector. The distance between vectors reflects the similarity of the emotional semantics represented by the labels. The model then traverses each label vector in the semantic space. For each label vector, the large language model calculates its cosine similarity with the speech emotion label vector and the visual emotion label vector, and then takes the average of these two similarity values. This average is used as an indicator to measure the comprehensive distance between the pre-defined emotion label and the speech emotion label and the visual emotion label. All emotion labels in the semantic space are calculated, and finally the emotion label with the smallest distance is selected and determined as the final emotion label.

13. The method according to claim 1, wherein Step S240 includes: After calling the speech emotion model and inputting the response emotion label and reply text, the speech emotion model determines the emotion feature parameters based on the response emotion label, and then adjusts the basic parameters of the speech based on the semantic information of the reply text and outputs the speech intonation parameters; The WaveNet model is used to convert the reply text into an acoustic feature sequence, which is then combined with the voice intonation parameters through a vocoder to convert it into an actual speech waveform, outputting an emotional reply voice.

Images