A multi-modal emotion large model fine-tuning method based on human feedback reinforcement learning

Abstract

The application belongs to the field of artificial intelligence, and specifically discloses a multi-modal emotion large model fine-tuning method based on human feedback reinforcement learning, which comprises the following steps: obtaining a face image, voice data and input text; dividing the face image into image blocks and inputting the image blocks into a multi-scale Transformer to extract multi-scale face features, and converting the voice data into a spectrogram and using a ResNet50 to extract voice features; calculating the correlation between the two features after splicing, obtaining updated features through dimension reduction and feedback, splicing the updated features into multi-modal emotion fusion features, and obtaining an emotion state through classification; forming a composite input by combining the emotion state and the text, making the model learn the corresponding relationship between the emotion state and the language expression through supervised fine-tuning, constructing a reward model to simulate human preferences, and using a PPO algorithm to reinforce fine-tuning. The application can make the reply generated by the large model maintain content fluency and have emotion expression adaptability in line with human preferences.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence, and more specifically, relates to a method for fine-tuning a multimodal emotion model based on reinforcement learning with human feedback. Background Technology

[0002] With the rapid development of technologies such as artificial intelligence, achieving natural and harmonious human-computer interaction has attracted much interest and attention from researchers. Emotional expression is an indispensable part of communication, conveying a wealth of information, such as the other party's current emotional state. Therefore, endowing machines with the ability to recognize, identify, and express emotions is particularly important. Among the many modes of emotional communication, facial expressions account for 55% of human emotional expression, voice information accounts for 20%, and the remainder consists of text information, physiological signals, and body posture. Therefore, by analyzing facial expressions, voice information, and text information, we can obtain a more comprehensive and richer understanding of inner emotional states and more accurately judge the emotions of the interacting parties.

[0003] The emergence of large-scale language models, such as ChatGPT and Claude, has ushered in a new era in the field of emotion recognition. By pre-training large generative language models on unsupervised corpora and then fine-tuning them for supervised NLP tasks, the models can achieve good performance for specific tasks. However, this fine-tuning approach has several shortcomings: the labeled samples cover a limited range of instruction types and scenarios, making it difficult to encompass the diverse contexts of real-world interactions; the supervisory signals struggle to accurately measure subtle quality differences in the model's output regarding emotional adaptability and soothingness, leading to the model easily generating responses in real-world open environments that do not conform to human preferences or emotional expectations, thus reducing the naturalness and emotional experience of human-computer interaction.

[0004] Therefore, how to ensure that the responses generated by the large model maintain both content fluency and emotional expression that aligns with human preferences during the fine-tuning process is a problem that urgently needs to be solved. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this application is to provide a method for fine-tuning a multimodal emotion model based on reinforcement learning of human feedback, which enables the responses generated by the large model to maintain both content fluency and emotional expression adaptability that conforms to human preferences.

[0006] To achieve the above objectives, in a first aspect, this application provides a method for fine-tuning a multimodal large-scale emotion model based on human feedback reinforcement learning, comprising the following steps: S10: Acquire face image, speech data, and input text; divide the face image into image patches of different scales and input them into the multi-scale Transformer module to extract multi-scale face features; convert the speech data into a spectrogram and use the ResNet50 network to extract speech spectrogram features; S20: Concatenate the multi-scale facial features and the speech spectrogram features to obtain initial fusion features; calculate intermodal correlations based on the initial fusion features, the multi-scale facial features, and the speech spectrogram features; map the correlation matrix and the feature matrix to a unified low-dimensional space, obtain features in the low-dimensional space through dimensionality reduction, and then feed the features in the low-dimensional space back to the original feature space to obtain updated facial features and speech features; concatenate the updated facial features and speech features to obtain multimodal emotion fusion features; and feed the multimodal emotion fusion features into a fully connected layer and a softmax classifier to obtain the emotional state. S30, the emotional state is used as an explicit condition to form a composite input with the text. The pre-trained language model is supervised and fine-tuned using supervised fine-tuning data, so that the model learns the correspondence between emotional state and language expression. A reward model is constructed to simulate human preferences for the emotional adaptability and soothingness of responses. Based on the reward value output by the reward model, the PPO algorithm is used to strengthen and fine-tune the supervised and fine-tuned language model so that the responses generated by the model conform to human preferences.

[0007] As a further preferred embodiment, in step S10, dividing the face image into image blocks of different scales includes: Face detection is performed using the MediaPipe algorithm. The detected face regions are cropped and uniformly scaled to a size of 224×224. The scaled face image has a height of H and a width of W. Input the scaled face image into the global branch and the local branch respectively; In the global branch, the image patch size is used. The scaled facial image is divided into regular sections, representing it as a two-dimensional grid composed of several non-overlapping blocks to capture the overall structure and coarse-grained facial expression changes. H and W All can be Divisible by integer, this indicates the total number of image patches in the global branch. Represented as: ; Image patch size Each block covers a certain area of ​​the face, the ( ) i , j The global image patches are represented as follows: ,in, , The rule-based partitioning method ensures that all blocks do not overlap and completely cover the entire face image. In the global branch, each image patch is flattened into a one-dimensional vector, and a linear mapping is used to project the flattened vector onto a dimension of... d In the embedded space.

[0008] As a further preferred embodiment, step S10, extracting multi-scale facial features, further includes: Since the CLS token dimension of the global branch is different from the patch token dimension of the local branch, dimension alignment is performed first: ,in For the projection function used for dimension alignment, The global branch CLS token after dimension alignment. The CLS token for the global branch; Concatenate the dimension-aligned global branch CLS token with the local branch patch tokens: , and then with As a query, with Attention is calculated between them using the following formula: ,in For patch tokens of local branches, , and For learnable parameter matrix, C and h These represent the embedding dimension and the number of attention heads, respectively. q, k, v These represent the query matrix, key matrix, and value matrix, respectively. Calculated by matrix multiplication q and k The similarity between them is transformed into attention weights through a softmax operation, and then compared with... v Multiplying these matrices yields the output matrix of the single-head self-attention mechanism: , ,in A Represents the attention weight matrix; After completing the cross-attention calculation, a feedforward neural network is applied. This network, combined with layer normalization and residual connections, yields the desired features. The feature concatenation operation is performed with the patch tokens of the global branch, and the calculation formula is as follows: ,in The output features of the CLS markers representing the global branch after processing by the cross-attention mechanism. This represents the new global branch feature obtained after the CLS marker in the global branch completes feature interaction. Projection and back-projection functions used for size alignment These are the patch tokens for the global branch.

[0009] As a further preferred embodiment, in step S10, converting the speech data into a spectrogram includes: Continuous speech signal is At a sampling frequency of Under these conditions, discrete speech sequences are obtained through analog-to-digital conversion: ,in The sampling interval is... N This represents the total number of sampling points contained in the speech segment; Pre-emphasis processing for discrete speech: ,in α This is the pre-emphasis coefficient. To be compared with the emphasized discrete speech; After obtaining the windowed frame signal, perform a short-time Fourier transform on each frame: ,in, For the first m Frame in k Complex frequencies at a frequency point K For FFT points, L Represents the total number of time frames; The power spectrum is used to characterize the frequency distribution of speech energy: ,in Used to reflect the first m Energy intensity of different frequency components in a frame.

[0010] As a further preferred option, step S20 specifically includes: Obtaining multi-scale facial features Consonant spectral features After that, among them and The dimensions of facial and speech features are represented, and the features from the two modalities are concatenated to obtain preliminary fused features. , ; Utilizing preliminary fusion features With speech spectrogram features To calculate the correlation between modes, a bilinear similarity function is used. A learnable weight matrix is ​​introduced to map the correlation matrix and the feature matrix to a unified low-dimensional space. The scale of the representation is aligned and redundant information is removed through dimensionality reduction. During dimensionality reduction, an L1 regularization constraint is applied to the mapping matrix, and the calculated L1 norm is multiplied by the regularization strength. λ v , λa Add it to the loss function In this process, the final loss function is obtained. Calculation formula: ; Obtaining features in low-dimensional space and Then, it is fed back into the original feature space, and enhanced features are constructed using linear mapping and residual connections: ,in and It is a linear mapping matrix that maps the multimodal update features in the low-dimensional space back to the original dimension, and uses residual connections to retain the original discriminative features of each modality while introducing multimodal information. The face and voice features updated through the cross-modal attention mechanism are concatenated to obtain the final multimodal emotion fusion feature representation: ; The fused features are fed into a fully connected layer and a softmax classifier to classify emotional states.

[0011] As a further preferred embodiment, step S30, which involves supervising the fine-tuning of the pre-trained language model using supervised fine-tuning data, includes: The Llama3 model performs word segmentation on the original text during the input stage, converting continuous natural language text into discrete symbol sequences. It adopts a sub-word-based word segmentation strategy, where the word segmenter divides the characters in the original text into a series of sub-word units according to a predefined vocabulary. Each word unit corresponds to a number in the vocabulary, and is mapped to the corresponding integer index through a lookup table. The mapping process is deterministic. After obtaining the discrete token sequence, each token is mapped to a continuous vector representation using a word embedding matrix; Based on word embedding representation, rotational position encoding is introduced to characterize the relative position information of different tokens in the sequence. For the first... i Each position, and its corresponding query vector and key vector Perform rotation operations on each pair of dimensions, apply rotation position encoding to the query vector and key vector, and then calculate the self-attention weights. During the supervised fine-tuning phase, text instructions and sentiment tags are concatenated as explicit conditions to form the model's input sequence, and the target output sequence is denoted as... Input sequence and target sequence After word segmentation and index mapping, it is converted into a token sequence that the model can process.

[0012] As a further preferred option, in step S30, a reward model is constructed to simulate human preferences for the emotional adaptability and soothing nature of responses, specifically including: By replacing the vocabulary prediction head of the Llama3 7B model with the reward head, the final reward score is obtained: Where [x, y] represents the concatenated input sequence, This indicates a Transformer encoder with a reward head. The Transformer part is responsible for semantic encoding, and the linear layer maps the final hidden state to a scalar reward value. The final output is: ,in h CLS This indicates the last hidden state. w and b Indicates learnable parameters; Modeling human selection Superior The probability of: ,in This indicates the selected response. A reply indicating rejection. It is the sigmoid function; Based on the above probabilistic modeling, optimizing the reward model parameters by maximizing the log-likelihood of human preferences is transformed into minimizing the negative log-likelihood loss. : .

[0013] As a further preferred embodiment, in step S30, the PPO algorithm is used to enhance and fine-tune the supervised fine-tuning language model, specifically including: A value network is introduced to estimate the expected return of a state, and its output is the state value. : ,in As a discount factor, S t Indicates the current state. Accumulate rewards for discounts; Value network outputs current state value and the value of the next state The output of the value network and the output of the reward model are both fed into the advantage function, which is used to calculate the temporal difference residual and advantage. First, calculate the time-series difference residuals: ,in This represents the output of the reward model, and the residual characterizes the degree of deviation between the actual one-step reward and the future value estimate relative to the current value prediction. Based on this, the dominance function is calculated using the GAE method: , where λ is a smoothing parameter used to make a trade-off between long-term and short-term information; In obtaining an advantage estimate Subsequently, PPO employs constrained policy gradient updates, limiting the policy change magnitude in each iteration through a truncation mechanism based on the probability ratio of the new and old policies. The probability ratio of the new and old policies is defined as follows: The objective function to be truncated is: ,in Indicates the state s t The following actions were taken. θ This represents the weight parameters of the policy network (Actor network) that is currently to be updated. θ old This represents the parameters of the old policy network used during sampling. π Represents the policy function. ε This refers to the cropping area.

[0014] Secondly, this application provides a multimodal emotion large model fine-tuning system based on human feedback reinforcement learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method as described in any one of the above.

[0015] Thirdly, this application provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method as described in any one of the above statements.

[0016] This application achieves the following effects: By multimodal fusion of multi-scale facial features and speech spectrogram features, the emotional state of the interactor can be accurately perceived. This emotional state is then used as an explicit condition alongside the input text, enabling the large model to perceive the interactor's emotional state while understanding the semantics of the text. Supervised fine-tuning allows the model to learn the correspondence between emotional state and language expression to maintain content fluency. A reward model is then constructed to simulate human preferences for emotional appropriateness and soothingness in responses. Based on the reward value, the PPO algorithm is used for reinforcement fine-tuning, driving the model's generation strategy towards human preferences. Through the synergistic effect of these methods, the problem of large models focusing only on textual information without considering the interactor's emotional state and responding in ways that do not conform to human preferences can be effectively solved. This approach maintains the fluency of the large model's responses while also ensuring emotional expression appropriateness that aligns with human preferences. Attached Figure Description

[0017] Figure 1 This is a diagram illustrating the overall algorithm framework of the multimodal emotion big data model provided in this application embodiment; Figure 2 This is a flowchart of the multi-scale face feature extraction algorithm provided in the embodiments of this application; Figure 3 This is the spectrogram corresponding to the speech data provided in the embodiments of this application; Figure 4 This is a schematic diagram of the cross-modal attention mechanism provided in an embodiment of this application; Figure 5 This is a post-training flowchart based on human emotional preferences provided in an embodiment of this application; Figure 6 This is a structural diagram of the Llama3 model provided in the embodiments of this application; Figure 7 This is a flowchart of the emotion monitoring fine-tuning provided in the embodiments of this application; Figure 8 This is a diagram illustrating the reward model construction provided in an embodiment of this application; Figure 9 This is a flowchart of the reward model training process provided in the embodiments of this application; Figure 10 This is a diagram of the PPO enhancement and fine-tuning algorithm provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] The purpose of this application is to provide a method for fine-tuning a multimodal emotion model based on reinforcement learning of human feedback. This method enables the emotion model to infer the emotional intent of the interactor based on the chat text content and real-time emotional state, and generate feedback that conforms to human preferences, thereby achieving a more natural and harmonious human-computer interaction.

[0020] The technical solution adopted in this application is a fine-tuning method for a multimodal emotion model based on human feedback reinforcement learning. The specific method includes the following steps: Step 1: Divide the face image into image patches of different scales and input them into a multi-scale Transformer module to model the interaction between local detail features and global features, thereby achieving complementary fusion of features at different scales. Next, convert the speech data into a spectrogram and use a ResNet50 network to further mine high-level emotional semantic features of the speech modality.

[0021] Step 2: An emotion feature fusion strategy based on cross-modal attention mechanism was designed: by mapping visual and speech features to a shared low-dimensional subspace, attention weights within and between modalities were learned, thereby obtaining a more compact and discriminative multimodal emotion representation.

[0022] Step 3: Use supervised fine-tuning data and human feedback fine-tuning datasets to enhance and fine-tune the large model, ensuring that the output content conforms to human preferences.

[0023] The beneficial effects of this application are as follows: This application can effectively solve the problem that large models only focus on textual information during interaction without considering the emotional state of the interactor, and the response content does not conform to human preferences. By perceiving the emotional state of the interactor through a multimodal emotion recognition algorithm and using it as an emotional condition input to the large model, and through human feedback for reinforcement and fine-tuning, the fluency and emotional soothingness of the large model's response content are significantly improved.

[0024] In one embodiment, the technical solution for achieving the above objective can be specifically as follows: Figure 1 As shown in the figure, this embodiment provides a method for fine-tuning a multimodal emotion model based on human feedback reinforcement learning, including the following steps: Step 1: The raw facial images in the facial emotion database, besides containing crucial expression information, also contain irrelevant information such as background and hair, which can easily interfere with subsequent feature extraction. To improve the stability and accuracy of facial emotion feature extraction, the raw facial data is first preprocessed (e.g., ...). Figure 2 (As shown).

[0025] (1) The MediaPipe algorithm is used to detect faces. The detected face areas are cropped and uniformly scaled to a size of 224×224, effectively removing background and other irrelevant noise interference.

[0026] In the global branch, a larger image patch size is used. Preprocessed face images By performing regular partitioning, the image is represented as a two-dimensional grid composed of several non-overlapping large-scale blocks to capture the overall structure and coarse-grained facial expression variations. Assume that H and W can both be... If divisible by an integer, the total number of image patches in the global branch can be expressed as:

[0027] Larger block size This ensures that each block covers a large area of ​​the face, which is beneficial for modeling global structural information. The (i, j)th global image can be represented as:

[0028] in, , This rule-based division method ensures that all blocks do not overlap and completely cover the entire face image.

[0029] In ViT, image patches need to be converted into vector form before being input into the Transformer encoder. To do this, each image patch is flattened into a one-dimensional vector:

[0030] in This indicates that a three-dimensional tensor is flattened into a one-dimensional vector in either channel-first or row-first order.

[0031] To map the original vectors in pixel space to the feature space where the Transformer resides, a linear mapping is used. Projected into an embedding space of dimension d:

[0032] in, and Learnable parameters shared across all branches. Let be the feature representation of the (i, j)th global image patch.

[0033] (2) Introducing a multi-scale feature fusion module based on attention mechanism, by adaptively allocating attention weights between global and local branches, the collaborative modeling and effective fusion of facial emotion features at different scales are achieved, thereby obtaining a multi-scale facial emotion representation that is both globally consistent and locally discriminative.

[0034] Using CLS tokens from each branch as "proxies" for information exchange with patch tokens from another branch helps introduce supplementary information at different scales. The merged information is then fed back to its own branch. Since the dimensions of large and small branches differ, dimensional alignment is the primary step.

[0035] in, This is a projection function used for dimension alignment, which aligns the CLS tokens of the larger branch with the patchtokens of the smaller branch in terms of dimensions, facilitating subsequent calculations.

[0036] To inject multi-scale information without compromising the global semantics already learned by the CLS tokens of the larger branches, they are concatenated with the patch tokens of the smaller branches:

[0037] The attention computation is performed on the sequence using the CLS token as the sole query. The CLS token can assign a high weight to its own representation to retain the original information, and can also selectively aggregate supplementary information from patch tokens of another scale according to task requirements, thereby achieving more robust cross-scale feature interaction.

[0038] by For querying, and Calculate attention between them:

[0039] in, , , Let C be the learnable parameter matrix, and let h be the embedding dimension and the number of attention heads, which are 768 and 12, respectively. After obtaining the q, k, and v matrices, the similarity between q and k is calculated through matrix multiplication to measure the degree of association between different positions. The similarity matrix is ​​then transformed into attention weights using a softmax operation, and multiplied by v to obtain the output matrix of the single-head self-attention mechanism.

[0040]

[0041]

[0042] Where A represents the attention weight matrix. After performing the cross-attention calculation, a feedforward neural network (FFN) is applied, and after combining layer normalization and residual connections: The output features obtained after processing the CLS labels (classification labels) representing the major branches through the cross-attention mechanism are... Perform feature concatenation operations with the patch tokens of the larger branch:

[0043] in, This represents the new large branch feature obtained after the CLS markers in the large branch complete feature interaction. Projection and back-projection functions for size alignment.

[0044] (3) Spectrogram (e.g.) Figure 3 The image shown represents the speech signal mapped onto the time-frequency plane using a short-time Fourier transform, which can simultaneously depict the dynamic changes of speech in the time dimension and the energy distribution in the frequency dimension.

[0045] Discretization and pre-emphasis, continuous semantic signal is At a sampling frequency of Under these conditions, discrete speech sequences are obtained through analog-to-digital conversion:

[0046] in, Where N is the sampling interval, and N is the total number of sampling points contained in the speech segment.

[0047] To enhance high-frequency components and compensate for the effect of vocal tract attenuation on high frequencies, discrete speech is usually pre-emphasized.

[0048] in, As the pre-emphasis coefficient, in this chapter A first-order high-pass filter is used to emphasize rapidly changing parts of the speech, which to some extent facilitates the extraction of subsequent emotional features.

[0049] Short-Time Fourier Transform and Power Spectrum: After obtaining the windowed frame signal, perform a Short-Time Fourier Transform (STFT) on each frame.

[0050] in, Let be the complex frequency at the k-th frequency point of the m-th frame, and K be the number of FFT points. The frequency distribution of speech energy is closely related to emotional state, and therefore is often characterized by power spectrum.

[0051] This reflects the energy intensity of different frequency components in the m-th frame.

[0052] Step 2: Visual information obtained from multi-scale facial images and audio information extracted from spectrograms can provide rich evidence for judging emotional state.

[0053] like Figure 4 As shown, after obtaining multi-scale facial features Consonant spectral features After that, among them, , This represents the dimension of facial and speech features. The features from the two modalities are concatenated to obtain preliminary fused features. .

[0054]

[0055] Utilizing initial fusion features With each modal feature To calculate intermodal correlations and enhance modeling capabilities, a bilinear similarity function is employed, incorporating a learnable weight matrix. get:

[0056] Introducing a learnable weight matrix , , , The correlation matrix and feature matrix are mapped to a unified low-dimensional space. Through dimensionality reduction, their representation scales can be aligned and some redundant information can be removed, resulting in a more compact representation and improving the training stability and computational efficiency of the model.

[0057]

[0058] in, and This means that the original features of each modality are linearly projected into a unified low-dimensional subspace, which serves to compress and align the features.

[0059] In the dimensionality reduction process, to avoid introducing too many redundant parameters into the mapping matrix and to improve the sparsity of the model, during training... and Apply Regular expression constraints

[0060] Then the calculated L1 norm is multiplied by the regularization strength. , Add this to the loss function to obtain the final loss calculation formula:

[0061] Obtaining features in low-dimensional space , Then, it needs to be fed back into the original feature space to obtain the final fused representation. Enhanced features are constructed using linear mapping and residual connections:

[0062] in, and It is a linear mapping matrix that maps the multimodal update features in the low-dimensional space back to the original dimension. It also uses residual connections to ensure that the original discriminative features of each modality are preserved while introducing multimodal information, which helps to stabilize training and avoid over-updating information.

[0063] Finally, the face and voice features updated through the cross-modal attention mechanism are concatenated to obtain the final multimodal emotion fusion feature representation:

[0064] This fusion feature includes both stable representations within a modality and explicitly integrates complementary information between modalities, resulting in a more compact overall representation with more concentrated effective information and stronger discriminative power. It is then fed into a fully connected layer and a softmax classifier to classify sentiment states.

[0065] Step 3: As Figure 5 As shown, text is extracted from speech through the speech recognition module, and then the emotional state obtained from multimodal emotion recognition is used as an explicit condition to form a composite input of "text + emotional state" together with the text. This allows the large model to perceive and utilize the corresponding emotional state while understanding the semantics of the text, and the model is fine-tuned by RLHF to make its generated responses more in line with human preferences.

[0066] (1) Use high-quality instruction-response data constructed by manual annotation or rules to supervise the learning of pre-trained language models and enhance the model’s performance on specific tasks or generative styles.

[0067] like Figure 6 As shown, the Llama3 model performs word segmentation on the original text during the input stage, converting continuous natural language text into discrete symbol sequences. This process employs a sub-word-based segmentation strategy, ensuring that the model can control the vocabulary size while maintaining vocabulary coverage, as shown in the equation:

[0068] in, The tokenizer represents the characters in the original text, and it divides them into a series of sub-word units based on a predefined vocabulary. This allows the text to be transformed from a character-level representation into a discrete sequence by assigning it a unique index number in the vocabulary.

[0069] Each word unit Each element in the vocabulary corresponds to a number, which is then mapped to a corresponding integer index through a lookup table:

[0070] in This indicates the vocabulary size, which is 128255. The mapping process is deterministic, meaning that the same subword will be mapped to the same token number in any input, thus ensuring the consistency and reproducibility of the model input.

[0071] After obtaining the discrete token sequence, each token is mapped to a continuous vector representation through a word embedding layer. Let the word embedding matrix be:

[0072] Where d represents the word vector dimension, which is 4096. For the entire input sequence, the corresponding word vector representation is:

[0073] in, Represents each sub-word unit Word vector representation in vector space. In this way, the original discrete text is transformed into a continuous sequence of vectors.

[0074] Since the Transformer structure itself does not have the ability to explicitly model positional information in a sequence, the model introduces Rotary Position Embedding (RPE) on top of the word embedding representation to characterize the relative positional information of different tokens in the sequence. For the i-th position, its corresponding query vector and key vector Perform the following rotation operation on each pair of dimensions:

[0075] in, This represents the rotation angle relative to the token position i. and Represents the orthogonal components of a vector in a two-dimensional subspace. Apply RoPE rotational position encoding to the query vector and key vector:

[0076] Calculate the self-attention weights using the rotated query and key vectors:

[0077] In this way, positional information is implicitly encoded into the calculation of attention weights, without the need to explicitly concatenate positional vectors at the input layer. While maintaining the semantic expressiveness of word vectors, the model effectively incorporates sequence positional information, providing crucial support for contextual modeling and long-sequence dialogue understanding in subsequent sentiment-supervised fine-tuning.

[0078] like Figure 7 As shown, during the fine-tuning phase, text instructions and sentiment tags are concatenated as explicit conditions to form the model's input sequence:

[0079] The target output sequence is denoted as:

[0080] Input sequence and target sequence After word segmentation and index mapping, it is converted into a token sequence that the model can process.

[0081] Given an input token sequence, Llama3 uses an autoregressive approach to progressively generate output tokens. For the t-th time step, the conditional probability of the model predicting the next token is:

[0082] in, This represents the model parameters. The probability distribution is calculated jointly by the multi-head self-attention mechanism and the feedforward network in the Transformer architecture.

[0083] The goal is to minimize the difference between the model-generated results and the actual target response. The overall loss function is defined as:

[0084] By maximizing the log-likelihood of generating the target response sequence under given input conditions, the model is guided to learn the correspondence between emotional state and language expression.

[0085] (2) By constructing reward signals related to emotional preferences, human evaluations of the emotional adaptability and soothingness of responses are incorporated into the model optimization process, so that the model no longer relies solely on static labels for learning, but continuously adjusts the generation strategy through reinforcement learning mechanisms, thereby further optimizing and aligning the quality of emotional expression.

[0086] like Figure 8 As shown, by replacing the vocabulary prediction head of the Llama3 7B model with the reward head, the final reward score is obtained:

[0087] in, This represents the concatenated input sequence. This represents a Transformer encoder plus a reward head. The Transformer part is responsible for semantic encoding, and the linear layer maps the final hidden state to a scalar reward value. The final output is:

[0088] in, This represents the last hidden state, and w and b represent learnable parameters.

[0089] Modeling human selection Superior The probability of:

[0090] in, Chosen replied. This indicates a rejected response. For the sigmoid function:

[0091] like Figure 9 As shown, based on the above probability modeling, the reward model parameters can be optimized by maximizing the log-likelihood of human preferences. In actual training, this is transformed into minimizing the negative log-likelihood loss.

[0092] When the reward model assigns a significantly higher reward value to the Chosen response than to the Rejected response, the preference loss approaches zero, indicating that the model has captured the human preference relationship well. When the difference between the two rewards is small, the loss value increases accordingly, thereby generating a stronger gradient signal to drive parameter updates and prompting the reward function to gradually approach the true human preference distribution.

[0093] Instant rewards alone Directly updating the strategy can easily lead to large gradient estimation variance and unstable training. Therefore, a value network is introduced to estimate the expected reward of a state, and its output is the state value:

[0094] in, As a discount factor. The value network outputs not only the current state value. It also outputs the value of the next state. In this task This represents the next state after generating a new token. Therefore, the output of the value network, together with the output of the reward model, enters the advantage function to calculate the temporal difference residual and advantage, thereby transforming the reward signal into an improvement direction that can be stably learned.

[0095] To measure the action In state Below the "average level", i.e., the value baseline To determine the relative merits of different factors, an advantage function is introduced. First, calculate the time-series difference residuals:

[0096] in, This represents the output of the reward model. The residual characterizes the deviation between the actual one-step reward and the estimated future value relative to the current value prediction, reflecting the immediate improvement trend brought about by the current action. Based on this, to further balance bias and variance, the GAE method is adopted:

[0097] in, This is a smoothing parameter used to make trade-offs between long-term and short-term information. When When, it indicates the state. Take action below If the return is higher than the baseline level, its probability should be increased in the strategy update; when Conversely, the opposite is true. At this point, the outputs of the reward model and the policy model are unified and integrated into a single advantage signal. This signal will directly determine the direction and magnitude of the policy update, and will be fed back to the policy network through the PPO truncation mechanism.

[0098] like Figure 10 As shown, in obtaining the advantage estimate Subsequently, PPO employs constrained policy gradient updates, limiting the magnitude of policy change in each iteration through a "new-old policy probability ratio + truncation" mechanism. The new-old policy probability ratio is defined as follows:

[0099] The objective function to be truncated is: .

[0100] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for fine-tuning a multimodal emotion model based on human feedback reinforcement learning, characterized in that, Includes the following steps: S10: Acquire face image, speech data, and input text; divide the face image into image patches of different scales and input them into the multi-scale Transformer module to extract multi-scale face features; convert the speech data into a spectrogram and use the ResNet50 network to extract speech spectrogram features; S20, the multi-scale facial features and the speech spectrogram features are concatenated to obtain the initial fusion features; Based on the initial fusion features, the multi-scale face features, and the speech spectrogram features, intermodal correlations are calculated; the correlation matrix and feature matrix are mapped to a unified low-dimensional space, and features in the low-dimensional space are obtained through dimensionality reduction; the features in the low-dimensional space are then fed back to the original feature space to obtain updated face and speech features; the updated face and speech features are concatenated to obtain multimodal emotion fusion features; the multimodal emotion fusion features are then fed into a fully connected layer and a softmax classifier to obtain the emotional state. S30, the emotional state is used as an explicit condition to form a composite input with the text. The pre-trained language model is supervised and fine-tuned using supervised fine-tuning data, so that the model learns the correspondence between emotional state and language expression. A reward model is constructed to simulate human preferences for the emotional adaptability and soothingness of responses. Based on the reward value output by the reward model, the PPO algorithm is used to strengthen and fine-tune the supervised and fine-tuned language model so that the responses generated by the model conform to human preferences.

2. The method for fine-tuning a multimodal emotion model based on human feedback reinforcement learning as described in claim 1, characterized in that, In step S10, dividing the face image into image blocks of different scales includes: Face detection is performed using the MediaPipe algorithm. The detected face regions are cropped and uniformly scaled to a size of 224×224. The scaled face image has a height of H and a width of W. Input the scaled face image into the global branch and the local branch respectively; In the global branch, the image patch size is used. The scaled facial image is divided into regular sections, representing it as a two-dimensional grid composed of several non-overlapping blocks to capture the overall structure and coarse-grained facial expression changes. H and W All can be Divisible by integer, this indicates the total number of image patches in the global branch. Represented as: ; Image patch size Each block covers a certain area of ​​the face, the ( ) i , j The global image patches are represented as follows: ,in, , The rule-based partitioning method ensures that all blocks do not overlap and completely cover the entire face image. In the global branch, each image patch is flattened into a one-dimensional vector, and a linear mapping is used to project the flattened vector onto a dimension of... d In the embedded space.

3. The method for fine-tuning a multimodal emotion model based on human feedback reinforcement learning as described in claim 2, characterized in that, In step S10, extracting multi-scale facial features also includes: Since the CLS token dimension of the global branch is different from the patch token dimension of the local branch, dimension alignment is performed first: ,in For the projection function used for dimension alignment, The global branch CLS token after dimension alignment. The CLS token for the global branch; Concatenate the dimension-aligned global branch CLS token with the local branch patch tokens: , and then with As a query, with Attention is calculated between them using the following formula: ,in For patch tokens of local branches, , and For learnable parameter matrix, C and h These represent the embedding dimension and the number of attention heads, respectively. q, k, v These represent the query matrix, key matrix, and value matrix, respectively. Calculated by matrix multiplication q and k The similarity between them is transformed into attention weights through a softmax operation, and then compared with... v Multiplying these matrices yields the output matrix of the single-head self-attention mechanism: , ,in A Represents the attention weight matrix; After completing the cross-attention calculation, a feedforward neural network is applied. This network, combined with layer normalization and residual connections, yields the desired features. The feature concatenation operation is performed with the patch tokens of the global branch, and the calculation formula is as follows: ,in The output features of the CLS markers representing the global branch after processing by the cross-attention mechanism. This represents the new global branch feature obtained after the CLS marker in the global branch completes feature interaction. Projection and back-projection functions used for size alignment These are the patch tokens for the global branch.

4. The method for fine-tuning a multimodal emotion model based on human feedback reinforcement learning as described in claim 1, characterized in that, In step S10, converting speech data into a spectrogram includes: Continuous speech signal is At a sampling frequency of Under these conditions, discrete speech sequences are obtained through analog-to-digital conversion: ,in The sampling interval is... N This represents the total number of sampling points contained in the speech segment; Pre-emphasis processing for discrete speech: ,in α This is the pre-emphasis coefficient. To be compared with the emphasized discrete speech; After obtaining the windowed frame signal, perform a short-time Fourier transform on each frame: ,in, For the first m Frame in k Complex frequencies at a frequency point K For FFT points, L Represents the total number of time frames; The power spectrum is used to characterize the frequency distribution of speech energy: ,in Used to reflect the first m Energy intensity of different frequency components in a frame.

5. The method for fine-tuning a multimodal emotion model based on human feedback reinforcement learning as described in claim 1, characterized in that, Step S20 specifically includes: Obtaining multi-scale facial features Harmony and speech spectrogram features After that, among them and The dimensions of facial and speech features are represented, and the features from the two modalities are concatenated to obtain preliminary fused features. , ; Utilizing preliminary fusion features With speech spectrogram features To calculate the correlation between modes, a bilinear similarity function is used. A learnable weight matrix is ​​introduced to map the correlation matrix and the feature matrix to a unified low-dimensional space. The scale of the representation is aligned and redundant information is removed through dimensionality reduction. During dimensionality reduction, an L1 regularization constraint is applied to the mapping matrix, and the calculated L1 norm is multiplied by the regularization strength. λ v , λ a Add it to the loss function In this process, the final loss function is obtained. Calculation formula: ; Obtaining features in low-dimensional space and Then, it is fed back into the original feature space, and enhanced features are constructed using linear mapping and residual connections: ,in and It is a linear mapping matrix that maps the multimodal update features in the low-dimensional space back to the original dimension, and uses residual connections to retain the original discriminative features of each modality while introducing multimodal information. The face and voice features updated through the cross-modal attention mechanism are concatenated to obtain the final multimodal emotion fusion feature representation: ; The fused features are fed into a fully connected layer and a softmax classifier to classify emotional states.

6. The method for fine-tuning a multimodal emotion model based on human feedback reinforcement learning as described in claim 1, characterized in that, In step S30, supervising the fine-tuning of the pre-trained language model using supervised fine-tuning data includes: The Llama3 model performs word segmentation on the original text during the input stage, converting continuous natural language text into discrete symbol sequences. It adopts a sub-word-based word segmentation strategy, where the word segmenter divides the characters in the original text into a series of sub-word units according to a predefined vocabulary. Each word unit corresponds to a number in the vocabulary, and is mapped to the corresponding integer index through a lookup table. The mapping process is deterministic. After obtaining the discrete token sequence, each token is mapped to a continuous vector representation using a word embedding matrix; Based on word embedding representation, rotational position encoding is introduced to characterize the relative position information of different tokens in the sequence. For the first... i Each position, and its corresponding query vector and key vector Perform rotation operations on each pair of dimensions, apply rotation position encoding to the query vector and key vector, and then calculate the self-attention weights. During the supervised fine-tuning phase, text instructions and sentiment tags are concatenated as explicit conditions to form the model's input sequence, and the target output sequence is denoted as... Input sequence and target sequence After word segmentation and index mapping, it is converted into a token sequence that the model can process.

7. The method for fine-tuning a multimodal emotion model based on human feedback reinforcement learning as described in claim 1, characterized in that, In step S30, a reward model is constructed to simulate human preferences for the emotional adaptability and soothing nature of responses, specifically including: By replacing the vocabulary prediction head of the Llama3 7B model with the reward head, the final reward score is obtained: Where [x, y] represents the concatenated input sequence, This indicates a Transformer encoder with a reward head. The Transformer part is responsible for semantic encoding, and the linear layer maps the final hidden state to a scalar reward value. The final output is: ,in h CLS This indicates the last hidden state. w and b Indicates learnable parameters; Modeling human selection Superior The probability of: ,in This indicates the selected response. A reply indicating rejection. It is the sigmoid function; Based on the above probabilistic modeling, optimizing the reward model parameters by maximizing the log-likelihood of human preferences is transformed into minimizing the negative log-likelihood loss. : .

8. The method for fine-tuning a multimodal emotion model based on human feedback reinforcement learning as described in claim 1, characterized in that, In step S30, the PPO algorithm is used to enhance and fine-tune the supervised language model, specifically including: A value network is introduced to estimate the expected return of a state, and its output is the state value. : ,in As a discount factor, S t Indicates the current state. Accumulate rewards for discounts; Value network outputs current state value and the value of the next state The output of the value network and the output of the reward model are both fed into the advantage function, which is used to calculate the temporal difference residual and advantage. First, calculate the time-series difference residuals: ,in This represents the output of the reward model, and the residual characterizes the degree of deviation between the actual one-step reward and the future value estimate relative to the current value prediction. Based on this, the dominance function is calculated using the GAE method: , where λ is a smoothing parameter used to make a trade-off between long-term and short-term information; In obtaining an advantage estimate Subsequently, PPO employs constrained policy gradient updates, limiting the policy change magnitude in each iteration through a truncation mechanism based on the probability ratio of the new and old policies. The probability ratio of the new and old policies is defined as follows: The objective function to be truncated is: ,in Indicates the state s t The following actions were taken. θ This represents the weight parameters of the policy network that are currently to be updated. θ old This represents the parameters of the old policy network used during sampling. π Represents the policy function. ε This refers to the cropping area.

9. A multimodal emotion large-scale model fine-tuning system based on human feedback reinforcement learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by a processor, the computer instructions implement the steps of the method as described in any one of claims 1 to 8.

Images