Speech large language model translation enhancement method based on emotion enhancement

Abstract

The invention discloses a speech large language model translation enhancement method based on emotion enhancement, and relates to the field of artificial intelligence. The problem that in the prior art, how to efficiently integrate voice modal information into an existing large language model is lacked, and an end-to-end voice automatic translation task is completed is solved. The method comprises the following steps: collecting a multi-language data set of voice and text pairs containing various emotion expressions, and preprocessing to form multi-language corpus pairs of original text voice and text and translated text voice and text; inputting the voice part of the multi-language corpus pair into the voice big language model, and translating the voice part by the voice big language model; evaluating the translation result in the step 2 by using an emotion integrating degree scoring model, and judging whether the speech emotion in the translation result is consistent with the emotion in the multi-language corpus pair of the original text speech and text; and scoring different results of the same question by using a scoring model, carrying out a DPO algorithm according to the sequence of scores from high to low, and updating parameters of the voice big language model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a speech large language model translation enhancement method based on emotion enhancement. Background Art

[0002] With the rapid development of deep learning and large language models (LLMs), machine translation technology has made significant progress. Although traditional neural machine translation (NMT) systems perform well in processing pure text translation tasks, with the widespread existence of multimodal data (such as images, voice, text, etc.) in practical applications, how to effectively process and fuse multimodal information has become an important topic in current research.

[0003] In the automatic speech translation task, the integration and processing of speech information is key. The existing research focuses on how to improve the model's understanding and generation capabilities by expanding the model architecture and introducing speech data. For example, researchers have significantly improved the performance of large language models in multimodal tasks by designing specialized pre-training tasks and speech expert learning methods. However, how to efficiently integrate speech modality information into the existing decoder-only large language model and activate its potential in the automatic speech translation task so that it can complete the end-to-end automatic speech translation task is the focus of recent research. Summary of the invention

[0004] In order to solve the problem lacking in the prior art of how to efficiently integrate speech modal information into the existing decoder-only large language model and activate its potential in the automatic speech translation task, the present invention is the focus of recent research to enable it to complete the end-to-end automatic speech translation task.

[0005] To solve the above technical problems, the present invention is achieved through the following technical solutions:

[0006] Solution 1: The present invention proposes a speech large language model translation enhancement method based on emotion enhancement, the method comprising the following steps:

[0007] Step 1: collecting a multilingual data set containing at least one speech and text pair of emotional expression, and preprocessing the multilingual data set to form a multilingual corpus pair of original speech and text and translated speech and text;

[0008] Step 2: input the speech part of the multilingual corpus pair in the multilingual data set in step 1 into the speech large language model, and let the speech large language model translate it;

[0009] Step 3: Use the sentiment fit scoring model to evaluate the translation result in step 2. If the speech sentiment in the translation result in step 2 is consistent with the sentiment in the multilingual corpus pair of the original speech and text in step 1, a positive reward is given; if it is inconsistent, a negative reward is given;

[0010] Step 4: Use the scoring model described in step 3 to score the different results of the same question in step 2, and perform the DPO algorithm based on the order of scores from high to low to update the parameters of the speech language model.

[0011] Furthermore, a preferred implementation manner is provided, wherein the method for preprocessing the multilingual data set in step 1 comprises noise reduction, feature extraction, and language screening.

[0012] Furthermore, a preferred implementation is provided, wherein step 2 also includes the step of designing a speech translation template based on emotion.

[0013] Furthermore, a preferred implementation is provided, wherein the emotion-based speech translation template is implemented using an instruction template.

[0014] Furthermore, a preferred implementation is provided, in which the large speech language model described in step 2 is implemented using a Qwen2-audio model.

[0015] Furthermore, a preferred implementation is provided, in which the emotion fit scoring model for the speech in step 3 is implemented using a HuBert-based emotion classification model.

[0016] Furthermore, a preferred embodiment is provided, wherein the HuBert emotion classification model includes scoring of three dimensions of speech: Arousal, Dominance, and Valence.

[0017] Further, a preferred implementation is provided, in which the DPO algorithm in step 4 is:

[0018]

[0019] Among them, x is the input speech, y w is the option with consistent sentiment in the translation result, y l is the option with inconsistent sentiment in the translation result, π θ (y|x) represents the predicted probability of the current model for translation y, π ref represents the probability distribution of the reference model, β is a hyperparameter that controls the degree to which the model deviates from the reference model, and σ is the logistic regression function for binary classification.

[0020] Solution 2: A computer storage medium is used to store a computer program. When the computer program is read by a computer, the computer executes any one of the methods described in Solution 1.

[0021] Solution three: A computer device includes a processor and a storage medium, wherein the memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes any one of the methods described in Solution one.

[0022] The present invention is beneficial in that:

[0023] The emotion-enhanced large language model translation reinforcement method described in the present invention adopts an emotion feedback-based reinforcement learning strategy that includes emotion information quantification technology and emotion-based reward function construction technology, which can improve the emotion transmission ability of the large speech model in the speech-to-text automatic translation task at a low cost.

[0024] The method described in the present invention can deeply mine the emotional information in speech and accurately map it to the translation result, effectively enhancing the effect of the large speech model on emotion processing during speech translation, especially when processing speech content containing rich emotions, the present invention has a significant performance. Through the emotion information quantification technology, the emotion in the speech can be accurately expressed in a computable way, providing a basis for subsequent processing. The emotion-based reward function construction technology can reasonably guide the model to optimize in the correct direction of emotion transmission during the training process, so that the translation result is highly consistent with the original speech in terms of emotion, greatly improving the naturalness and emotional coherence of the translation.

[0025] The present invention is also applicable to the field of automatic speech translation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is an overall structural block diagram of a speech large language model translation enhancement method based on emotion enhancement described in implementation mode 1.

[0027] Figure 2 This is a flow chart of reinforcement learning based on emotion feedback in a speech large language model translation reinforcement method based on emotion enhancement described in implementation mode 1.

[0028] Figure 3 This is a schematic diagram of an emotion-based speech translation template in a speech large language model translation enhancement method based on emotion enhancement described in Implementation Method 1.

[0029] Figure 4 This is a schematic diagram of the emotion-enhanced translation effect in the emotion-enhanced speech large language model translation enhancement method described in Implementation Method 1. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the implementation methods of the present application clearer, the technical solutions in the implementation methods of the present application will be clearly and completely described below in conjunction with the drawings in the implementation methods of the present application. Obviously, the described implementation methods are only part of the implementation methods of the present application, not all of the implementation methods.

[0031] Embodiment 1: This embodiment provides a method for enhancing the translation of a speech large language model based on emotion enhancement, the method comprising the following steps:

[0032] Step 1: collecting a multilingual data set containing at least one speech and text pair of emotional expression, and preprocessing the multilingual data set to form a multilingual corpus pair of original speech and text and translated speech and text;

[0033] Step 2: input the speech part of the multilingual corpus pair in the multilingual data set in step 1 into the speech large language model, and let the speech large language model translate it;

[0034] Step 3: The model with the highest emotional fit is used as the scoring model. If the speech emotion in the translation result of step 2 is consistent with the emotion in the multilingual corpus of the original speech and text in step 1, a positive reward is given; if they are inconsistent, a negative reward is given;

[0035] Step 4: Use the scoring model described in step 3 to score the different results of the same question in step 2, and perform the DPO algorithm based on the order of scores from high to low to update the parameters of the speech language model.

[0036] Implementation method 2: This implementation method further limits the method for enhancing the translation of a large speech language model based on emotion enhancement described in implementation method 1. The method for preprocessing the multilingual data set in step 1 includes noise reduction, feature extraction, and language screening.

[0037] Implementation method three: This implementation method further limits the method for enhancing speech large language model translation based on emotion enhancement described in implementation method one, and step 2 also includes the step of designing a speech translation template based on emotion.

[0038] Implementation mode 4: This implementation mode further limits the method for enhancing the translation of a large speech language model based on emotion enhancement described in implementation mode 3. The emotion-based speech translation template is implemented using an instruction template.

[0039] Implementation method five: This implementation method further limits the method for enhancing the translation of a large speech language model based on emotion enhancement described in implementation method one. The large speech language model described in step 2 is implemented using the Qwen2-audio model.

[0040] The speech large language model used in step 2 of this embodiment is not limited to the Qwen2-audio model. The speech large language model used in the prior art can also be used to translate it.

[0041] Implementation method six: This implementation method is a further limitation of the speech large language model translation enhancement method based on emotion enhancement described in implementation method one. In step 3, the emotion fit scoring model for speech is implemented using a HuBert-based emotion classification model.

[0042] Implementation method seven: This implementation method is a further limitation of the speech large language model translation enhancement method based on emotion enhancement described in implementation method six. The HuBert emotion classification model includes scoring of three dimensions of speech: Arousal, Dominance, and Valence.

[0043] Implementation 8: This implementation is a further limitation of the speech large language model translation enhancement method based on emotion enhancement described in Implementation 1. The DPO algorithm in step 4 is:

[0044]

[0045] Among them, x is the input voice, y w is the option with consistent sentiment in the translation result, y l is the option with inconsistent sentiment in the translation result, π θ (y|x) represents the predicted probability of the current model for translation y, π ref represents the probability distribution of the reference model, β is a hyperparameter that controls the degree to which the model deviates from the reference model, and σ is the logistic regression function for binary classification.

[0046] Embodiment 9: This embodiment proposes a computer storage medium for storing a computer program. When the computer program is read by a computer, the computer executes the method described in any one of embodiments 1 to 8.

[0047] Embodiment 10. This embodiment proposes a computer device, including a processor and a storage medium, wherein a computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the method described in any one of embodiments 1 to 8.

[0048] Implementation eleven: This implementation proposes an example, which is used to explain the above-mentioned implementations one to ten. The specific example is as follows:

[0049] See also Figures 1 to 4This embodiment is described. The purpose of this embodiment is to integrate emotional information into the translation of speech and text through the existing emotion classification model to improve the translation effect. Although the combination of emotional information can be fine-tuned using instructions, the combination of the advanced and popular reinforcement learning DPO algorithm and the speech large language model will achieve better results.

[0050] in, Figure 1 This is the structural block diagram of the speech language model based on which this implementation is based. In the reinforcement learning stage, we will mainly train and improve the parameters of the large model and the audio projection network.

[0051] In the speech large language model, the automatic translation task of speech needs to focus not only on the accuracy of semantics, but also on the effective processing and transmission of emotional information in speech. Emotional information plays a vital role in speech and has a direct impact on the naturalness and emotional consistency of translation. Therefore, a reinforcement learning strategy based on emotional feedback will be introduced to further enhance the emotion transmission ability of the speech large model when processing speech translation tasks.

[0052] DPO (Direct Policy Optimization) is an algorithm for optimizing policies in reinforcement learning. DPO aims to directly optimize the performance of the policy by searching in the policy space to find the optimal policy parameters so that the policy performs best in a given environment. The core idea of ​​the DPO algorithm is gradient-based optimization, which adjusts the policy parameters by calculating the expected return (i.e., policy gradient) of a policy performing an action in a certain state. The DPO optimization process based on emotion is as follows: Figure 2 shown.

[0053] Collect multilingual data of speech-text pairs containing various emotional expressions. Preprocess the speech data, including noise reduction, feature extraction, language screening, etc., to finally form multilingual corpus pairs of original speech and text and translated speech and text.

[0054] Input the speech part of the corpus in the data set of step 1 into the speech model and let the speech model translate it. Different results should be generated for the same translation question multiple times, so as to subsequently screen out answers that match the emotions and answers that do not match the emotions.

[0055] Determine a suitable sentiment fit scoring model. If the sentiment in the translation result of step 3 is consistent with the sentiment of the original speech, a positive reward is given; if it is inconsistent, a negative reward is given. Considering that the sentiment of the output text is difficult to identify, the BLEU score should also be taken into account. The higher the BLEU score, the more positive rewards should be given. The sentiment fit scoring model can be scored using GPT-4.

[0056] Use the sentiment fit scoring model in step 3 to score the different results of the same question in step 2, and perform the DPO algorithm based on the order of scores from high to low, and then update the model parameters.

[0057] Currently, many audio encoders for large speech and language models rely mainly on Transformer-based encoder models, but this model performs poorly in recognizing human emotions in speech. Figure 3 The speech translation instruction template shown in the figure is used to improve the effect of emotion recognition. By introducing more flexible instruction templates, the model's ability to process speech with different emotions can be enhanced, thereby improving the accuracy and naturalness of translation.

[0058] This implementation requires that the pre-trained speech large language model has excellent multilingual speech understanding capabilities, so we choose Qwen2-audio as the basic model. Qwen2-audio has powerful multilingual processing capabilities and can effectively capture speech features and speech semantic relationships in different languages. By adopting this model, we aim to improve the performance of the speech translation system in multilingual scenarios, thereby ensuring the accuracy and fluency of the translation. In addition, the Qwen2-audio audio encoder also chooses Whistper as the pre-trained encoder based on it. Although the Whisper model performs well in understanding speech content and semantic relationships, it has certain deficiencies in its ability to retain and understand emotions. This limitation may affect the accuracy and naturalness of the translation system when processing speech that needs to convey emotional colors. Therefore, in this implementation, the emotional information of the speech in the speech is input into the model to achieve better translation results.

[0059] This implementation will use the currently best HuBert-based sentiment classification model, which can score speech in three dimensions: Arousal, Dominance, and Valence. We will use the highest-scoring emotion as the emotional information of the speech and embed it into the translation instruction template to guide the model to generate different answers. For different translations generated based on this emotion, a BLEU value evaluation will be performed compared with the standard translation. A higher BLEU value indicates that the translation is more consistent with the original meaning, while a lower BLEU value reflects a lower emotional fit. The answers sorted by BLEU value will be used as a reference for the DPO algorithm to optimize the model parameters.

[0060] In this embodiment, the DPO algorithm based on emotional fit is a simplified reinforcement learning algorithm that avoids the complexity of traditional reinforcement learning by directly optimizing preferences. This method does not require explicit modeling rewards, but optimizes the strategy by maximizing the classification loss function. The core of this algorithm is to increase the probability of the model generating correct outputs and reduce the probability of generating incorrect outputs.

[0061] In this case, the DPO algorithm can be optimized by increasing the probability of translation results that maintain emotional consistency. DPO is optimized through binary cross entropy loss, making the probability of emotionally consistent translations greater and the probability of emotionally inconsistent translations smaller. The DPO loss function formula based on emotional consistency in the present embodiment in the dataset D is as follows:

[0062]

[0063] Where x is the input speech, y w is the option with consistent sentiment in the translation result, y l is the option with inconsistent sentiment in the translation result, π θ (y|x) represents the predicted probability of the current model for translation y, π ref represents the probability distribution of the reference model, β is a hyperparameter that controls the degree to which the model deviates from the reference model, and σ is the logistic regression function for binary classification.

[0064] The goal of this formula is to increase the sentiment-consistent translation y w and reduce the probability of sentiment inconsistency translation y l The model generates multiple translations for each speech input, and DPO ranks these translations based on sentiment consistency and adjusts the model parameters so that the sentiment-consistent translations have the highest scores.

[0065] The reinforcement learning strategy based on emotional feedback adopted in this embodiment includes emotional information quantification technology and emotion-based reward function construction technology, which can improve the emotion transmission ability of the large speech model in the task of automatic speech-to-text translation at a low cost. This strategy can deeply mine the emotional information in the speech and accurately map it to the translation results, and efficiently enhance the effect of the large speech language model on emotion processing during speech translation. In particular, when processing speech content containing rich emotions, this embodiment has a significant performance. Through the emotion information quantification technology, the emotions in the speech can be accurately expressed in a computable way, providing a basis for subsequent processing. The emotion-based reward function construction technology can reasonably guide the model to optimize in the correct direction of emotion transmission during the training process, so that the translation results are highly consistent with the original speech in terms of emotion, greatly improving the naturalness and emotional coherence of the translation. The final effect is shown as follows Figure 4 shown.

[0066] Those skilled in the art will appreciate that the above are only preferred embodiments of the present invention, and the various embodiments of the present disclosure and / or the features described in the claims may be combined or combined in various ways, even if such combinations or combinations are not explicitly described in the present disclosure. It is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art may still modify the technical solutions described in the aforementioned embodiments, or perform equivalent substitutions on some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

[0067] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A speech large language model translation enhancement method based on emotion enhancement, characterized in that: The method comprises the following steps: Step 1: collecting a multilingual data set containing at least one speech and text pair of emotional expression, and preprocessing the multilingual data set to form a multilingual corpus pair of original speech and text and translated speech and text; Step 2: input the speech part of the multilingual corpus pair in the multilingual data set in step 1 into the speech large language model, and let the speech large language model translate it; Step 3: Use the sentiment fit scoring model to evaluate the translation result in step 2. If the speech sentiment in the translation result in step 2 is consistent with the sentiment in the multilingual corpus pair of the original speech and text in step 1, a positive reward is given; if it is inconsistent, a negative reward is given; Step 4: Use the scoring model described in step 3 to score the different results of the same question in step 2, and perform the DPO algorithm based on the order of scores from high to low to update the parameters of the speech language model.

2. The method for enhancing speech large language model translation based on emotion enhancement according to claim 1, characterized in that: The method for preprocessing the multilingual data set in step 1 includes noise reduction, feature extraction, and language screening.

3. The method for enhancing speech large language model translation based on emotion enhancement according to claim 1, characterized in that: Step 2 also includes the step of designing a speech translation template based on emotion.

4. The method for enhancing speech large language model translation based on emotion enhancement according to claim 1, characterized in that: The emotion-based speech translation template is implemented using an instruction template.

5. The method for enhancing speech large language model translation based on emotion enhancement according to claim 1, characterized in that: The speech large language model described in step 2 is implemented using the Qwen2-audio model.

6. The method for enhancing speech translation based on a large language model based on emotion enhancement according to claim 1, characterized in that: In step 3, the emotion fit scoring model for speech is implemented using a HuBert-based emotion classification model.

7. The method for enhancing speech large language model translation based on emotion enhancement according to claim 6 is characterized in that: The HuBert emotion classification model includes scoring of three dimensions: Arousal, Dominance, and Valence of speech.

8. The method for enhancing speech large language model translation based on emotion enhancement according to claim 1, characterized in that: The DPO algorithm in step 4 is: Among them, x is the input voice, y w is the option with consistent sentiment in the translation result, y l is the option with inconsistent sentiment in the translation result, π θ (y|x) represents the predicted probability of the current model for translation y, π ref represents the probability distribution of the reference model, β is a hyperparameter that controls the degree to which the model deviates from the reference model, and σ is the logistic regression function for binary classification.

9. A computer storage medium for storing a computer program, characterized in that: When the computer program is read by a computer, the computer executes the method according to any one of claims 1 to 8.

10. A computer device comprising a processor and a storage medium, characterized in that A computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the method according to any one of claims 1 to 8.

Images