A voice emotion recognition model construction method and device

Abstract

The application discloses a speech emotion recognition model construction method and device. The method comprises the following steps: acquiring speech data; encoding audio features of the speech data to obtain audio encoding; and encoding emotion labels corresponding to the speech data to obtain emotion encoding; determining target encoding according to the audio encoding and the emotion encoding; and training a pre-determined to-be-trained speech emotion recognition model based on the target encoding to obtain a target speech emotion recognition model updated after training. According to the technical scheme, the encoded speech data is subjected to emotion recognition by constructing the target speech emotion recognition model, and the accuracy and precision of the recognition result can be effectively improved.

Description

Technical Field

[0001] This invention relates to the field of speech emotion recognition technology, and in particular to a method and apparatus for constructing a speech emotion recognition model. Background Technology

[0002] Most existing speech emotion recognition methods first convert speech data into text data, then input the text data into a pre-trained emotion recognition model. The model then classifies and outputs an emotion label corresponding to the input text data, thereby obtaining the emotion associated with the speech data. However, this method has significant drawbacks. During the process of converting speech to text before performing emotion recognition, a large amount of key information contained in the speech data, such as tone, speed, and intonation, is lost, leading to insufficient accuracy and a high error rate in emotion recognition. Summary of the Invention

[0003] This invention provides a method and apparatus for constructing a speech emotion recognition model. Based on the audio encoding and emotion encoding of speech data, a target speech emotion recognition model is constructed, which can make full use of the contextual information in the speech data, thereby effectively improving the accuracy and precision of the emotion recognition results in the emotion recognition process.

[0004] According to one aspect of the present invention, a method for constructing a speech emotion recognition model is provided, the method comprising:

[0005] Acquire voice data;

[0006] The audio features of the speech data are encoded to obtain audio encoding; and the emotion tags corresponding to the speech data are encoded to obtain emotion encoding.

[0007] Based on the audio encoding and the emotion encoding, the target encoding is determined;

[0008] The target encoding is used to train a pre-determined speech emotion recognition model to obtain a trained and updated target speech emotion recognition model.

[0009] According to another aspect of the present invention, a speech emotion recognition model building apparatus is provided, the apparatus comprising:

[0010] The voice data acquisition module is used to acquire voice data;

[0011] The encoding module is used to encode the audio features of the speech data to obtain audio encoding; and to encode the emotion tags corresponding to the speech data to obtain emotion encoding;

[0012] The target encoding determination module is used to determine the target encoding based on the audio encoding and the emotion encoding;

[0013] The target speech emotion recognition model acquisition module is used to train a pre-determined speech emotion recognition model to be trained based on the target encoding, so as to obtain the trained and updated target speech emotion recognition model.

[0014] The technical solution of this invention acquires speech data, then encodes the audio features and corresponding emotion tags of the speech data to obtain audio codes and emotion codes. The audio codes and emotion codes are fused to generate a target code, and a pre-determined speech emotion recognition model is trained based on the target code to obtain a trained and updated target speech emotion recognition model. This technical solution, by constructing a target speech emotion recognition model based on the audio codes and emotion codes of speech data, can fully utilize the contextual information in the speech data, thereby effectively improving the accuracy and precision of emotion recognition results during the emotion recognition process. It effectively solves the problems of insufficient accuracy and high error rate in existing speech emotion recognition technologies.

[0015] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of a speech emotion recognition model construction method provided in Embodiment 1 of the present invention;

[0018] Figure 2 This is a schematic diagram illustrating the process of constructing a speech emotion recognition model according to Embodiment 2 of the present invention;

[0019] Figure 3 This is a schematic diagram illustrating the construction process of another speech emotion recognition model provided in Embodiment 3 of the present invention;

[0020] Figure 4 This is a schematic diagram illustrating the construction process of another speech emotion recognition model provided in Embodiment 4 of the present invention;

[0021] Figure 5 A flowchart illustrating another method for constructing a speech emotion recognition model provided in Embodiment 5 of the present invention;

[0022] Figure 6A flowchart illustrating the application process of the target speech emotion recognition model provided in Embodiment Six of the present invention;

[0023] Figure 7 This is a schematic diagram of a speech emotion recognition model construction device provided in Embodiment 7 of the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] Example 1

[0027] Figure 1 This is a flowchart illustrating a method for constructing a speech emotion recognition model according to Embodiment 1 of the present invention. This embodiment is applicable to situations involving the recognition of speech emotions. The method can be executed by a speech emotion recognition model construction device, which can be implemented in hardware and / or software and can be configured within a device. For example, the device can be a backend server or other device with communication and computing capabilities. Figure 1 As shown, the method includes:

[0028] S110, Acquire voice data.

[0029] In this scheme, voice data refers to the digital representation of various information that exists in the form of voice.

[0030] In this embodiment, voice data can be obtained from a voice database; voice data can also be collected using a recording tool; or voice data can be obtained from a cloud platform.

[0031] S120. Encode the audio features of the speech data to obtain audio encoding; and encode the emotion tags corresponding to the speech data to obtain emotion encoding.

[0032] Audio features refer to various parameters and attributes that describe the characteristics of an audio signal. For example, audio features can include amplitude, frequency, spectrum, formants, and harmonics. Audio coding is the process of processing audio features using specific algorithms to convert them into a format more suitable for storage and transmission.

[0033] In this embodiment, emotion tags are used to describe and define various emotional states. For example, emotion tags can be happiness, anger, sadness, fear, etc. Emotion encoding is the process of processing emotion tags using a specific algorithm to convert them into a format more suitable for storage and transmission. The number of emotion tags can be one or multiple.

[0034] In this scheme, audio features are obtained by extracting features from speech data, and then audio features are encoded to obtain audio codes. For example, for 44kHz speech data, every 440 audio sampling points are extracted into one audio code, thus obtaining 100Hz audio codes, that is, 100 audio codes are obtained per second of speech.

[0035] Audio coding can consist of a single-layer or multi-layer structure. Multi-layer audio coding, by superimposing different levels of coding, can capture and retain richer and more detailed audio features.

[0036] Specifically, feature extraction algorithms can be used to extract features from speech data to obtain audio features; alternatively, pre-trained feature extraction models can be used to extract features from speech data to obtain audio features. Feature extraction algorithms can include Mel-frequency cepstral coefficients, linear predictive cepstral coefficients, etc. Feature extraction models can include machine learning models, deep learning models, etc.

[0037] Furthermore, after obtaining the audio features, an encoder can be used to encode the audio features to obtain the audio encoding; alternatively, a pre-trained encoding model can be used to encode the audio features to obtain the audio encoding. The encoder can be a MERT encoder or a Mel encoder. The encoding model can be a machine learning model, a deep learning model, etc.

[0038] In this embodiment, speech data can be processed based on machine learning algorithms to determine the emotion labels corresponding to the speech data; alternatively, speech data can be processed based on manual annotation methods to determine the emotion labels corresponding to the speech data.

[0039] Furthermore, after determining the emotion labels corresponding to the speech data, an encoder can be used to encode the emotion labels to obtain the emotion code; alternatively, a pre-trained encoding model can be used to encode the emotion labels to obtain the emotion code. The encoder can be a MERT encoder or a Mel encoder. The encoding model can be a machine learning model, a deep learning model, etc.

[0040] S130. Determine the target code based on the audio code and the emotion code.

[0041] In this embodiment, the audio code and the emotion code can be concatenated to obtain the target code.

[0042] In this scheme, the target encoding is a new encoding obtained by concatenating audio encoding and emotion encoding. This new encoding integrates audio and emotion information, enabling a more comprehensive description of the content and emotional features contained in the speech data. The target encoding can be represented using row vectors or matrices.

[0043] Optionally, the speech data includes speech data of at least one sentence of contextual information and speech data for predicting emotion. The step of determining the target code based on the audio encoding and the emotion encoding includes:

[0044] The audio codes corresponding to the speech data of the context and / or background information, the audio codes corresponding to the speech data for predicting emotion, and the emotion codes are all concatenated as fitting data to determine the target code; or,

[0045] The audio codes corresponding to the speech data of the preceding and / or following information are used as prompt data. The audio codes corresponding to the speech data of the predicted emotion and the emotion codes are concatenated as fitting data to determine the target code.

[0046] In this embodiment, the voice data containing contextual and / or background information refers to the content surrounding the dialogue before and after the target voice segment, which can cover multi-turn interaction information to provide contextual support; the voice data for predicting emotions refers to the target voice segment to be analyzed.

[0047] The voice data of the preceding and / or following information, as well as the voice data for predicting emotions, can be attributed to the same role or generated by different roles.

[0048] In this solution, sentiment analysis of speech data during natural language processing often relies on contextual information. The same sentence can convey vastly different emotions depending on the context. For example, if the speech data used to predict sentiment is "You're really amazing," and the preceding text is "With extraordinary wisdom and tireless efforts, you successfully overcame many difficult problems and achieved groundbreaking results, which is truly admirable," then "You're really amazing" conveys a strong sense of praise. However, if the preceding text becomes "Frequent errors in key areas caused huge losses to the team, yet you still tried to justify yourself," then "You're really amazing" will create an ironic effect through semantic contrast.

[0049] Specifically, the audio codes corresponding to the speech data of the context and / or background information, the audio codes corresponding to the speech data of the predicted emotion, and the emotion codes can be concatenated to generate the target code.

[0050] In this scheme, the audio codes corresponding to the speech data of contextual and / or background information can also be used as cue data. This cue data can provide key background information and guiding clues for the subsequent emotion recognition process, helping the model to more accurately capture the emotional features in the speech. Based on this, the audio codes and emotion codes corresponding to the speech data for predicting the emotion are concatenated to generate the target code.

[0051] By integrating contextual information into speech data for emotion recognition, we can not only effectively improve the accuracy of emotion recognition, but also deeply analyze complex emotional expressions, providing a more comprehensive and accurate basis for sentiment analysis.

[0052] Optionally, the speech data containing at least one sentence of context information includes speech data of at least two characters, wherein the speech data of different characters are segmented by a third encoding.

[0053] In this embodiment, the third encoding can be represented by characters, numbers, or complex data structures.

[0054] In this scheme, third-party coding is a type of coding specifically designed to represent character-related information. When the voice data contains voice data from at least two characters, third-party coding can be used to distinguish the voice data of different characters. For example, in a dialogue scenario, multiple different characters interact with each other. Character A says, "I encountered something really interesting today," and Character B responds, "Really? Tell me about it!" In this case, the content said by Character B can be analyzed in depth using third-party coding to understand the emotions expressed.

[0055] By segmenting the voice data of different characters using a third encoding method, the characters' emotions can be presented in a quantifiable and analyzable way, providing strong support for subsequent in-depth research, analysis, and decision-making.

[0056] S140. Based on the target encoding, the pre-determined speech emotion recognition model to be trained is trained to obtain the updated target speech emotion recognition model.

[0057] The target speech emotion recognition model is an autoregressive language model.

[0058] In this scheme, the target encoding is used as input to iteratively train the speech emotion recognition model. In this way, the model can effectively capture more contextual information, realize in-depth mining of speech emotion features, thereby significantly improving the accuracy of emotion recognition and building a target speech emotion recognition model with better performance.

[0059] The technical solution of this invention involves acquiring speech data, then encoding the audio features and corresponding emotion tags of the speech data to obtain audio codes and emotion codes. The audio codes and emotion codes are then fused to determine a target code, and a pre-determined speech emotion recognition model is trained based on the target code to obtain a trained and updated target speech emotion recognition model. By implementing this technical solution, a target speech emotion recognition model is constructed based on the audio codes and emotion codes of the speech data, which can fully utilize the contextual information in the speech data, thereby effectively improving the accuracy and precision of emotion recognition results during the emotion recognition process.

[0060] Example 2

[0061] Figure 2 This is a schematic diagram illustrating the construction process of a speech emotion recognition model according to Embodiment 2 of the present invention. The relationship between this embodiment and the above embodiments is a detailed description of the target encoding determination process. Figure 2 As shown, the method includes:

[0062] S210, Acquire voice data.

[0063] S220. Encode the audio features of the speech data to obtain audio encoding; and encode the emotion tags corresponding to the speech data to obtain emotion encoding.

[0064] S230. The audio code and the emotion code are concatenated according to a preset concatenation order to obtain the target code; wherein, the preset concatenation order includes concatenating the emotion code before and / or after the audio code.

[0065] In this scheme, the emotion code can be concatenated before the audio code to generate the target code. For example, if the emotion code is ABC and the audio code is DEF, the concatenated target code can be ABCDEF, or it can be represented in the form of [ABC, DEF].

[0066] In this embodiment, the emotion code can also be concatenated after the audio code to generate the target code. For example, if the emotion code is ABC and the audio code is DEF, the concatenated target code can be DEFABC, or it can be represented in the form of [DEF, ABC].

[0067] Following a pre-defined splicing order helps standardize and normalize data processing. This allows audio and emotion codes from different sources and formats to be combined in a unified way, facilitating subsequent data storage, management, and transmission. It also helps ensure data consistency and stability during processing, reducing errors and deviations caused by inconsistent data formats.

[0068] S240. Based on the target encoding, the pre-determined speech emotion recognition model to be trained is trained to obtain the updated target speech emotion recognition model.

[0069] The technical solution of this invention involves acquiring speech data, then encoding the audio features and corresponding emotion tags of the speech data to obtain audio codes and emotion codes. The audio codes and emotion codes are concatenated to determine the target code, and a pre-determined speech emotion recognition model is trained based on the target code to obtain a trained and updated target speech emotion recognition model. By implementing this technical solution, a target speech emotion recognition model is constructed based on the audio codes and emotion codes of the speech data, which can fully utilize the contextual information in the speech data, thereby effectively improving the accuracy and precision of emotion recognition results during the emotion recognition process.

[0070] Example 3

[0071] Figure 3 This is a schematic diagram illustrating the construction process of another speech emotion recognition model provided in Embodiment 3 of the present invention. The relationship between this embodiment and the above embodiments is a detailed description of the target encoding determination process. Figure 3 As shown, the method includes:

[0072] S310, Acquire voice data.

[0073] S320. Encode the audio features of the speech data to obtain audio encoding; and encode the emotion tags corresponding to the speech data to obtain emotion encoding.

[0074] S330. The audio encoding is processed in layers to obtain a first layer audio encoding and a second layer audio encoding; wherein the granularity of the first layer audio encoding is larger than that of the second layer audio encoding, or the number of first layer audio encodings is smaller than that of the second layer audio encoding.

[0075] In this scheme, audio encoding can be layered to obtain first-layer and second-layer audio encodings with different granularities. The granularity of the first-layer audio encoding is larger than that of the second-layer audio encoding.

[0076] Specifically, the audio code is mapped to the closest discretized first-layer audio code. The residual information between the actual audio code and the first-layer audio code is calculated, and the residual information is compared with the discretized second-layer audio code, selecting the closest second-layer audio code.

[0077] This involves finding the first layer of audio encoding that most closely resembles the original audio encoding based on preset discretization rules. For example, a quantization range can be set, dividing the audio encoding into different discrete values ​​according to its numerical range, and then selecting the closest discrete value as the corresponding first layer of audio encoding.

[0078] In this embodiment, the residual information between the actual audio code and the first-layer audio code can be calculated using a time-domain residual calculation method. Alternatively, the residual information between the actual audio code and the first-layer audio code can be calculated based on a machine learning model.

[0079] Furthermore, the closest second-layer audio code can be found by analyzing the similarity between the residual information and the discretized second-layer audio code. For example, the distance between the residual information and each discretized second-layer audio code can be calculated and used as an indicator of the degree of similarity. The smaller the distance value, the higher the similarity between the two. Finally, the second-layer audio code that is closest to the residual information is selected.

[0080] This layered encoding strategy, by superimposing the first and second layers of audio encoding, can approximate the original audio features with higher accuracy. Simultaneously, it significantly reduces encoding storage overhead and model training complexity. That is, while traditional single-layer encoding requires constructing an N×M encoding library to achieve the same restoration effect, the layered encoding framework only requires a two-layer encoding library of N+M. Furthermore, as the number of encoding layers increases, the improvement in compression efficiency and signal fidelity becomes increasingly significant.

[0081] In this scheme, different numbers of first-layer and second-layer audio codes can be obtained by layering the audio encoding. The number of first-layer audio codes is less than the number of second-layer audio codes.

[0082] Specifically, the audio encoding is sampled at a preset sampling rate to extract a preset number of first-layer audio codes, and then a preset number of second-layer audio codes are extracted from these first-layer codes. The audio code extraction is performed from bottom to top, and the last extracted code is used as the first-layer audio code, which is a highly compressed result. Based on this first-layer audio code, refinement is performed layer by layer. During input, each layer of audio code is input sequentially according to its layer order, and the output is also processed layer by layer. For example, by setting 100 sampling points to sample the audio encoding, one first-layer audio code is extracted, and one second-layer audio code is extracted from the ten extracted first-layer audio codes.

[0083] By employing a bottom-up sampling approach, the final extracted code is used as the first layer of audio encoding, resulting in highly compressed data that reduces data volume and facilitates storage and transmission. Following a hierarchical order ensures a certain degree of accuracy in audio encoding. In audio analysis, precise audio encoding provides more accurate audio feature information, aiding in more accurate audio emotion recognition.

[0084] S340. The first layer audio code and the second layer audio code are concatenated with the emotion code according to a preset concatenation order to obtain the target code; wherein, the preset concatenation order includes concatenating the emotion code before the first layer audio code and the second layer audio code; or concatenating the emotion code after the first layer audio code and the second layer audio code.

[0085] Specifically, the emotion code can be placed before the first and second audio codes to generate the target code; alternatively, the emotion code can be placed after the first and second audio codes to generate the target code.

[0086] In this scheme, audio encoding is processed in layers to obtain first-layer audio encoding and second-layer audio encoding with different granularities. During the splicing process, the first-layer audio encoding and the second-layer audio encoding can be superimposed, delayed, or spliced ​​together.

[0087] In this embodiment, different numbers of first-layer audio codes and second-layer audio codes are obtained by performing layered processing on the audio codes. During the splicing process, the first-layer audio codes and second-layer audio codes can be superimposed or spliced ​​one by one, that is, 10 second-layer audio codes corresponding to 1 first-layer audio code can be superimposed or spliced; or they can be spliced ​​layer by layer to form the target code, that is, 10 first-layer audio codes first, and the corresponding 100 second-layer audio codes second, are combined to form the target code.

[0088] S350. Based on the target encoding, the pre-determined speech emotion recognition model to be trained is trained to obtain the updated target speech emotion recognition model.

[0089] The technical solution of this invention involves acquiring speech data, then encoding the audio features and corresponding emotion tags of the speech data to obtain audio codes and emotion codes. The audio codes are processed in layers to obtain a first-layer audio code and a second-layer audio code. These first-layer, second-layer, and emotion codes are then combined and concatenated to determine the target code. Based on the target code, a pre-determined speech emotion recognition model is trained to obtain a trained and updated target speech emotion recognition model. By implementing this technical solution, a target speech emotion recognition model is constructed based on the audio codes and emotion codes of the speech data. This fully utilizes the contextual information in the speech data, thereby effectively improving the accuracy and precision of emotion recognition results during the emotion recognition process.

[0090] Example 4

[0091] Figure 4 This is a schematic diagram illustrating the construction process of another speech emotion recognition model provided in Embodiment 4 of the present invention. The relationship between this embodiment and the above embodiments is a detailed description of the target encoding determination process. Figure 4 As shown, the method includes:

[0092] S410, Acquire voice data.

[0093] S420. Encode the audio features of the speech data to obtain audio encoding; and encode the emotion tags corresponding to the speech data to obtain emotion encoding.

[0094] S430. Add a preset first code at the beginning of the audio encoding and add a preset second code at the end of the audio encoding; wherein, the first code is used to indicate the start of prediction by the speech emotion recognition model to be trained; and the second code is used to indicate the end of audio prediction by the speech emotion recognition model to be trained.

[0095] In this embodiment, the first encoding and the second encoding can be represented by characters, numbers, or complex data structures. The first encoding and the second encoding can be set based on the model training requirements. For example, the first encoding can be 0, and the second encoding can be 1.

[0096] In this embodiment, during the training process of the speech emotion recognition model, the first encoding is used to mark the time node when the speech emotion recognition model to be trained begins prediction. This means that from that moment on, the model will initiate the audio feature analysis and prediction process for the input audio encoding. The second encoding is used to mark the end of audio prediction for the speech emotion recognition model to be trained. It precisely instructs the speech emotion recognition model to be trained to complete all prediction tasks for the audio encoding, and can initiate the analysis and prediction of emotion features for the input audio encoding. After outputting the corresponding predicted emotion encoding, it marks the formal end of the model's emotion analysis process for the specific audio. Through the orderly cooperation of these two encodings, the prediction process of the model can be effectively standardized and managed, ensuring the accuracy and efficiency of the training process of the speech emotion recognition model to be trained.

[0097] Specifically, the integrity of the audio encoding is marked by embedding a pre-set first code at the beginning of the audio encoding and adding a pre-set second code at the end.

[0098] In this solution, the audio encoding can be directly modified to embed the first and second encodings into the start and end positions of the audio encoding, respectively; alternatively, digital watermarking technology can be used to embed the first and second encodings into the start and end positions of the audio encoding, respectively.

[0099] The first and second codes serve as clear start and end markers, allowing the speech emotion recognition model to accurately identify the start and end points of the audio encoding. This prevents the model from mistakenly including irrelevant audio encodings or incorrectly truncating audio encodings, thus improving the accuracy and stability of emotion recognition. Furthermore, the first code indicates the start of the prediction, enabling the model to quickly enter a working state and avoiding unnecessary computation and waiting time. The second code allows the model to analyze and process the audio encoding in a timely manner, predicting and outputting the emotion encoding, preventing the model from continuing ineffective operation after the audio ends, thereby saving computational resources and improving overall recognition efficiency.

[0100] S440. Set the emotion code after the second code to construct the target code.

[0101] Specifically, the first code is added at the beginning of the audio encoding, the second code is added at the end, and the emotion code is set after the second code to construct the target code.

[0102] S450. Based on the target encoding, the pre-determined speech emotion recognition model to be trained is trained to obtain the updated target speech emotion recognition model.

[0103] The technical solution of this invention involves acquiring speech data, then encoding the audio features and corresponding emotion tags of the speech data to obtain audio encoding and emotion encoding. A preset first encoding is added at the beginning of the audio encoding, and a preset second encoding is added at the end of the audio encoding. The emotion encoding is then placed after the second encoding to construct a target encoding. Based on the target encoding, a pre-determined speech emotion recognition model to be trained is trained to obtain a trained and updated target speech emotion recognition model. By implementing this technical solution, a target speech emotion recognition model is constructed based on the audio encoding and emotion encoding of the speech data. This fully utilizes the contextual information in the speech data, thereby effectively improving the accuracy and precision of the emotion recognition results during the emotion recognition process.

[0104] Example 5

[0105] Figure 5 This is a flowchart illustrating another method for constructing a speech emotion recognition model according to Embodiment 5 of the present invention. The relationship between this embodiment and the above embodiments is a detailed description of the training process of the speech emotion recognition model to be trained. The speech emotion recognition model to be trained is an autoregressive language model, such as... Figure 5 As shown, the method includes:

[0106] S510, acquire voice data.

[0107] S520. Encode the audio features of the speech data to obtain audio encoding; and encode the emotion tags corresponding to the speech data to obtain emotion encoding.

[0108] S530. Determine the target code based on the audio code and the emotion code.

[0109] S540. Determine the initial parameters of the speech emotion recognition model to be trained.

[0110] In this embodiment, the speech emotion recognition model to be trained is an autoregressive language model.

[0111] Specifically, the initial parameters of the speech emotion recognition model to be trained include network structure parameters and training hyperparameters. Network structure parameters include the number of model layers, the number of attention heads, and the embedding dimension; these parameters determine the model's feature extraction capability and information interaction efficiency. Training hyperparameters include key indicators such as the learning rate and the number of training epochs, used to regulate the model's training process and convergence speed. By scientifically configuring these parameters, the model's performance in speech emotion recognition tasks can be effectively optimized.

[0112] S550. Input the target code into the speech emotion recognition model to be trained and output the prediction code, wherein the prediction code includes the prediction speech code and the prediction emotion code.

[0113] In this scheme, the target code is input into the speech emotion recognition model to be trained. The speech emotion recognition model to be trained performs in-depth analysis and processing of the target code, performs prediction operation, and finally outputs the predicted speech code and the predicted emotion code.

[0114] S560. Based on the predicted encoding and the target encoding, determine the loss function value;

[0115] The loss function is a function used to measure the difference between the model's predictions and the actual results. Loss functions can include mean squared error, mean absolute error, mean absolute percentage error, log loss function, etc.

[0116] Specifically, during model training, the loss function value is calculated using predictive encoding and target encoding. By calculating this loss function value, the difference between the model's predictive encoding and target encoding can be scientifically quantified, thereby comprehensively and accurately evaluating the model's predictive performance and generalization ability in the current training state.

[0117] Optionally, determining the loss function value based on the predicted encoding and the target encoding includes:

[0118] Determine the weights of the audio encoding and the emotion encoding;

[0119] Based on the weights of the audio encoding and the emotion encoding, the loss function values ​​of the audio encoding and the emotion encoding are weighted and combined to determine the loss function value.

[0120] The weights of audio encoding and emotion encoding can be set based on machine learning algorithms; alternatively, they can be set based on experience or domain knowledge.

[0121] In this embodiment, the sum of the weights of the audio encoding and the emotion encoding is equal to one.

[0122] In this scheme, the loss function values ​​of audio encoding and emotion encoding are weighted and calculated according to their respective weights to obtain the final loss function value. Specifically, the loss function value of audio encoding is multiplied by the weight of audio encoding, and the loss function value of emotion encoding is multiplied by the weight of emotion encoding, and then the two products are added together.

[0123] This weighted combination approach comprehensively considers the impact of audio encoding and emotion encoding on the final result, thereby determining a comprehensive loss function value. This comprehensive loss function value can be used for subsequent model training and optimization.

[0124] In this embodiment, different weights can be assigned to audio encoding and emotion encoding when calculating the loss function value. Optionally, the weight of emotion encoding can be set to be greater than that of audio encoding, i.e., emotion encoding is given a higher weight and audio encoding a lower weight; alternatively, the weight of emotion encoding can be set to one and the weight of audio encoding to zero. In this case, audio encoding only serves as auxiliary information to help the model understand the contextual semantics, enabling the model to output predicted codes more accurately. However, audio encoding will not participate in the model's parameter optimization process. This strategy can effectively guide the training direction of the speech emotion recognition model to be trained, enabling it to converge quickly and significantly improving the accuracy and efficiency of emotion prediction.

[0125] S570. Adjust the initial parameters of the speech emotion recognition model to be trained according to the loss function value to obtain the updated target speech emotion recognition model.

[0126] In this scheme, the gradients of each initial parameter in the model are calculated using the loss function value and the backpropagation algorithm. The gradient represents the rate of change of the loss function under the current parameters, indicating the direction and magnitude of parameter adjustment. Then, based on the calculated gradients, an appropriate optimization algorithm, such as stochastic gradient descent (SGD) or adaptive moment estimation (Adam), is selected to update the initial parameters.

[0127] Furthermore, during the parameter update process, the entire target encoding needs to be trained iteratively multiple times. Each iteration repeats the above process. Through continuous iterative training, the model parameters are gradually adjusted, making the model's predicted encoding increasingly closer to the target encoding, and the loss function value gradually decreases. When the loss function value reaches a pre-set small value, or when the change in the loss function value is no longer significant after multiple iterations, the model is considered to have converged. The model obtained at this point is the trained and updated target speech emotion recognition model.

[0128] In this approach, the target encoding, encompassing both audio and emotion encoding, is used as training data during the training of the speech emotion recognition model. The model predicts and fits the audio and emotion encodings from the training data. Unlike existing emotion classification models, this speech emotion recognition model learns not only the emotion classification information corresponding to the text semantic output but also richer information such as context, intonation, tone, stress, and speech rate. By comprehensively considering all audio features, the model outputs the corresponding emotion encoding, significantly improving the accuracy of emotion judgment.

[0129] The technical solution of this invention involves encoding the audio features and corresponding emotion tags of the acquired speech data to obtain audio encoding and emotion encoding. The audio encoding and emotion encoding are then concatenated to determine the target encoding. Next, the initial parameters of the speech emotion recognition model to be trained are determined, and the target encoding is input into the model, which outputs a predicted encoding. Based on the predicted encoding and the target encoding, a loss function value is calculated and determined. The initial parameters of the speech emotion recognition model to be trained are then adjusted according to this loss function value, ultimately resulting in a trained and updated target speech emotion recognition model. By implementing this technical solution, a target speech emotion recognition model is constructed based on the audio encoding and emotion encoding of the speech data, which can fully utilize the contextual information in the speech data, thereby effectively improving the accuracy and precision of emotion recognition results during the emotion recognition process.

[0130] Example 6

[0131] Figure 6 This is a flowchart illustrating the application process of the target speech emotion recognition model provided in Embodiment Six of the present invention. The relationship between this embodiment and the previous embodiments is a detailed description of the application process of the trained target speech emotion recognition model. Figure 6 As shown, the method includes:

[0132] S610, Acquire target speech data.

[0133] In this scheme, target speech data refers to the digital representation of various information existing in speech form.

[0134] In this embodiment, target voice data can be obtained from a voice database; target voice data can also be collected using a recording tool; or target voice data can be obtained from a cloud platform.

[0135] S620. The target speech data is encoded and input into the target speech emotion recognition model, and the target emotion code is output.

[0136] In this scheme, target audio features are obtained by extracting features from target speech data, and target audio codes are obtained by encoding the target audio features.

[0137] The target audio coding can consist of a single-layer or multi-layer structure. Multi-layer audio coding, by superimposing different levels of coding, can capture and retain richer and more detailed audio features.

[0138] Specifically, feature extraction algorithms can be used to extract features from target speech data to obtain target audio features; alternatively, pre-trained feature extraction models can be used to extract features from target speech data to obtain target audio features. Feature extraction algorithms can include Mel-frequency cepstral coefficients, linear predictive cepstral coefficients, etc. Feature extraction models can include machine learning models, deep learning models, etc.

[0139] Furthermore, after obtaining the target audio features, an encoder can be used to encode the target audio features to obtain the target audio encoding; alternatively, a pre-trained encoding model can be used to encode the target audio features to obtain the target audio encoding. The encoder can include a MERT encoder and a Mel encoder. The encoding model can include machine learning models, deep learning models, etc.

[0140] In this embodiment, after obtaining the target audio code, the target audio code is input into the target speech emotion recognition model, which then predicts the target audio code and outputs the target emotion code.

[0141] Specifically, when processing input target speech data, the target speech emotion recognition model predicts and generates both a complete predicted audio code and a predicted emotion code. However, when finally sampling and outputting, only the predicted emotion code is output. This approach allows the model to fully consider and utilize all contextual and audio information during the emotion coding prediction process.

[0142] S630. Based on the target emotion encoding, determine the target emotion label corresponding to the target speech data.

[0143] In this solution, decoding technology can be used to parse the target emotion code and accurately output the emotion label corresponding to the target speech data.

[0144] The technical solution of this invention involves acquiring target speech data, encoding the target speech data, inputting it into a target speech emotion recognition model, outputting a target emotion code, and determining the target emotion tag corresponding to the target speech data based on the target emotion code. By implementing this technical solution and constructing a target speech emotion recognition model to perform emotion recognition on the encoded speech data, the accuracy and precision of the recognition results can be effectively improved.

[0145] Example 7

[0146] Figure 7 This is a schematic diagram of a speech emotion recognition model construction device provided in Embodiment 7 of the present invention. Figure 7 As shown, the device includes:

[0147] The voice data acquisition module 710 is used to acquire voice data;

[0148] The encoding module 720 is used to encode the audio features of the speech data to obtain audio encoding; and to encode the emotion tags corresponding to the speech data to obtain emotion encoding;

[0149] The target encoding determination module 730 is used to determine the target encoding based on the audio encoding and the emotion encoding;

[0150] The target speech emotion recognition model acquisition module 740 is used to train a pre-determined speech emotion recognition model to be trained based on the target encoding, so as to obtain the trained and updated target speech emotion recognition model.

[0151] Optionally, the target encoding determination module 730 is specifically used for:

[0152] The audio code and the emotion code are concatenated according to a preset concatenation order to obtain the target code; wherein, the preset concatenation order includes concatenating the emotion code before and / or after the audio code.

[0153] Optionally, the target encoding determination module 730 is also used for:

[0154] The audio encoding is processed in layers to obtain a first layer of audio encoding and a second layer of audio encoding; wherein the granularity of the first layer of audio encoding is larger than that of the second layer of audio encoding, or the number of first layer audio encodings is smaller than that of the second layer of audio encoding.

[0155] The first layer audio code and the second layer audio code are concatenated with the emotion code according to a preset concatenation order to obtain the target code; wherein, the preset concatenation order includes concatenating the emotion code before the first layer audio code and the second layer audio code; or concatenating the emotion code after the first layer audio code and the second layer audio code.

[0156] Optionally, the target encoding determination module 730 is also used for:

[0157] A preset first code is added at the beginning of the audio encoding and a preset second code is added at the end of the audio encoding; wherein, the first code is used to indicate the start of prediction by the speech emotion recognition model to be trained; and the second code is used to indicate the end of audio prediction by the speech emotion recognition model to be trained.

[0158] The emotion code is set after the second code to construct the target code.

[0159] Optionally, the speech emotion recognition model to be trained is an autoregressive language model, and the target speech emotion recognition model obtaining module 740 includes:

[0160] The initial parameter determination unit is used to determine the initial parameters of the speech emotion recognition model to be trained.

[0161] A predictive coding output unit is used to input the target coding into the speech emotion recognition model to be trained and output predictive coding, wherein the predictive coding includes predictive speech coding and predictive emotion coding;

[0162] The loss function value determination unit is used to determine the loss function value based on the predicted encoding and the target encoding;

[0163] The target speech emotion recognition model obtaining unit is used to adjust the initial parameters of the speech emotion recognition model to be trained based on the loss function value, so as to obtain the trained and updated target speech emotion recognition model.

[0164] Optional, loss function value determination unit, specifically used for:

[0165] Determine the weights of the audio encoding and the emotion encoding;

[0166] Based on the weights of the audio encoding and the emotion encoding, the loss function values ​​of the audio encoding and the emotion encoding are weighted and combined to determine the loss function value.

[0167] Optionally, the speech data includes speech data of at least one sentence of contextual information and speech data for predicting emotion. The target encoding determination module 730 is further configured to:

[0168] The audio codes corresponding to the speech data of the context and / or background information, the audio codes corresponding to the speech data for predicting emotion, and the emotion codes are all concatenated as fitting data to determine the target code; or,

[0169] The audio codes corresponding to the speech data of the preceding and / or following information are used as prompt data. The audio codes corresponding to the speech data of the predicted emotion and the emotion codes are concatenated as fitting data to determine the target code.

[0170] The speech emotion recognition model construction device provided in this embodiment of the invention can execute the speech emotion recognition model construction method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0171] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0172] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for constructing a speech emotion recognition model, characterized in that, include: Acquire voice data; The audio features of the speech data are encoded to obtain the audio code; And the emotion tags corresponding to the voice data are encoded to obtain emotion codes; Based on the audio encoding and the emotion encoding, the target encoding is determined; The target speech emotion recognition model is trained based on the target encoding to obtain the updated target speech emotion recognition model; wherein, the target speech emotion recognition model is an autoregressive language model. The determination of the target code based on the audio code and the emotion code includes: The audio encoding is processed in layers to obtain a first layer of audio encoding and a second layer of audio encoding; wherein the granularity of the first layer of audio encoding is larger than that of the second layer of audio encoding, or the number of first layer audio encodings is smaller than that of the second layer of audio encoding. The first layer audio code and the second layer audio code are concatenated with the emotion code according to a preset concatenation order to obtain the target code; wherein, the preset concatenation order includes concatenating the emotion code before the first layer audio code and the second layer audio code; or concatenating the emotion code after the first layer audio code and the second layer audio code. or, A preset first code is added at the beginning of the audio encoding and a preset second code is added at the end of the audio encoding; wherein, the first code is used to indicate the start of prediction by the speech emotion recognition model to be trained; and the second code is used to indicate the end of audio prediction by the speech emotion recognition model to be trained. The emotion code is set after the second code to construct the target code.

2. The method according to claim 1, characterized in that, Based on the audio encoding and the emotion encoding, the target encoding is determined, including: The audio code and the emotion code are concatenated according to a preset concatenation order to obtain the target code; wherein, the preset concatenation order includes concatenating the emotion code before and / or after the audio code.

3. The method according to claim 1, characterized in that, The speech emotion recognition model to be trained is an autoregressive language model. The step of training the pre-determined speech emotion recognition model to be trained based on the target encoding to obtain the updated target speech emotion recognition model includes: Determine the initial parameters of the speech emotion recognition model to be trained; The target code is input into the speech emotion recognition model to be trained, and the predicted code is output, wherein the predicted code includes a predicted speech code and a predicted emotion code; Based on the predicted encoding and the target encoding, the loss function value is determined; The initial parameters of the speech emotion recognition model to be trained are adjusted based on the loss function value to obtain the updated target speech emotion recognition model.

4. The method according to claim 3, characterized in that, The step of determining the loss function value based on the predicted encoding and the target encoding includes: Determine the weights of the audio encoding and the emotion encoding; Based on the weights of the audio encoding and the emotion encoding, the loss function values ​​of the audio encoding and the emotion encoding are weighted and combined to determine the loss function value.

5. The method according to claim 1, characterized in that, The speech data includes speech data of at least one sentence of contextual information and speech data for predicting emotion. The step of determining the target code based on the audio encoding and the emotion encoding includes: The audio codes corresponding to the speech data of the context and / or background information, the audio codes corresponding to the speech data for predicting emotion, and the emotion codes are all concatenated as fitting data to determine the target code; or, The audio codes corresponding to the speech data of the preceding and / or following information are used as prompt data. The audio codes corresponding to the speech data of the predicted emotion and the emotion codes are concatenated as fitting data to determine the target code.

6. The method according to claim 5, characterized in that, The speech data containing at least one sentence of context information includes speech data of at least two characters, wherein the speech data of different characters are segmented by a third encoding.

7. The method according to any one of claims 1-6, characterized in that, The application process of the target speech emotion recognition model includes: Acquire target speech data; The target speech data is encoded and then input into the target speech emotion recognition model, which outputs the target emotion code. Based on the target emotion encoding, the target emotion label corresponding to the target speech data is determined.

8. A device for constructing a speech emotion recognition model, characterized in that, include: The voice data acquisition module is used to acquire voice data; The encoding module is used to encode the audio features of the speech data to obtain audio encoding; And the emotion tags corresponding to the voice data are encoded to obtain emotion codes; The target encoding determination module is used to determine the target encoding based on the audio encoding and the emotion encoding; The target speech emotion recognition model acquisition module is used to train a pre-determined speech emotion recognition model to be trained based on the target encoding, so as to obtain a trained and updated target speech emotion recognition model; wherein, the target speech emotion recognition model is an autoregressive language model; The target encoding determination module is used for: The audio encoding is processed in layers to obtain a first layer of audio encoding and a second layer of audio encoding; wherein the granularity of the first layer of audio encoding is larger than that of the second layer of audio encoding, or the number of first layer audio encodings is smaller than that of the second layer of audio encoding. The first layer audio code and the second layer audio code are concatenated with the emotion code according to a preset concatenation order to obtain the target code; wherein, the preset concatenation order includes concatenating the emotion code before the first layer audio code and the second layer audio code; or concatenating the emotion code after the first layer audio code and the second layer audio code. or, A preset first code is added at the beginning of the audio encoding and a preset second code is added at the end of the audio encoding; wherein, the first code is used to indicate the start of prediction by the speech emotion recognition model to be trained; and the second code is used to indicate the end of audio prediction by the speech emotion recognition model to be trained. The emotion code is set after the second code to construct the target code.

Images