Voice emotion conversion method, voice emotion conversion device, equipment and storage medium

Abstract

The embodiments of the present application provide a method for speech emotion conversion, a speech emotion conversion device, an apparatus and a storage medium, and belong to the field of artificial intelligence technology. The method includes: obtaining initial spectrum data and initial emotion labels of initial emotion audio; encoding the initial spectrum data according to an audio code book model to obtain first encoded data; performing pitch extraction on the initial spectrum data according to a pitch extraction algorithm to obtain target pitch information; obtaining a target emotion label, and encoding and synthesizing the first encoded data, the target pitch information, the initial emotion label and the target emotion label to obtain second encoded data; inputting the second encoded data into a pre-trained translation model for encoding and translation to obtain target code book sequence information; generating target spectrum data according to the target code book sequence information; performing voice encoding restoration on the target spectrum data to obtain target emotion audio. The embodiments of the present application can improve the efficiency and accuracy of speech emotion conversion.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method for converting speech emotions, a speech emotion conversion device, equipment, and a storage medium. Background Art

[0002] With the rise of the Internet and the entertainment industry, people have an increasing demand for audio and video editing. For example, converting one person's voice into another person's voice, or adding various sound effects to enhance the function of speech, etc. At present, in order to generate data with emotions, the existing speech emotion conversion method is to generate speech with different emotions by training the corresponding text generation speech model. However, this method has the following disadvantages: (1) It is necessary to collect a large amount of data with different emotions for the sample objects and train them separately for these data, which reduces the training efficiency; (2) It is necessary to use text as an intermediate medium to convert speech, which increases the number of models required, resulting in a loss of accuracy and reducing the conversion accuracy of the model. Therefore, how to improve the efficiency and accuracy of speech emotion conversion has become a technical problem that needs to be solved urgently. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a voice emotion conversion method, voice emotion conversion device, equipment and storage medium, which can improve the efficiency and accuracy of voice emotion conversion.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application provides a method for converting speech emotions, the method comprising:

[0005] Obtain initial spectrum data and initial emotion labels of the initial emotion audio;

[0006] Encoding the initial spectrum data according to a pre-trained audio codebook model to obtain first encoded data;

[0007] Performing pitch extraction on each spectrum frame data in the initial spectrum data according to a preset pitch extraction algorithm to obtain target pitch information;

[0008] Obtaining a target emotion label, and performing encoding synthesis based on the first coded data, the target pitch information, the initial emotion label, and the target emotion label to obtain second coded data;

[0009] Inputting the second encoded data into a pre-trained translation model for encoding translation to obtain target codebook sequence information;

[0010] generating target spectrum data according to the target codebook sequence information;

[0011] The target spectrum data is vocoded and restored to obtain target emotional audio.

[0012] In some embodiments, encoding the initial spectrum data according to a pre-trained audio codebook model to obtain first encoded data includes:

[0013] Encoding the initial spectrum data to obtain audio encoding data;

[0014] The audio coding data is quantized and queried according to a pre-trained audio codebook model to obtain first coding data.

[0015] In some embodiments, the pre-trained audio codebook model is trained by the following method:

[0016] Constructing an audio sample set, wherein the audio sample set includes a plurality of original sample spectrum data;

[0017] Constructing an initial codebook model;

[0018] Encoding each of the original sample spectrum data by an encoder to obtain first sample encoded data;

[0019] Inputting the first sample coded data into the initial codebook model for quantization query to obtain second sample coded data;

[0020] Decoding the second sample coded data by a decoder to obtain target sample spectrum data;

[0021] Determine a vector quantization loss value according to the original sample spectrum data, the second sample encoding data and the target sample spectrum data;

[0022] Determine a discriminator loss value according to the original sample spectrum data and the target sample spectrum data;

[0023] Determine a total codebook loss value according to the vector quantization loss value and the discriminator loss value;

[0024] The audio parameters of the initial codebook model are adjusted according to the original sample spectrum data and the target sample spectrum data, and the adjusted initial codebook model is continuously trained based on the audio sample set until the total codebook loss value meets a preset training end condition, so as to obtain the audio codebook model.

[0025] In some embodiments, the first coded data includes coded frame data corresponding to a plurality of the initial spectral data, and the obtaining of the target emotion label and encoding synthesis based on the first coded data, the target pitch information, the initial emotion label, and the target emotion label to obtain the second coded data includes:

[0026] Obtaining the target pitch information corresponding to each spectrum frame data in the initial spectrum data, wherein each coded frame data in the first coded data is obtained by encoding each of the spectrum frame data, and there is a corresponding relationship between the spectrum frame data, the coded frame data, and the target pitch information;

[0027] Inserting the target pitch information corresponding to the encoded frame data into the first encoded data frame by frame to obtain third encoded data;

[0028] A target emotion label is obtained, and encoding and synthesizing is performed based on the third encoded data, the initial emotion label, and the target emotion label to obtain second encoded data.

[0029] In some embodiments, after obtaining the target emotion label and performing encoding synthesis based on the first encoded data, the target pitch information, the initial emotion label, and the target emotion label to obtain the second encoded data, the method further includes:

[0030] Adding an encoding start flag at the beginning of the second encoded data, wherein the encoding start flag is used to indicate the start of processing the second encoded data;

[0031] An encoding end flag is added to the end portion of the second encoded data, where the encoding end flag is used to indicate that processing of the second encoded data is completed.

[0032] In some embodiments, the pre-trained translation model is trained by the following method:

[0033] Constructing an emotional speech conversion sample set, the emotional speech conversion sample set including a plurality of training samples, each training sample including a first emotional speech sample and a second emotional speech sample, the first emotional speech sample including first emotional spectrum data and a first emotional label, the second emotional speech sample including second emotional spectrum data and a second emotional label, the first emotional spectrum data and the second emotional spectrum data corresponding to the same speech content;

[0034] Encoding the first emotion spectrum data and the second emotion spectrum data respectively by an encoder to obtain first emotion coding data corresponding to the first emotion spectrum data and second emotion coding data corresponding to the second emotion spectrum data;

[0035] Performing pitch extraction on each sample spectrum frame data in the first emotion spectrum data and the second emotion spectrum data according to the pitch extraction algorithm to obtain first sample pitch information corresponding to the first emotion spectrum data and second sample pitch information corresponding to the second emotion spectrum data;

[0036] Inserting the first sample pitch information into the first emotion coding data frame by frame to obtain third emotion coding data;

[0037] Perform encoding synthesis based on the first emotion label, the second emotion label and the third emotion coding data to obtain fourth emotion coding data;

[0038] inserting the second sample pitch information into the second emotion coded data frame by frame to obtain fifth emotion coded data;

[0039] Constructing an initial translation model, and inputting the fourth emotion coding data into the initial translation model for model training to obtain target emotion coding data;

[0040] determining a loss value according to the fifth emotion encoding data and the target emotion encoding data;

[0041] determining a voiceprint loss value according to the target emotion coding data and the fourth emotion coding data;

[0042] The training parameters of the initial translation model are adjusted according to the fifth emotion coding data and the target emotion coding data, and the adjusted model is continued to be trained based on the emotional speech conversion sample set until the loss value and the voiceprint loss value meet the preset training end conditions to obtain the translation model.

[0043] In some embodiments, generating target spectrum data according to the target codebook sequence information includes:

[0044] removing pitch information from the target codebook sequence information to obtain codebook sequence information to be processed;

[0045] Target spectrum data is generated according to the codebook sequence information to be processed.

[0046] A second aspect of the embodiments of the present application provides a device for converting speech emotions, the device comprising:

[0047] A spectrum data acquisition module is used to obtain initial spectrum data and initial emotion labels of the initial emotion audio;

[0048] a spectrum encoding module, configured to encode the initial spectrum data according to a pre-trained audio codebook model to obtain first encoded data;

[0049] A pitch extraction module is used to extract the pitch of each spectrum frame data in the initial spectrum data according to a preset pitch extraction algorithm to obtain target pitch information;

[0050] A coding synthesis module, configured to obtain a target emotion label, and perform coding synthesis based on the first coded data, the target pitch information, the initial emotion label, and the target emotion label to obtain second coded data;

[0051] a code translation module, configured to input the second coded data into a pre-trained translation model for code translation to obtain target codebook sequence information;

[0052] a spectrum data generating module, configured to generate target spectrum data according to the target codebook sequence information;

[0053] The vocoding restoration module is used to perform vocoding restoration on the target spectrum data to obtain target emotional audio.

[0054] A third aspect of an embodiment of the present application provides a computer device, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor is used to execute a method as described in any one of the embodiments of the first aspect of the present application.

[0055] The fourth aspect of the embodiments of the present application proposes a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a computer, the computer is used to execute the method described in any one of the embodiments of the first aspect of the present application.

[0056] The speech emotion conversion method, speech emotion conversion device, equipment and storage medium proposed in the embodiments of the present application obtain the initial spectrum data and initial emotion label of the initial emotion audio, encode the initial spectrum data according to the pre-trained audio code book model, and obtain the first encoded data. In order to make the generated target emotion audio smoother and more accurate, the pitch of each spectrum frame data in the initial spectrum data is extracted according to the preset pitch extraction algorithm to obtain the target pitch information. In order to achieve the emotion conversion of the initial emotion audio, the target emotion label is obtained, and the encoding and synthesis are performed according to the first encoding data, the target pitch information, the initial emotion label and the target emotion label to obtain the second encoded data. The second encoded data is input into the pre-trained translation model for encoding translation to obtain the target code book sequence information. Finally, the target spectrum data is generated according to the target code book sequence information, and the target spectrum data is restored by voice coding to obtain the target emotion audio. The embodiments of the present application improve the efficiency and accuracy of speech emotion conversion by combining the audio code book model and the pitch extraction algorithm to perform speech emotion conversion on the initial emotion audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a first flow chart of the method for converting speech emotions provided by an embodiment of the present application;

[0058] Figure 2 yes Figure 1 Flowchart of step S120 in FIG.

[0059] Figure 3 is a flowchart of the training process of the audio codebook model provided in an embodiment of the present application;

[0060] Figure 4 is a structural diagram of an audio codebook model provided in an embodiment of the present application;

[0061] Figure 5 yes Figure 1 Flowchart of step S140 in FIG.

[0062] Figure 6 This is a second flow chart of the method for converting speech emotions provided by an embodiment of the present application;

[0063] Figure 7 This is a flowchart of the training process of the translation model provided in the embodiment of the present application;

[0064] Figure 8 yes Figure 1 Flowchart of step S160 in FIG.

[0065] Figure 9 This is a block diagram of the module structure of the speech emotion conversion device provided in an embodiment of the present application;

[0066] Figure 10 This is a schematic diagram of the hardware structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0068] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0070] In addition, the described features, structures or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, well-known methods, devices, implementations or operations are not shown or described in detail to avoid blurring various aspects of the present disclosure.

[0071] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0072] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0073] First, let’s analyze some of the terms used in this application:

[0074] Artificial Intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and create new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0075] Mel Scale: Human frequency perception is non-linear, and we are more sensitive to low-frequency signals than high-frequency ones. For example, we can easily detect the difference between 500 and 1000 Hz, but find it difficult to detect the difference between 7500 and 8000 Hz. Mel Scale is a nonlinear transformation of Hz. For signals measured in Mel Scale units, the perception of signals with the same frequency difference can be almost the same.

[0076] Mel spectrum: refers to the spectrum at the Mel scale, which is obtained by multiplying the spectrum by several Mel filters.

[0077] The Transformer model also uses an encoder-decoder structure. This model is widely used in the field of natural language processing (NLP), such as machine translation, question-answering systems, text summarization, and speech recognition. The Transformer model, based on the self-attention mechanism, is currently the most advanced neural network architecture. It consists of an encoder and a decoder. The encoder extracts feature information from the text, generating a feature vector for each word in the text, thereby obtaining a feature vector for the entire text. The decoder uses the feature vectors extracted by the encoder to generate keywords that match the feature information as output.

[0078] Encoder-Decoder: It is a common model framework in deep learning. Many common applications are designed using the encoder-decoder framework. The encoder and decoder parts can be any text, voice, image, and video data. Various models can be designed based on the encoder-decoder framework.

[0079] Generative Pre-Training-2 (GPT2 model): This model is composed of the decoder part of a multi-layer unidirectional Transformer. It is essentially an autoregressive model. Autoregression means that each time a new word is generated, it is added to the original input sentence as a new input sentence.

[0080] With the rise of the Internet and the entertainment industry, people have an increasing demand for audio and video editing. For example, converting one person's voice into another person's voice, or adding various sound effects to enhance the function of speech, etc. At present, in order to generate data with emotions, the existing speech emotion conversion method is to generate speech with different emotions by training the corresponding text generation speech model. However, this method has the following disadvantages: (1) It is necessary to collect a large amount of data with different emotions for the sample objects and train them separately for these data, which reduces the training efficiency; (2) It is necessary to use text as an intermediate medium to convert speech, which increases the number of models required, resulting in a loss of accuracy and reducing the conversion accuracy of the model. Therefore, how to improve the efficiency and accuracy of speech emotion conversion has become a technical problem that needs to be solved urgently.

[0081] Based on this, the embodiments of the present application propose a voice emotion conversion method, a voice emotion conversion device, a device and a storage medium, which can improve the efficiency and accuracy of voice emotion conversion.

[0082] The embodiments of this application can acquire and process relevant data based on artificial intelligence technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operating / interactive systems, and mechatronics. Artificial intelligence software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0083] The speech emotion conversion method provided in the embodiment of the present application relates to the field of artificial intelligence. The speech emotion conversion method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer or a smart watch, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; the software can be an application that implements the speech emotion conversion method, etc., but is not limited to the above forms.

[0084] Embodiments of the present application can be used in numerous general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0085] Please refer to Figure 1 , Figure 1This is an optional flowchart of the method for converting speech emotions provided in the embodiment of the present application. Figure 1 The method may specifically include but is not limited to steps S110 to S170.

[0086] Step S110, obtaining initial spectrum data and initial emotion labels of the initial emotion audio;

[0087] Step S120, encoding the initial spectrum data according to the pre-trained audio codebook model to obtain first encoded data;

[0088] Step S130, performing pitch extraction on each spectrum frame data in the initial spectrum data according to a preset pitch extraction algorithm to obtain target pitch information;

[0089] Step S140, obtaining a target emotion label, and performing encoding synthesis based on the first coded data, the target pitch information, the initial emotion label, and the target emotion label to obtain second coded data;

[0090] Step S150: input the second coded data into a pre-trained translation model for coding translation to obtain target codebook sequence information;

[0091] Step S160, generating target spectrum data according to the target codebook sequence information;

[0092] Step S170 , performing voice encoding restoration on the target spectrum data to obtain the target emotional audio.

[0093] In step S110 of some embodiments, in order to generate audio data with emotion, first, the initial spectrum data and initial emotion label of the initial emotion audio are obtained. Specifically, the initial emotion audio is used to represent the audio data corresponding to emotion A before the speech emotion conversion, that is, the initial spectrum data and initial emotion label corresponding to emotion A are obtained. According to the definition of emotion labels in the data set used for model training, the initial emotion label may include but is not limited to any one of: "Neutral", "Happy", "Angry", "Sad", and "Surprise".

[0094] It should be noted that in order to improve the perception accuracy of the frequency changes of the audio in this application, so as to more effectively perceive the amplitude information and frequency domain information related to the initial emotional audio, the initial spectrum data obtained can adopt Mel spectrum data, wherein the Mel spectrum data is equivalent to converting the speech signal of the obtained initial emotional audio into a speech feature that can be trained by the model. Among them, the method of obtaining Mel spectrum data based on audio may include but is not limited to: first, pre-emphasize, frame and window the original audio signal to obtain multiple single-frame audio signals; then, perform short-time Fourier transform on each single-frame audio signal to obtain a short-time amplitude spectrum; finally, pass the obtained short-time amplitude spectrum through a Mel filter group to obtain Mel spectrum data.

[0095] In step S120 of some embodiments, in order to improve the efficiency of the speech emotion conversion method, the initial spectrum data is encoded according to the pre-trained audio code book model to obtain first encoded data. That is, the embodiment of the present application uses the audio code book model to realize speech emotion conversion on low-dimensional coding, which solves the low efficiency problem of spectrum graph conversion used in the existing technology.

[0096] It should be noted that the present application encodes the initial spectrum data according to the encoder E(·) of the pre-trained audio codebook model to obtain the first encoded data. Among them, the encoder E(·) can be implemented using a symmetrical multi-layer 2D convolution structure, and its specific network model is not specifically limited. In addition, in order to make the data features extracted by the encoder more effective and reduce the computational complexity of the encoding process, each encoder layer of the encoder E(·) can be set to a value with a stride greater than 1 for dimensionality reduction, thereby obtaining lower-dimensional encoded data. Similarly, the corresponding decoder can restore the reduced-dimensional data by data interpolation and amplification.

[0097] In step S130 of some embodiments, in order to make the generated audio of the target emotion smoother and thus control the rhythm of the generated target emotion audio, pitch extraction is performed on each spectrum frame data in the initial spectrum data according to a preset pitch extraction algorithm to obtain target pitch information. Specifically, since the pitch extraction algorithm performs pitch extraction on each frame data in the spectrum graph, all spectrum frame data in the initial spectrum data are obtained, and pitch extraction is performed on each spectrum frame data in the initial spectrum data according to the preset pitch extraction algorithm to obtain target pitch information. Then, based on the target pitch information, the changes in each spectrum frame data can be more effectively identified, thereby better constraining the pitch of the generated target emotion audio and improving the accuracy of speech emotion conversion.

[0098] It should be noted that, in order to improve the ability of the target pitch information to control the pitch of the generated target emotional audio, the embodiment of the present application can adopt a method of expanding the codebook to add more tokens to represent the target pitch information. That is, the codebook can be divided into 256 pitch levels through Mel scale quantization to serve as the pitch reference of the Transformer model corresponding to the pitch extraction algorithm. Specifically, the Mel scale can be converted using the following formula (1).

[0099]

[0100] Among them, Mel(·) represents the Mel function, F m (·) represents the Mel-scale conversion formula, f represents the assumed audio frequency, sr represents the audio sampling rate, nmels represents the number of pitch levels 256, Floor(·) represents the floor function, and ln(·) represents the logarithmic function with the constant e as the base.

[0101] In step S140 of some embodiments, in order to achieve speech emotion conversion of the initial emotion audio corresponding to emotion A to obtain the target emotion audio corresponding to emotion B, the target emotion label corresponding to emotion B is obtained, and encoding and synthesis are performed based on the first encoded data, the target pitch information, the initial emotion label, and the target emotion label to obtain second encoded data. Since the obtained first encoded data, the target pitch information, the initial emotion label, and the target emotion label are all equivalent to serialized encoded data, the second encoded data is obtained through encoding and synthesis.

[0102] In step S150 of some embodiments, the second encoded data is input into a pre-trained translation model for encoding translation to obtain target codebook sequence information.

[0103] It should be noted that the pre-trained translation model can adopt a Transform model in the form of an encoder-decoder. For example, the GPT2 model is used as the translation model to encode and translate the second encoded data synthesized by encoding, that is, to obtain the target code book sequence information related to the target emotion label.

[0104] In step S160 of some embodiments, since the obtained target codebook sequence information is a sequence encoded using preset audio codebook numbers, i.e., the numbers therein are all the numbers of vectors in the audio codebook, target spectrum data is generated based on the target codebook sequence information. Specifically, when the output initial spectrum data is mel spectrum data, the decoder G(·) corresponding to the audio codebook model decodes and restores the target codebook sequence information to obtain the target spectrum data, and the target spectrum data is also in the form of mel spectrum.

[0105] In step S170 of some embodiments, in order to accurately output the target emotional audio corresponding to emotion B, the target spectrum data is voiced and restored to obtain the target emotional audio. Specifically, the pre-trained HifiGAN vocoder is used to perform voice coding and restore the target emotional audio to obtain the target emotional audio, the audio content of the target emotional audio is the same as the audio content of the initial emotional audio, and the corresponding emotional label is converted from the initial emotional label to the target emotional label, that is, the speech emotion conversion from emotion A to emotion B is achieved. The embodiment of the present application utilizes the method of Vector Quantized Generative Adversarial Network (VqGAN) to express the initial emotional audio by converting the initial emotional audio into low-dimensional latent variables, and uses the Transformer model to translate the speaker and emotional label into latent variables corresponding to the speaker and emotion, and then decodes the latent variable through a decoder to obtain the audio after emotion conversion.

[0106] Please refer to Figure 2 , Figure 2 Flowchart of the specific method of step S120 in some embodiments of the present application. In some embodiments of the present application, step S120 specifically includes but is not limited to step S210 and step S220. Figure 2 These two steps are introduced in detail.

[0107] Step S210, encoding the initial spectrum data to obtain audio encoding data;

[0108] Step S220 : performing a quantization query on the audio coded data according to the pre-trained audio codebook model to obtain first coded data.

[0109] In step S210 of some embodiments, in order to improve the efficiency of the speech emotion conversion method, the initial spectrum data is encoded to obtain audio coded data. Specifically, when the initial spectrum data adopts Mel spectrum data, the initial spectrum data is encoded by the encoder E(·) to obtain audio coded data. In step S220 of some embodiments, the audio coded data is quantized and queried according to the pre-trained audio codebook model to obtain first coded data. Specifically, the audio codebook Z corresponding to the audio codebook model is pre-trained, and the audio codebook Z uses a finite number of vectors to represent the data corresponding to the coded data. For example, it can be represented by codes such as 1, 4, and 2, or it can be represented by 3, 5, and 2. The audio coded data is passed through the audio codebook Z and quantized and queried by the quantization function to obtain the quantized first coded data.

[0110] Please refer to Figure 3 , Figure 3The flowchart of the training process of the audio codebook model in some embodiments of the present application. In some embodiments of the present application, the training process of the audio codebook model specifically includes but is not limited to steps S310 to S390. Figure 3 These nine steps are introduced in detail.

[0111] Step S310: constructing an audio sample set, where the audio sample set includes a plurality of original sample spectrum data;

[0112] Step S320, constructing an initial codebook model;

[0113] Step S330, encoding each original sample spectrum data by an encoder to obtain first sample encoded data;

[0114] Step S340: input the first sample coded data into the initial codebook model for quantization query to obtain second sample coded data;

[0115] Step S350, decoding the second sample coded data by a decoder to obtain target sample spectrum data;

[0116] Step S360, determining a vector quantization loss value according to the original sample spectrum data, the second sample encoded data, and the target sample spectrum data;

[0117] Step S370, determining a discriminator loss value based on the original sample spectrum data and the target sample spectrum data;

[0118] Step S380, determining a total codebook loss value according to the vector quantization loss value and the discriminator loss value;

[0119] Step S390 , adjusting the audio parameters of the initial codebook model according to the original sample spectrum data and the target sample spectrum data, and continuing to train the adjusted initial codebook model based on the audio sample set until the total codebook loss value meets the preset training end condition, thereby obtaining an audio codebook model.

[0120] In step S310 of some embodiments, first, an audio sample set is constructed, and the audio sample set includes a plurality of original sample spectrum data. It should be noted that the audio sample set used can be a public emotional speech dataset for speech synthesis and speech conversion, and the emotional speech dataset includes a plurality of sample objects, and a plurality of original sample spectrum data of each sample object in different emotional states are collected, wherein the emotional states include neutral, happy, angry, sad, and surprised, and their corresponding emotion labels are "neutral", "happy", "angry", "sad", and "surprise".

[0121] In steps S320 and S330 of some embodiments, in order to train the audio codebook model, an initial codebook model is constructed, and the Mel spectrum data x corresponding to each original sample spectrum data is encoded by the encoder E(·) to obtain the first sample encoded data in, T represents the number of frames of the original sample spectrum data, and F represents the number of Mel spectrum n_mels. The corresponding audio parameters are described in Table 1.

[0122] parameter Numerical Sampling rate 16000 n_mels 80 n_fft 512 hop_length 10ms win_length 25ms fmin 0 fmax 8000

[0123] Table 1

[0124] Where n_fft is the number of Fourier transform data points, hop_length is the number of audio samples between adjacent short-time Fourier transform columns, win_length is the length of the window function, fmin is the minimum frequency, and fmax is the maximum frequency.

[0125] In step S340 of some embodiments, the first sample encoded data Input the audio codebook Z of the initial codebook model and perform quantization query using the quantization function q(·) to obtain the quantized second sample coded data z q Wherein, the definition of the quantization function is shown in formula (2).

[0126]

[0127] Where t represents the time frame, f represents the Mel spectrum, z k Indicates the data being quantified.

[0128] In step S350 of some embodiments, the second sample coded data z is encoded by the decoder G(·). q Decode to obtain the Mel spectrum data corresponding to the target sample spectrum data Thus, the audio codebook model is trained according to the Mel spectrum data x corresponding to the original sample spectrum data.

[0129] In step S360 of some embodiments, in order to improve the restoration capability of the obtained audio codebook model, the present application jointly optimizes the audio codebook model based on the vector quantization loss value and the discriminator loss value. Specifically, according to the original sample spectrum data x, the second sample encoding data z q and target sample spectrum data Determine the vector quantization loss value The vector quantization loss value The specific definition of is shown in the following formula (3).

[0130]

[0131] in, represents the reconstruction error, which is used to train the encoder and decoder, sg(·) represents the gradient block, Represents the VQ loss, which is used to update the embedding space. represents the commitment loss, which is used to constrain the encoder E(·), and β represents the first weight coefficient.

[0132] In step S370 of some embodiments, a discriminator loss value is determined based on the original sample spectrum data and the target sample spectrum data. Specifically, the discriminator loss value corresponding to the GAN of the discriminator D(·) calculation model is calculated. The discriminator loss value The specific definition of is shown in the following formula (4).

[0133]

[0134] It should be noted that the discriminator D(·) can be implemented using symmetrical multi-layer 2D convolution, which is not limited here.

[0135] In step S380 of some embodiments, the total codebook loss value is determined based on the vector quantization loss value and the discriminator loss value. Specifically, in order to better judge the difference of the spectrum, the present application calculates the difference between the two codes by using the Learned Perceptual Audio Patch Similarity (LPAPS). The obtained spectrum loss value The specific calculation process is shown in the following formula (5).

[0136]

[0137] Among them, d i Represents the size of the original sample spectrum data x in the dimension, and the dimension is represented by the time frame t and the Mel spectrum f. Then the total loss value of the codebook is The calculation formula is shown in (6).

[0138]

[0139] In step S390 of some embodiments, audio parameters of the initial codebook model are adjusted based on the original sample spectrum data and the target sample spectrum data, and the adjusted initial codebook model is continuously trained based on the audio sample set until the total codebook loss value satisfies a preset training termination condition, thereby obtaining an audio codebook model. It should be noted that the preset training termination condition may be when the total codebook loss value is less than a preset loss value threshold, or when the recognition accuracy of the initial codebook model is greater than or equal to a preset accuracy threshold. The content within the initial codebook model is continuously adjusted to ensure that the input mel-spectrogram and the decoded mel-spectrogram remain consistent.

[0140] It should be noted that, in order to improve the accuracy of speech emotion conversion, the embodiment of the present application also includes backpropagation of the gradient, and uses the AdamW optimizer to optimize the entire model, where the learning rate is 1e-4.

[0141] In some specific embodiments, such as Figure 4 As shown, the Mel spectrum data x corresponding to the original sample spectrum data 410 is encoded by the encoder E(·) to obtain the first sample encoded data The first sample encoded data Input the audio codebook Z of the initial codebook model and perform quantization query using the quantization function q(·) to obtain the quantized second sample coded data z q Among them, the second sample encoding data z q The numbers 13, 64, 1, etc. in the code represent different quantization codes and have different meanings in the audio codebook Z. The second sample coded data z is encoded by the decoder G(·). q Decode and obtain the Mel spectrum data corresponding to the target sample spectrum data 420 The discriminator loss value of the model calculated based on the discriminator D(·) According to the original sample spectrum data x, the second sample encoding data z q and target sample spectrum data Determined vector quantization loss value And according to the original sample spectrum data x and the target sample spectrum data Determined spectrum loss value The audio codebook model is trained.

[0142] Please refer to Figure 5 , Figure 5 Flowchart of the specific method of step S140 in some embodiments of the present application. In some embodiments of the present application, the first coded data includes coded frame data corresponding to a plurality of initial spectrum data, and step S140 specifically includes but is not limited to steps S510 to S530. Figure 5These three steps are introduced in detail.

[0143] Step S510, obtaining target pitch information corresponding to each spectrum frame data in the initial spectrum data, wherein each coded frame data in the first coded data is obtained by encoding each spectrum frame data, and there is a correspondence between the spectrum frame data, the coded frame data, and the target pitch information;

[0144] Step S520, inserting target pitch information corresponding to the encoded frame data into the first encoded data frame by frame to obtain third encoded data;

[0145] Step S530 , obtaining a target emotion label, and performing encoding synthesis based on the third encoded data, the initial emotion label, and the target emotion label to obtain second encoded data.

[0146] In steps S510 to S530 of some embodiments, in order to achieve speech emotion conversion of the initial emotional audio corresponding to emotion A to obtain the target emotional audio corresponding to emotion B, the target pitch information corresponding to each spectrum frame data in the initial spectrum data is obtained, wherein each coded frame data in the first coded data is obtained by encoding each spectrum frame data, and there is a corresponding relationship between the spectrum frame data, the coded frame data and the target pitch information. Specifically, the first coded data includes coded frame data corresponding to a plurality of initial spectrum data, that is, each frame of the initial spectrum data is encoded to obtain a coded frame data, thereby obtaining the first coded data. The target pitch information corresponding to each coded frame data is inserted frame by frame into the first coded data to obtain the third coded data. Afterwards, the target emotion label is obtained, and coding synthesis is performed based on the third coded data, the initial emotion label and the target emotion label, that is, the second coded data is obtained by coding splicing.

[0147] Please refer to Figure 6 , Figure 6 This is another optional flow chart of the speech emotion conversion method provided by the embodiment of the present application. In some embodiments of the present application, after step S140, the method of the embodiment of the present application further includes but is not limited to steps S610 and S620. Figure 6 These two steps are introduced in detail.

[0148] Step S610: Adding a coding start flag to the beginning of the second coded data, where the coding start flag is used to indicate the start of processing the second coded data.

[0149] Step S620: Add an encoding end flag to the end portion of the second encoded data. The encoding end flag is used to indicate the end of processing the second encoded data.

[0150] In some embodiments, in step S610 and step S620, since the speech speed under different emotions is different, the length of the encoded data obtained from the audio data is also different. Therefore, it is necessary to add a special encoding mark to let the model know the start and end of the encoded data. Specifically, add the encoding start mark at the beginning of the second encoded data. <bos>, encoding start mark <bos>Used to indicate the start of processing the second coded data. Add the coding end mark at the end of the second coded data <eos>, encoding end mark <eos>Used to indicate the completion of processing the second encoded data.

[0151] Please refer to Figure 7 , Figure 7 This is a flowchart of the training process of the translation model provided in the embodiment of the present application. In some embodiments of the present application, the training process of the pre-trained translation model may specifically include but is not limited to steps S710 to S7110. Figure 7 These ten steps are introduced in detail.

[0152] Step S710: constructing an emotional speech conversion sample set, where the emotional speech conversion sample set includes a plurality of training samples, each training sample includes a first emotional speech sample and a second emotional speech sample, the first emotional speech sample includes first emotional spectrum data and a first emotional label, the second emotional speech sample includes second emotional spectrum data and a second emotional label, and the first emotional spectrum data and the second emotional spectrum data correspond to the same speech content;

[0153] Step S720: Encode the first emotion spectrum data and the second emotion spectrum data respectively through an encoder to obtain first emotion encoding data corresponding to the first emotion spectrum data and second emotion encoding data corresponding to the second emotion spectrum data;

[0154] Step S730, performing pitch extraction on each sample spectrum frame data in the first emotion spectrum data and the second emotion spectrum data according to a pitch extraction algorithm, to obtain first sample pitch information corresponding to the first emotion spectrum data and second sample pitch information corresponding to the second emotion spectrum data;

[0155] Step S740, inserting the first sample pitch information into the first emotion coding data frame by frame to obtain third emotion coding data;

[0156] Step S750, performing encoding synthesis based on the first emotion label, the second emotion label and the third emotion encoding data to obtain fourth emotion encoding data;

[0157] Step S760, inserting the second sample pitch information into the second emotion coded data frame by frame to obtain fifth emotion coded data;

[0158] Step S770: constructing an initial translation model, and inputting the fourth emotion coding data into the initial translation model for model training to obtain target emotion coding data;

[0159] Step S780, determining a loss value according to the fifth emotion coding data and the target emotion coding data;

[0160] Step S790, determining a voiceprint loss value according to the target emotion coding data and the fourth emotion coding data;

[0161] Step S7110: Adjust the training parameters of the initial translation model according to the fifth emotion coding data and the target emotion coding data, and continue to train the adjusted model based on the emotional speech conversion sample set until the loss value and the voiceprint loss value meet the preset training end conditions to obtain the translation model.

[0162] In some embodiments, in steps S710 to S760, to achieve multi-emotion speech emotion conversion, a sample set of emotion-to-speech conversion is first constructed. The sample set includes multiple training samples, each of which includes a first emotion speech sample and a second emotion speech sample. The first emotion speech sample includes first emotion spectrum data and a first emotion label, and the second emotion speech sample includes second emotion spectrum data and a second emotion label. The first emotion spectrum data and the second emotion spectrum data correspond to the same speech content, i.e., the first emotion speech sample corresponds to emotion A, and the second emotion speech sample corresponds to emotion B. An encoder is used to encode the first emotion spectrum data and the second emotion spectrum data, respectively, to obtain first emotion encoding data corresponding to the first emotion spectrum data and second emotion encoding data corresponding to the second emotion spectrum data. To smoothen the generated audio of the target emotion and thereby control the rhythm of the generated audio of the target emotion, pitch extraction is performed on each sample spectrum frame in the first emotion spectrum data and the second emotion spectrum data, respectively, using a pitch extraction algorithm. This obtains first sample pitch information corresponding to the first emotion spectrum data and second sample pitch information corresponding to the second emotion spectrum data. The first sample pitch information is then inserted frame by frame into the first emotion encoding data to obtain third emotion encoding data. Then, the first emotion label, the second emotion label and the third emotion coding data are coded and synthesized to obtain the fourth emotion coding data. At the same time, the second sample pitch information is inserted into the second emotion coding data frame by frame to obtain the fifth emotion coding data

[0163] In steps S770 to 7110 of some embodiments, the GPT2 model in codec form is used to construct an initial translation model, and the fourth emotion encoding data is converted to Input into the initial translation model for model training to obtain the target emotion coding data. In order to train the required translation model, according to the fifth emotion coding data The loss value is determined by the target emotion coding data, that is, the fifth emotion coding data is obtained by calculating the cross entropy. The loss value between the target emotion encoding data The target emotion coding data generator generates the corresponding spectrum data. Then the voiceprint model is used to calculate the cosine distance between the vector corresponding to the generated spectrum data and the voiceprint vector corresponding to the input first emotion spectrum data to obtain the voiceprint loss value. As shown in formula (7), by combining the loss value Voiceprint loss value Get the total loss value Where α represents the second weight coefficient. The training parameters of the initial translation model are adjusted, and the adjusted model is trained based on the emotional speech conversion sample set until the total loss value is The preset training end conditions are met to obtain the translation model.

[0164]

[0165] It should be noted that, in order to improve the accuracy of speech emotion conversion, the embodiment of the present application also includes backpropagation of the gradient, and uses the AdamW optimizer to optimize the entire model, where the learning rate is 1e-4.

[0166] Please refer to Figure 8 , Figure 8 This is a specific flow chart of step S160 provided in an embodiment of the present application. In some embodiments of the present application, step S160 may specifically include but is not limited to step S810 and step S820. Figure 8 These two steps are introduced in detail.

[0167] Step S810, removing pitch information from the target codebook sequence information to obtain codebook sequence information to be processed;

[0168] Step S820: Generate target spectrum data according to the codebook sequence information to be processed.

[0169] In steps S810 and S820 of some embodiments, in order to accurately obtain the target spectrum data, the pitch information of the target code book sequence information is removed to obtain the code book sequence information to be processed. The code book sequence information to be processed is decoded by the decoder G(·) to obtain the mel spectrum data corresponding to the target spectrum data. The mel spectrum data is voiced and restored by the HifiGAN vocoder to obtain the target spectrum data. The embodiment of the present application can automatically generate audio data under the target emotion directly from the given audio data combined with the emotion labels before and after the conversion, and perform speech emotion conversion on the initial emotional audio by combining the audio code book model and the pitch extraction algorithm, converting the latent variables under one emotion into the latent variables under another emotion, and finally achieving the effect of arbitrary emotion conversion of the given speech by using the voiceprint and pitch information as control conditions for training, thereby improving the efficiency and accuracy of speech emotion conversion.

[0170] The present application also provides a speech emotion conversion device, please refer to Figure 9 , which can implement the above-mentioned voice emotion conversion method, the device includes: a spectrum data acquisition module 910, a spectrum encoding module 920, a pitch extraction module 930, a code synthesis module 940, a code translation module 950, a spectrum data generation module 960 and a voice code restoration module 970.

[0171] The spectrum data acquisition module 910 is used to obtain the initial spectrum data and initial emotion label of the initial emotion audio;

[0172] a spectrum encoding module 920, configured to encode the initial spectrum data according to a pre-trained audio codebook model to obtain first encoded data;

[0173] The pitch extraction module 930 is used to extract the pitch of each spectrum frame data in the initial spectrum data according to a preset pitch extraction algorithm to obtain target pitch information;

[0174] The coding synthesis module 940 is used to obtain a target emotion label and perform coding synthesis based on the first coded data, the target pitch information, the initial emotion label and the target emotion label to obtain second coded data;

[0175] The code translation module 950 is configured to input the second code data into a pre-trained translation model for code translation to obtain target codebook sequence information;

[0176] The spectrum data generating module 960 is configured to generate target spectrum data according to the target codebook sequence information;

[0177] The vocoding restoration module 970 is used to perform vocoding restoration on the target spectrum data to obtain the target emotional audio.

[0178] The speech emotion conversion device of the embodiment of the present application is used to execute the speech emotion conversion method in the above embodiment. Its specific processing process is the same as the speech emotion conversion method in the above embodiment, and will not be repeated here.

[0179] An embodiment of the present application also provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor is used to execute the voice emotion conversion method in the above embodiment.

[0180] The following combination Figure 10 The hardware structure of the computer device is described in detail. The computer device includes: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040 and a bus 1050.

[0181] The processor 1010 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0182] The memory 1020 can be implemented in the form of ROM (Read Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020 and is called by the processor 1010 to execute the voice emotion conversion method of the embodiments of this application.

[0183] Input / output interface 1030, used to implement information input and output;

[0184] Communication interface 1040, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0185] bus 1050 , which transmits information between various components of the device (e.g., processor 1010 , memory 1020 , input / output interface 1030 , and communication interface 1040 );

[0186] The processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 are connected to each other in communication within the device via a bus 1050 .

[0187] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a computer, the computer is used to execute the voice emotion conversion method of the embodiment of the present application.

[0188] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0189] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0190] It will be understood by those skilled in the art that Figures 1 to 8 The technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figures, or a combination of certain steps, or different steps.

[0191] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0192] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0193] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0194] It should be understood that in this application, "at least one (item)" means one or more, and "more" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or plural.

[0195] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0196] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0197] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0198] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0199] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.< / eos> < / eos> < / bos> < / bos>

Claims

1. A method for converting speech emotion, characterized in that: The method comprises: Obtain initial spectrum data and initial emotion labels of the initial emotion audio; The initial spectrum data is encoded according to a pre-trained audio codebook model to obtain first encoded data; the encoding of the initial spectrum data according to the pre-trained audio codebook model to obtain first encoded data includes: encoding the initial spectrum data to obtain audio encoded data; performing quantization query on the audio encoded data according to the pre-trained audio codebook model to obtain first encoded data; the pre-trained audio codebook model is trained by the following method: constructing an audio sample set, the audio sample set including a plurality of original sample spectrum data; constructing an initial codebook model; encoding each of the original sample spectrum data by an encoder to obtain first sample encoded data; inputting the first sample encoded data into the initial codebook model for The method comprises the steps of: performing a quantization query to obtain second sample coded data; decoding the second sample coded data by a decoder to obtain target sample spectrum data; determining a vector quantization loss value based on the original sample spectrum data, the second sample coded data, and the target sample spectrum data; determining a discriminator loss value based on the original sample spectrum data and the target sample spectrum data; determining a total codebook loss value based on the vector quantization loss value and the discriminator loss value; adjusting audio parameters of the initial codebook model based on the original sample spectrum data and the target sample spectrum data, and continuously training the adjusted initial codebook model based on the audio sample set until the total codebook loss value meets a preset training end condition, thereby obtaining the audio codebook model; Performing pitch extraction on each spectrum frame data in the initial spectrum data according to a preset pitch extraction algorithm to obtain target pitch information; Obtaining a target emotion label, and performing encoding synthesis based on the first coded data, the target pitch information, the initial emotion label, and the target emotion label to obtain second coded data; Inputting the second encoded data into a pre-trained translation model for encoding translation to obtain target codebook sequence information; generating target spectrum data according to the target codebook sequence information; The target spectrum data is vocoded and restored to obtain target emotional audio.

2. The method for converting speech emotion according to claim 1, wherein: The first coded data includes coded frame data corresponding to a plurality of the initial spectral data, and the target emotion label is obtained, and encoding and synthesizing the first coded data, the target pitch information, the initial emotion label, and the target emotion label to obtain the second coded data includes: Obtaining the target pitch information corresponding to each spectrum frame data in the initial spectrum data, wherein each coded frame data in the first coded data is obtained by encoding each of the spectrum frame data, and there is a corresponding relationship between the spectrum frame data, the coded frame data, and the target pitch information; Inserting the target pitch information corresponding to the encoded frame data into the first encoded data frame by frame to obtain third encoded data; A target emotion label is obtained, and encoding and synthesizing is performed based on the third encoded data, the initial emotion label, and the target emotion label to obtain second encoded data.

3. The method for converting speech emotion according to claim 1, wherein: After obtaining the target emotion label and performing encoding synthesis based on the first coded data, the target pitch information, the initial emotion label, and the target emotion label to obtain the second coded data, the speech emotion conversion method further includes: Adding an encoding start flag at the beginning of the second encoded data, wherein the encoding start flag is used to indicate the start of processing the second encoded data; An encoding end flag is added to the end portion of the second encoded data, where the encoding end flag is used to indicate that processing of the second encoded data is completed.

4. The method for converting speech emotion according to any one of claims 1 to 3, characterized in that: The pre-trained translation model is trained by the following method: Constructing an emotional speech conversion sample set, the emotional speech conversion sample set including a plurality of training samples, each training sample including a first emotional speech sample and a second emotional speech sample, the first emotional speech sample including first emotional spectrum data and a first emotional label, the second emotional speech sample including second emotional spectrum data and a second emotional label, the first emotional spectrum data and the second emotional spectrum data corresponding to the same speech content; Encoding the first emotion spectrum data and the second emotion spectrum data respectively by an encoder to obtain first emotion coding data corresponding to the first emotion spectrum data and second emotion coding data corresponding to the second emotion spectrum data; Performing pitch extraction on each sample spectrum frame data in the first emotion spectrum data and the second emotion spectrum data according to the pitch extraction algorithm to obtain first sample pitch information corresponding to the first emotion spectrum data and second sample pitch information corresponding to the second emotion spectrum data; Inserting the first sample pitch information into the first emotion coding data frame by frame to obtain third emotion coding data; Perform encoding synthesis based on the first emotion label, the second emotion label and the third emotion coding data to obtain fourth emotion coding data; inserting the second sample pitch information into the second emotion coded data frame by frame to obtain fifth emotion coded data; Constructing an initial translation model, and inputting the fourth emotion coding data into the initial translation model for model training to obtain target emotion coding data; determining a first loss value according to the fifth emotion encoding data and the target emotion encoding data; determining a voiceprint loss value according to the target emotion coding data and the fourth emotion coding data; The training parameters of the initial translation model are adjusted according to the fifth emotion coding data and the target emotion coding data, and the adjusted model is continued to be trained based on the emotional speech conversion sample set until the first loss value and the voiceprint loss value meet the preset training end conditions to obtain the translation model.

5. The method for converting speech emotion according to any one of claims 1 to 3, characterized in that: Generating target spectrum data according to the target codebook sequence information includes: removing pitch information from the target codebook sequence information to obtain codebook sequence information to be processed; Target spectrum data is generated according to the codebook sequence information to be processed.

6. A voice emotion conversion device, characterized in that: The device comprises: A spectrum data acquisition module is used to obtain initial spectrum data and initial emotion labels of the initial emotion audio; The spectrum encoding module is configured to encode the initial spectrum data according to a pre-trained audio codebook model to obtain first encoded data; the encoding of the initial spectrum data according to the pre-trained audio codebook model to obtain first encoded data includes: encoding the initial spectrum data to obtain audio encoded data; performing a quantization query on the audio encoded data according to the pre-trained audio codebook model to obtain first encoded data; the pre-trained audio codebook model is trained by the following method: constructing an audio sample set, the audio sample set including a plurality of original sample spectrum data; constructing an initial codebook model; encoding each of the original sample spectrum data by an encoder to obtain first sample encoded data; and inputting the first sample encoded data into the initial codebook model. The codebook model is quantized and queried to obtain second sample coded data; the second sample coded data is decoded by a decoder to obtain target sample spectrum data; a vector quantization loss value is determined based on the original sample spectrum data, the second sample coded data, and the target sample spectrum data; a discriminator loss value is determined based on the original sample spectrum data and the target sample spectrum data; a total codebook loss value is determined based on the vector quantization loss value and the discriminator loss value; audio parameters of the initial codebook model are adjusted based on the original sample spectrum data and the target sample spectrum data, and the adjusted initial codebook model is continuously trained based on the audio sample set until the total codebook loss value meets a preset training end condition, thereby obtaining the audio codebook model; A pitch extraction module is used to extract the pitch of each spectrum frame data in the initial spectrum data according to a preset pitch extraction algorithm to obtain target pitch information; A coding synthesis module, configured to obtain a target emotion label, and perform coding synthesis based on the first coded data, the target pitch information, the initial emotion label, and the target emotion label to obtain second coded data; a code translation module, configured to input the second coded data into a pre-trained translation model for code translation to obtain target codebook sequence information; a spectrum data generating module, configured to generate target spectrum data according to the target codebook sequence information; The vocoding restoration module is used to perform vocoding restoration on the target spectrum data to obtain target emotional audio.

7. A computer device, characterized in that: The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor is configured to perform: The method for converting speech emotion according to any one of claims 1 to 5.

8. A storage medium, wherein the storage medium is a computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program. When the computer program is executed by a computer, the computer is configured to: The method for converting speech emotion according to any one of claims 1 to 5.

Images