Audio generation method and device, electronic equipment and storage medium

Abstract

The invention discloses an audio generation method and device, electronic equipment and a storage medium, and belongs to the technical field of electronic equipment.The method comprises the steps that a reference object and laughter feature data related to the reference object are obtained, the reference object comprises a reference text and reference audio, the reference text is a text used for laughter synthesis, and the reference audio is a text used for laughter synthesis; the reference audio is used for indicating to generate the audio of the preset style; determining laughter speech spectrum data according to the reference object and the laughter feature data through a spectrum generation model; the laughter speech spectrum data is converted into laughter synthesis audio, the laughter synthesis audio is audio of a preset style, and the laughter synthesis audio comprises text content of the reference text and laughter related to emotion represented by the text content.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to an audio generation method, device, electronic device and storage medium. Background Art

[0002] Laughter is a sound expression that is full of infectiousness and positive energy. Whether in film and television production, game development, virtual reality experience or online social interaction, controllable laughter synthesis audio generation technology has broad application prospects.

[0003] In the related art, laughter audio clips can be selected from the laughter library for manual editing to obtain laughter synthesis audio. However, the manual editing method is too dependent on manual editing skills and experience, which not only causes large fluctuations in the quality of laughter synthesis audio due to differences in the operation levels of different manual workers, but also increases the number of operations for generating laughter synthesis audio, reducing the efficiency of generating laughter synthesis audio. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an audio generation method, device, electronic device and storage medium, which can reduce the operations of editing laughter synthesized audio and improve the efficiency and quality of generating laughter synthesized audio.

[0005] In a first aspect, an embodiment of the present application provides an audio generation method, comprising:

[0006] Acquire a reference object and laughter feature data related to the reference object, wherein the reference object includes a reference text and a reference audio, wherein the reference text is a text used for laughter synthesis, and the reference audio is an audio used to instruct generation of a preset style;

[0007] Determine the laughter speech spectrum data according to the reference object and the laughter feature data by using the spectrum generation model;

[0008] The laughter speech spectrum data is converted into laughter synthesis audio, wherein the laughter synthesis audio is an audio of a preset style, and the laughter synthesis audio includes text content of a reference text and laughter related to the emotion represented by the text content.

[0009] In a second aspect, an embodiment of the present application provides an audio generation device, including:

[0010] An acquisition module, used for acquiring a reference object and laughter feature data related to the reference object, wherein the reference object includes a reference text and a reference audio, wherein the reference text is a text used for performing laughter synthesis, and the reference audio is a reference audio used for indicating generation of audio of a preset style or type;

[0011] A determination module, used to determine the laughter speech spectrum data according to the reference object and the laughter feature data by using the spectrum generation model;

[0012] A conversion module is used to convert the laughter speech spectrum data into laughter synthesis audio, wherein the laughter synthesis audio is an audio of a preset style or type and includes text content of a reference text and laughter related to emotions represented by the text content.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, and when the program or instruction is executed by the processor, the steps of the audio generation method shown in the first aspect are implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the audio generation method shown in the first aspect are implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor and a display interface, the display interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of the audio generation method shown in the first aspect.

[0016] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the steps of the audio generation method shown in the first aspect.

[0017] In an embodiment of the present application, the laughter voice spectrum data can be determined by a spectrum generation model according to a reference object and laughter feature data related to the reference object, wherein the reference object includes a reference text and a reference audio, the reference text is a text for performing laughter synthesis, and the reference audio is an audio for indicating the generation of a preset style; then, the laughter voice spectrum data is converted into a laughter synthesis audio of a preset style and including the text content of the reference text and the emotion-related laughter represented by the text content. In this way, based on the reference text, the reference audio and the laughter feature data, laughter synthesis audio can be automatically generated, without the user spending a lot of time on manual editing, simplifying the operation of generating laughter synthesis audio, reducing the difficulty of generating laughter synthesis audio, and improving the efficiency and quality of generating laughter synthesis audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of an audio generation method provided for some embodiments of the present application;

[0019] Figure 2 A schematic diagram of an interface of an audio generation method provided in some embodiments of the present application;

[0020] Figure 3 A schematic diagram of the structure of an audio generating device provided in some embodiments of the present application;

[0021] Figure 4 A schematic diagram of the structure of an electronic device provided for some embodiments of the present application;

[0022] Figure 5 A schematic diagram of the hardware structure of an electronic device provided for some embodiments of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0024] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0025] The present application provides an audio generation method, device and electronic device. Figures 1 to 5 , the audio generation method provided in the embodiment of the present application is described in detail through specific embodiments and their application scenarios.

[0026] First, combine Figure 1 An audio generation method provided in an embodiment of the present application is described in detail.

[0027] Figure 1 A flowchart of an audio generation method provided for some embodiments of the present application.

[0028] like Figure 1 As shown, the audio generation method provided in the embodiment of the present application can be applied to electronic devices. Based on this, the audio generation method can include step 110 and step 130, as shown below.

[0029] Step 110, obtaining a reference object and laughter feature data related to the reference object, wherein the reference object includes a reference text and a reference audio, wherein the reference text is a text used for performing laughter synthesis, and the reference audio is an audio used for indicating generation of a preset style;

[0030] Step 120, determining laughter speech spectrum data according to a reference object and laughter feature data by using a spectrum generation model;

[0031] Step 130: convert the laughter speech spectrum data into laughter synthesis audio, wherein the laughter synthesis audio is an audio of a preset style, and the laughter synthesis audio includes the text content of the reference text and laughter related to the emotion represented by the text content.

[0032] In this way, laughter synthesis audio can be automatically generated based on reference text, reference audio and laughter feature data, without the user having to spend a lot of time on manual editing. This simplifies the operation of generating laughter synthesis audio, reduces the difficulty of generating laughter synthesis audio, and improves the efficiency and quality of generating laughter synthesis audio.

[0033] The above steps are described in detail below, as shown below.

[0034] First, referring to step 110 , in some embodiments of the present application, a reference object may be acquired in the following manner, and the audio generation method may further include steps 1401 to 1405 .

[0035] Step 1401: display a laughter synthesis interface, where the laughter synthesis interface includes a first area and a second area.

[0036] Step 1402: Receive a first input from a user on a first area.

[0037] Step 1403: In response to the first input, determine the text input by the user in the first area as reference text.

[0038] Step 1404: Receive a second input from the user on the second area.

[0039] Step 1405 , in response to the second input, determining the preset audio selected by the user in the second area as the reference audio; or determining the user audio newly added by the user in the second area as the reference audio.

[0040] For example, the order of the first input and the second input is not limited in the embodiment of the present application. Here, the user can touch the second area first, and then touch the first area. For example, Figure 2As shown, the electronic device first receives the user's input to the second area and obtains the reference audio uploaded by the user. Then, the electronic device receives the user's input to the first input and obtains the reference text input by the user "Do you remember the last time we went to the zoo? That monkey actually imitated your expression hahahaha".

[0041] Here, it should be noted that the preset style of the reference audio in the embodiment of the present application can be the style represented by the audio content, that is, the emotional style expressed by the melody and harmony of the audio, such as relaxed and cheerful, sad autumn, etc. The preset style can also be the style of the audio type of the audio, that is, rock, folk, electronic, light music.

[0042] In addition, the reference audio in the embodiment of the present application can be associated with an instruction (Prompt) of the reference audio, and the Prompt is used to guide the speech spectrum generation model to generate audio of a preset style corresponding to the reference audio, wherein the Prompt is a key component of human-computer interaction and can effectively control and optimize the response quality and relevance of the model.

[0043] In some other embodiments of the present application, laughter feature data may be obtained in the following two ways.

[0044] In some embodiments, the laughter synthesis interface also includes a third area, and the third area is used to display at least one of the following preset options: a preset laughter audio option, a preset laughter label option. Based on this, before step 110, the audio generation method may also include steps 1406 to 1407.

[0045] Step 1406: Receive a third input from the user on a third area.

[0046] Step 1407 , in response to the third input, determining the laughter feature data corresponding to the target preset option selected by the user in the third area as the laughter feature data associated with the reference object.

[0047] For example, reference may still be made to Figure 2 The user can input into the third area, thereby selecting a target preset option from the happy laughter preset options, arrogant laughter preset options and sarcastic laughter preset options displayed in the third area, so as to determine the laughter feature data corresponding to the target preset option as the laughter feature data related to the reference object.

[0048] In some other embodiments, before step 110 , the audio generation method may further include step 1408 .

[0049] Laughter feature data associated with a reference object is generated according to the first text semantics of the reference text and the second text semantics of the text corresponding to the reference audio.

[0050] Exemplarily, a semantic recognition model can be used to perform semantic recognition on a reference text to obtain the first text semantics. Similarly, a semantic recognition model can be used to perform semantic recognition on the text corresponding to the reference audio to obtain the second text semantics. Then, the first text semantics and the second text semantics are input into the large model to obtain relevant laughter feature data.

[0051] Next, referring to step 120 , in some embodiments of the present application, the spectrum generation model includes a data processing model and a speech spectrum generation model. Based on this, step 120 may specifically include step 1201 and step 1202 .

[0052] Step 1201, through the data processing model, the reference object and the laughter feature data are processed to obtain a spectrum generation object; wherein the spectrum generation object includes the phoneme expansion data corresponding to the reference text, the speech spectrum data of the reference audio and the emotional data of the laughter feature data, the phoneme expansion data includes the data of the assembled phoneme data of the assembled text after the phoneme data frame is expanded, and the assembled text is the text spliced ​​together with the audio text of the reference text and the reference audio.

[0053] Exemplarily, the input of the data processing model may include three inputs: reference text, reference audio, and laughter feature data, wherein the reference audio may be associated with the prompt of the reference audio, for example, the reference audio is "good luck", and the prompt of the reference audio may be an instruction for guiding the speech spectrum generation model to generate an audio of "relaxed and cheerful" corresponding to the reference audio. The laughter feature data is associated with the laughter prompt, for example, the laughter feature data may include a happy laughter corresponding to the happy laughter preset option, in which case the laughter prompt may be used to guide the speech spectrum generation model to generate an instruction for the audio including "happy laughter".

[0054] Based on this, the data processing model will extract the Mel spectrum features of the reference audio, and send the Mel spectrum features to the speech recognition model (Automatic Speech Recognition, ASR) to extract the audio text, and then splice the audio text and the reference text in the time dimension to obtain the assembled text, and convert it into phoneme representation, i.e. phoneme expansion data, through the text front end. In addition, the data processing model will extract the Mel spectrum features corresponding to the reference audio, and send the Mel spectrum features to the Mel encoder for feature extraction and dimensionality reduction to obtain speech spectrum data. In addition, the laughter audio and laughter prompt will be sent to the laughter emotion detection model to extract emotional data such as laughter embedding.

[0055] In some embodiments, the data processing model includes a first data processing model for processing a reference object, the first data processing model includes a text encoding model and a phoneme duration prediction model, and the spectrum generation object includes phoneme expansion data. Based on this, before the above step 1201, the audio generation method may also include steps 2101 and 2102.

[0056] Step 2101, perform speech recognition processing on the audio spectrum data of the reference audio to obtain an audio text.

[0057] Exemplarily, the audio spectrum data of the reference audio may be processed by speech recognition through ASR to obtain an audio text. For example, the audio text may be the lyrics of the reference audio, or the speech text of the lyrics and harmony parts of the reference audio.

[0058] Step 2102, splicing the reference text and the audio text to obtain a compiled text.

[0059] Exemplarily, the audio text may be concatenated before the reference text to improve recognition efficiency.

[0060] Based on this, step 1201 may specifically include steps 12011 to 12013.

[0061] Step 12011, through the text encoding model, the compiled text is converted into phonemes to obtain compiled phoneme data, where the compiled phoneme data includes a phoneme sequence corresponding to a text sentence in the compiled text. Here, the phoneme in the embodiment of the present application refers to the smallest sound unit that can distinguish meaning in human language.

[0062] Exemplarily, the compiled phoneme data may be the smallest unit in the actual pronunciation of each text in the compiled text, such as “ / h / ”, “ / ǎo / ”.

[0063] Step 12012, using a phoneme duration prediction model, calculate the second phoneme duration of each phoneme according to the first phoneme duration of each phoneme in the phoneme sequence, where the second phoneme duration is used to represent the duration that the phoneme is expected to last in the reference audio.

[0064] For example, the second phoneme duration of each phoneme can be predicted based on the first phoneme duration of each phoneme through a trained phoneme duration predictor. If the first phoneme duration of / ǎo / is 0.2s, the phoneme duration prediction model can be used to calculate that the expected duration of / ǎo / in the reference audio may be 0.5s.

[0065] Step 12013, adjusting the first phoneme duration of each phoneme in the assembled phoneme data according to the second phoneme duration to obtain phoneme extension data, wherein the number of data frames of the phoneme extension data is greater than the number of data frames of the assembled phoneme data.

[0066] Exemplarily, since the second phoneme duration of / ǎo / is greater than the first phoneme duration, the first phoneme duration of / ǎo / in the assembled phoneme data can be adjusted according to the second phoneme duration of / ǎo / , that is, the 0.2s duration of / ǎo / in the assembled phoneme data can be extended to 0.5s. In this way, phoneme expansion data will be obtained. At this time, if there is at least one phoneme with a second phoneme duration greater than the first phoneme duration, the number of data frames of the phoneme expansion data obtained by adjusting the phoneme duration is usually greater than the number of data frames of the assembled phoneme data.

[0067] In other embodiments, the data processing model includes a second data processing model for processing a reference object, the second data processing model is determined by a spectrum feature encoder, and the spectrum generation object includes speech spectrum data of the reference audio. Based on this, step 1201 may specifically include step 12014, performing feature extraction on the audio spectrum data of the reference audio through the second data processing model to obtain speech spectrum data, and the data dimension of the speech spectrum data is lower than the data dimension of the audio spectrum data.

[0068] Exemplarily, the audio spectrum data of the reference audio may include pure music spectrum, pure vocal spectrum, and mixed spectrum of music and vocals. Therefore, the second data processing model can be used to extract the spectrum of human body, i.e., pure vocal spectrum, and mixed spectrum of music and vocals from the audio spectrum data of the reference audio as speech spectrum data.

[0069] In some other embodiments, the data processing model includes a third data processing model for processing laughter feature data, the spectrum generation object includes emotional data of the laughter feature data, and the third data processing model is trained by a laughter emotion recognition model. Based on this, step 1201 may specifically include step 12015, performing emotion recognition on the laughter feature data through the third data processing model to obtain emotional data.

[0070] Exemplarily, the third data processing model can extract laughter feature data from laughter, such as pitch, rhythm, intensity, etc. These laughter feature data can reflect a person's emotional state. Then, the extracted laughter feature data is analyzed through machine learning or deep learning algorithms. By analyzing the laughter feature data, the emotional data behind the laughter can be identified, such as happiness, embarrassment, sarcasm, etc. This identification helps to understand a person's emotional state more deeply.

[0071] In this way, when the emotion data is identified through the third data processing model, technical means such as laughter feature data and analysis of laughter feature data are used to improve the accuracy and efficiency of emotion data identification.

[0072] Step 1202, determining laughter speech spectrum data according to the spectrum generation object by using the speech spectrum generation model.

[0073] Among them, step 1202 may specifically include steps 12021 to 12025.

[0074] Step 12021, align each phoneme in the phoneme expansion data with the spectrum features in the speech spectrum data of the reference audio according to the second phoneme duration of each phoneme in the phoneme expansion data, to obtain aligned audio spectrum data, the aligned audio spectrum data including the speech spectrum data and the mask spectrum data of the mask area.

[0075] Exemplarily, based on the second phoneme duration of each phoneme, an alignment operation is performed between the spectrum feature, ie, the Mel feature, in the speech spectrum data of the reference audio and the corresponding text phoneme feature.

[0076] In this way, the consistency between the audio features and the text features in the duration dimension can be ensured.

[0077] Step 12022, reconstruct the audio spectrum data of the masked area through the emotion data and the phoneme expansion data to obtain the reconstructed audio spectrum data of the masked area.

[0078] Exemplarily, the audio spectrum data of the masked area, such as Mel features, are reconstructed through the emotion data and the phoneme expansion data, so that the phoneme features and the Mel features are accurately aligned in the time dimension.

[0079] In this way, the phoneme features and Mel features are guaranteed to remain consistent in the time dimension, thereby improving the accuracy of feature encoding and the generation effect of the model.

[0080] Step 12023: Replace the mask spectrum data in the aligned audio spectrum data with the reconstructed audio spectrum data to obtain expanded mask spectrum data.

[0081] Step 12024, based on the emotion data, determine the position of the laughter feature data in the expanded mask spectrum data.

[0082] For example, since different emotional data will affect the position of the laughter feature data in the extended mask spectrum data, for example, if the emotional data represents happiness, the laughter feature data will be distributed in the front and back positions of the extended mask spectrum data; if the emotional data represents ridicule, the laughter feature data will be distributed in a relatively middle position in the extended mask spectrum data.

[0083] Step 12025, according to the position, the laughter feature data and the expanded mask spectrum data are fused to obtain the laughter speech spectrum data.

[0084] Exemplarily, the position can be determined according to step 12024, and the laughter feature data and the expanded mask spectrum data can be fused to obtain the laughter speech spectrum data.

[0085] The spectrum generation model mentioned above can be trained through the following steps.

[0086] Based on this, before step 120, the audio generation method may include steps 3101 to 3106, as shown below.

[0087] Step 3101, obtaining a sample reference object and sample laughter feature data, where the sample reference object includes a sample reference audio and a sample reference text.

[0088] Exemplarily, since it is difficult to obtain proprietary sample laughter feature data and sample reference audio, two stages of training are used to deal with the problem of insufficient sample laughter feature data and sample reference audio, namely the pre-training process and the fine-tuning process. For the pre-training process, the Libri-Light dataset is selected, wherein the Libri-Light dataset is a benchmark test platform for unsupervised or limited supervised automatic speech recognition. The dataset contains 60,000 hours of data. The sample reference audio in Libri-Light covers a variety of types such as novels, essays, poems, and literature, fully demonstrating the diverse text and language expression characteristics. This makes the dataset have high naturalness and diversity in speech synthesis tasks. In the fine-tuning training stage, 300 hours of sample laughter feature data were collected on the Internet. Unlike the pre-training dataset, which is mostly natural data of daily conversations and reading aloud, the sample laughter feature data used for fine-tuning includes laughter in the aforementioned various types of sample reference audio related scenarios.

[0089] Since the Libri-Light dataset does not provide the corresponding text information of the audio, a series of data preprocessing steps are required to generate sample reference text that can be used for model training. The specific process is as follows: First, use the pre-trained speech recognition model (Whisper) to transcribe each audio segment to generate the corresponding sample reference text. In order to ensure the accuracy of model reasoning, all audio must first be standardized, such as adjusting to a sampling rate of 16kHz.

[0090] Step 3102: Based on the sample reference object and the sample laughter feature data, the first preprocessing model is trained until the first preset training condition is met, thereby obtaining a second preprocessing model.

[0091] Among them, in some embodiments of the present application, the first preprocessing model includes a first sample preprocessing model for processing a sample reference object, the first sample preprocessing model includes a text encoder and a phoneme duration predictor, the second sample data processing model includes a first data processing model in the data processing model for processing a reference object, the sample reference text is the text of the sample reference audio after speech recognition processing, and the sample reference audio and the sample reference text are time-aligned at the phoneme level. Based on this, step 3102 can specifically include steps 31021 to 31023.

[0092] Step 31021, performing phoneme conversion processing on the sample reference text through a text encoder to obtain sample phoneme data, where the sample phoneme data includes a sample phoneme sequence corresponding to a text sentence in the sample reference text.

[0093] For example, the text encoder can be used to compress high-dimensional sample reference text into a compact vector representation, enabling the model to efficiently process large-scale data without losing key information. It also plays a role in aligning different modal information such as sample reference text and sample reference audio, ensuring that the information between the modalities can be effectively integrated and work together.

[0094] Specifically, a text encoder with a multi-layer deep learning model (Transformer) architecture for processing sequence data adopts a multi-head self-attention mechanism, which can simultaneously focus on different positions in the text and capture global and local semantic information. It can be specifically expressed as shown in the following formula (1):

[0095] Word embedding representation: Each text word w in the text information i Convert to its corresponding word embedding vector e i , which is achieved by embedding the matrix E:

[0096] e i =E(w i ) (1)

[0097] Where E is the pre-trained or randomly initialized embedding matrix, w i is the index of the text word.

[0098] Encoding process: For a text sentence with a sequence length of T in the text information, the text encoder generates a hidden state h t To capture the contextual relationship data in the text. For the Transformer model, it relies on the self-attention mechanism, as shown in formula (2):

[0099] h t =Attention(Q,K,V) (2)

[0100] Output representation: The output generated by the text encoder is sentence-level sample phoneme data, which is represented by H, that is, H = [h1,h2,...,h T ,].

[0101] Step 31022, through the phoneme duration predictor, calculate the second sample phoneme duration of each sample phoneme according to the first sample phoneme duration of each sample phoneme in the sample phoneme sequence, and the second sample phoneme duration is used to represent the duration of the sample phoneme expected to last in the sample reference audio.

[0102] For example, if the first sample phoneme duration of the sample phoneme / ǎo / is 3s, the phoneme duration predictor can calculate the expected duration of the sample phoneme / ǎo / in the sample reference audio, i.e., the second sample phoneme duration, for example, 4s, based on the first sample phoneme duration of the sample phoneme / ǎo / being 3s. Step 31023, when the difference between the second sample phoneme duration and the third sample phoneme duration is less than or equal to the second threshold, the text encoder is determined as the text encoding model in the first data processing model, and the phoneme duration predictor is determined as the phoneme duration prediction model in the first data processing model, and the third sample phoneme duration is the actual duration of the sample phoneme in the sample reference audio.

[0103] Exemplarily, if the second sample phoneme duration of the sample phoneme / ǎo / is, for example, 4S, and the actual duration of the sample phoneme / ǎo / in the sample reference audio, that is, the third sample phoneme duration, is, for example, 5S, then the difference between the two is less than or equal to the second threshold value 3, then the text encoder can be determined as the text encoding model in the first data processing model, and the phoneme duration predictor can be determined as the phoneme duration prediction model in the first data processing model.

[0104] In some other embodiments of the present application, the first preprocessing model includes a second sample preprocessing model for processing a sample reference object, the second sample preprocessing model includes a spectral feature encoder, and the second preprocessing model includes a second data processing model in the data processing model for processing a reference object. Based on this, step 3102 can specifically include steps 31024 to 31025.

[0105] Step 31024, using a spectrum feature encoder, extract features from the sample audio spectrum data of the sample reference audio to obtain sample speech spectrum data, wherein the data dimension of the sample speech spectrum data is lower than the data dimension of the sample audio spectrum data.

[0106] Step 31025, when the proportion of the sample speech spectrum data in the speech spectrum data in the sample audio spectrum data is greater than or equal to the third threshold, the spectrum feature encoder is determined as the second data processing model.

[0107] Exemplarily, the spectral feature encoder, namely the Mel spectrogram encoder, retains the time-frequency information of the speech by extracting the spectral features of the audio signal, so that the model can perceive the frequency distribution and time-varying characteristics of the audio. In order to improve processing efficiency, the encoder usually uses a one-dimensional average pooling layer to downsample the input Mel features multiple times, which can effectively reduce the dimension of the features and save computing resources for subsequent processing. In terms of structural design, the Mel spectrogram encoder usually includes a one-dimensional convolutional layer, layer normalization, activation function, and average pooling layer. It can extract higher-level Mel spectrogram features layer by layer to ensure full capture of time dependency and spectral information.

[0108] In some other embodiments of the present application, the first preprocessing model includes a laughter emotion recognition model for processing sample laughter feature data, and the second preprocessing model includes a third data processing model in the data processing model for processing laughter feature data. Based on this, step 3102 can specifically include steps 31026 to 31028.

[0109] Step 31026, extract sample laughter feature data from the sample audio.

[0110] Step 31027, identify the sample laughter feature data through the laughter emotion recognition model to obtain the first sample emotion data of the sample laughter feature data.

[0111] Step 31028, when the difference value between the first sample emotion data and the second sample emotion data corresponding to the sample laughter feature data is less than or equal to the fourth threshold, the laughter emotion recognition model is determined as the third data processing model.

[0112] Exemplarily, the extraction of emotional data, i.e., laughter latent variables, is a key step in the controllable generation model of laughter, which aims to extract emotional control variables in audio. Specifically, the process uses an open source speech emotion recognition model similar to emotion2vec to extract emotional vectors from audio. After extraction, the length of the emotion vector is adjusted by accessing the linear layer to keep it consistent with other input features of the model. During the training process, in order to ensure the stability of the extracted emotion vector, the weights of the laughter latent variable extraction model are usually frozen, and only the subsequent linear layers are trained. Such a design ensures the effectiveness of the emotion vector in capturing the emotional characteristics of laughter, and can be adaptively adjusted during the generation process, thus becoming the core link in achieving controllable generation of laughter.

[0113] Therefore, the embodiments of the present application can utilize the laughter detection model to effectively extract the emotional latent variables in laughter, and can achieve accurate and effective generation of laughter through fine-tuning training on a proprietary laughter dataset.

[0114] Step 3103, through the second preprocessing model, the reference object and the sample laughter feature data are processed to obtain a sample spectrum generation object, the sample spectrum generation object includes sample phoneme expansion data corresponding to the sample reference text, sample speech spectrum data of the sample reference audio and sample emotion data of the sample laughter feature data, the sample phoneme expansion data includes data of the sample assembly phoneme data of the sample assembly text after the phoneme data frame is expanded, and the sample assembly text is the text spliced ​​together with the sample reference text and the sample audio text of the sample reference audio.

[0115] Step 3104: Generate an object according to the sample spectrum through a sample spectrum decoder to construct first sample laughter speech spectrum data.

[0116] For example, in the embodiment of the present application, the sample spectrum decoder and the sample spectrum discriminator can form a generative adversarial network, where the sample spectrum decoder attempts to generate realistic data to deceive the discriminator, while the sample spectrum discriminator is responsible for distinguishing between real data and generated data. The two constantly confront each other and adjust parameters through back propagation. Finally, the generator generates highly realistic data to achieve tasks such as data synthesis and simulation.

[0117] Specifically, step 3104 may include steps 31041 to 31045.

[0118] Step 31041, extracting the sample audio spectrum data of the sample reference audio through the sample spectrum decoder.

[0119] Step 31042: perform mask processing on the sample audio spectrum data to obtain sample mask spectrum data, where the sample mask spectrum data includes sample mask spectrum data of at least one sample mask area.

[0120] Exemplarily, during the training phase, random masking is performed on the Mel spectrogram to enhance the robustness and generalization ability of the model. The specific operation is to randomly select a starting position on the time axis of the Mel spectrogram, and then mask a period of time after the starting position. The length of the mask is randomly determined, that is, random within a certain range, and a maximum mask length is set to prevent masking too much information. Finally, according to the generated mask position and size, the value of the corresponding area in the Mel spectrogram is set to zero.

[0121] The specific operation is as follows: First, given the input Mel sequence X = {x1, x2, ..., x T}, where T represents the number of frames representing Mel, and each element x i Represents the Mel spectrum. Generate a mask length L by setting the mask length in the range [L min ,L max], the length L of the random mask can be expressed as: L = Random (L min ,L max ), generate the mask starting position s and the mask area I, s = Random (0, TL), I = [s, s + L). Mask operation, mask some frames of the input sequence, and the masked sequence X is expressed by the following formula (3):

[0122]

[0123] Step 31043, according to the second sample phoneme duration of each sample phoneme in the sample phoneme expansion data, align each sample phoneme in the sample phoneme expansion data with the sample spectrum feature in the sample speech spectrum data of the sample reference audio to obtain sample aligned audio spectrum data, the sample aligned audio spectrum data including the sample speech spectrum data and the sample mask spectrum data of the sample mask area.

[0124] Exemplarily, a tool for text-to-speech alignment (Montreal Forced Aligner, MFA) is used to perform phoneme-level timing alignment on the sample spectral features of the sample reference audio and the sample reference phonemes. The MFA tool uses an acoustic model to annotate each phoneme in the audio signal of the sample spectral features, and generates the start and end time of each phoneme, thereby obtaining the precise phoneme duration and improving the accuracy of the alignment of the sample phoneme with the sample spectral features.

[0125] Step 31044, reconstruct the audio spectrum data of the sample mask area through the sample emotion data and the sample phoneme expansion data, and obtain the sample reconstructed audio spectrum data of the sample mask area.

[0126] Exemplarily, the phoneme duration predictor and LR phoneme embedding frame expansion. Among them, the function of the phoneme duration predictor is to predict the duration of each phoneme. In the data preprocessing stage of training, the actual duration of each phoneme has been obtained through the MFA alignment tool. In the frame expansion stage of phoneme embedding, on the one hand, the duration predictor is trained by using the real phoneme duration, and on the other hand, the phoneme embedding is frame expanded using these real duration information. In terms of model structure, the duration predictor adopts a multi-layer Transformer stacking structure. This design fully extracts contextual information, thereby improving the accuracy of phoneme duration prediction. This method not only enhances the model's ability to predict phoneme duration, but also ensures the naturalness and coherence of the generated speech. In order to prevent the training of the phoneme duration predictor from affecting the training of the entire module, the embodiment of the present application adopts a gradient truncation method to train the module separately. Specifically, the mean square error loss can be used to promote model convergence. The specific formula (4) is as follows:

[0127]

[0128] Among them, d i Indicates the duration result predicted by the i-th phoneme, It represents the actual duration corresponding to the ith phoneme. Assuming there are three phoneme duration samples, the actual value y = [y1, y2, y3,], and the predicted value Then its loss can be expressed as:

[0129] Step 31045, replace the sample mask spectrum data in the sample aligned audio spectrum data with the sample reconstructed audio spectrum data to obtain sample expanded mask spectrum data.

[0130] Step 31046, based on the sample emotion data, determine the sample position of the sample laughter feature data in the sample expanded mask spectrum data.

[0131] Step 31047, according to the sample position, the sample laughter feature data and the sample expanded mask spectrum data are fused to obtain the first sample laughter speech spectrum data.

[0132] Exemplarily, after receiving the sample text information, sample audio information and sample laughter feature data, the Mel spectrogram decoder in the embodiment of the present application will fuse these three kinds of information to obtain the first sample laughter speech spectrum data. Based on the fused information, the Mel generator starts the feature extraction and conversion process. It uses a series of neural networks and mathematical transformations to convert the input encoded information into a spectral representation on the Mel frequency scale. In terms of model structure, unlike other models using the Transformer architecture, a more efficient ConvNeXt V2 architecture is adopted. The ConvNeXt V2 architecture significantly improves the performance of pure convolutional networks on various recognition benchmarks by using a fully convolutional masked autoencoder framework (FCMAE) and a new global response normalization (GRN) layer. For the Mel decoder, the embodiment of the present application can use the mean squared error (MSE) loss to guide model optimization. The specific calculation formula (5) is as follows:

[0133]

[0134] where y i Indicates the generated first sample laughter speech spectrum data, namely the Mel spectrogram, Indicates that the second sample laughter speech spectrum data is the real Mel spectrogram, Indicates the difference between the Mel spectrum generated by the randomly masked part and the real Mel spectrum.

[0135] Therefore, the embodiment of the present application can use a random masking scheme to randomly mask the input Mel data, thereby creating different training samples and enhancing the diversity of the data. At the same time, by introducing random masks, the model is forced to learn the invariance and robustness characteristics in the data, which helps to improve the generalization ability of the model when facing new data and reduce the risk of overfitting. In addition, by pre-training on large-scale big data and fine-tuning on relatively limited small data laughter data, the difficulty of obtaining laughter data can be effectively overcome without changing the performance of the model.

[0136] Step 3105, determining the difference between the first sample laughter speech spectrum data and the second sample laughter speech spectrum data through a sample spectrum discriminator, wherein the second sample laughter speech spectrum data is reference laughter speech spectrum data corresponding to the sample reference object and the sample laughter feature data.

[0137] Step 3106: when the difference value is less than or equal to the first threshold, determine the second preprocessing model as the data processing model, and determine the sample spectrum decoder as the speech spectrum generation model.

[0138] Exemplarily, the sample spectrum discriminator can be used to judge the difference between the Mel spectrogram generated by the generative model and the Mel spectrogram obtained by real audio conversion. During the training process, an adversarial training method is adopted, and the generative model and the discriminator compete with each other. The generative model attempts to generate a Mel spectrogram that can deceive the discriminator, while the discriminator strives to distinguish between the real and generated Mel spectrograms. This adversarial process prompts both parties to continuously improve their capabilities, and ultimately enables the generative model to generate high-quality Mel spectrograms. Set the real Mel spectrum classification to 1, and the loss function of the discriminator is shown in the following formula (6):

[0139]

[0140] The loss function of the generator is shown in the following formula (7):

[0141]

[0142] Where y represents the first sample laughter speech spectrum data, i.e. the generated Mel spectrogram, The second sample laughter speech spectrum data level real Mel spectrogram, D(m(y)) is the output of the discriminator for the generated Mel spectrum data (or audio), that is, the probability that the discriminator considers it to be real data. After the model training is completed, the Mel discriminator is discarded, and the trained vocoder is used as a spectrum generator to convert the Mel spectrogram into laughter synthesis audio.

[0143] Then, involving step 130, in some embodiments of the present application, the laughter speech spectrum data generated by the Mel encoder can be sent to the vocoder to synthesize the corresponding laughter synthesis audio. Here, a vocoder commonly used in speech synthesis can be selected, such as HiFiGAN, BigVGAN, etc.

[0144] In this way, the audio generation method in the embodiment of the present application can be applied to the scene of generating laughter synthetic audio under interesting scenes or tasks set in virtual games, so that laughter is triggered when the user completes it, or in a virtual reality experience, the appearance of laughter can be controlled according to the atmosphere and plot changes of the scene.

[0145] It should be noted that the audio generation method provided in the embodiments of the present application can be executed by electronic devices such as mobile phones, tablet computers, laptop computers, PDAs, wearable devices, etc. In some embodiments of the present application, the audio generation method provided in the embodiments of the present application is described by taking the electronic device as the execution subject to execute the audio generation method as an example.

[0146] The audio generation method provided in the embodiment of the present application can be executed by an audio generation device. In the embodiment of the present application, an audio generation device executing the audio generation method is taken as an example to illustrate the device of the audio generation method provided in the embodiment of the present application.

[0147] The present application also provides an audio generation device. Figure 3 Provide detailed explanation.

[0148] Figure 3 A schematic diagram of the structure of an audio generating device provided for some embodiments of the present application.

[0149] like Figure 3 As shown, the audio generating device 30 can be applied to an electronic device, and the audio generating device 30 can specifically include:

[0150] An acquisition module 301 is used to acquire a reference object and laughter feature data related to the reference object, wherein the reference object includes a reference text and a reference audio, wherein the reference text is a text used for performing laughter synthesis, and the reference audio is a reference audio used to indicate the generation of an audio of a preset style;

[0151] A determination module 302, for determining laughter speech spectrum data according to a reference object and laughter feature data by using a spectrum generation model;

[0152] The conversion module 303 is used to convert the laughter speech spectrum data into laughter synthesis audio, wherein the laughter synthesis audio is in a preset style and includes the text content of the reference text and the audio of laughter related to the emotion represented by the text content.

[0153] The audio generating device 30 in the embodiment of the present application is described in detail below, as shown below.

[0154] In some embodiments of the present application, the determination module 302 can be specifically used to, when the spectrum generation model includes a data processing model and a speech spectrum generation model, process the reference object and the laughter feature data through the data processing model to obtain a spectrum generation object; wherein the spectrum generation object includes the phoneme expansion data corresponding to the reference text, the speech spectrum data of the reference audio, and the emotion data of the laughter feature data, the phoneme expansion data includes the data of the assembled phoneme data of the assembled text after the phoneme data frame is expanded, and the assembled text is the text obtained by splicing the reference text and the audio text of the reference audio;

[0155] The speech spectrum data of laughter is determined by generating a speech spectrum model according to the spectrum generation object.

[0156] In some embodiments of the present application, the audio generation device 30 may further include a processing module for performing speech recognition processing on the audio spectrum data of the reference audio to obtain an audio text when the data processing model includes a first data processing model for processing a reference object, the first data processing model includes a text encoding model and a phoneme duration prediction model, and the spectrum generation object includes phoneme expansion data;

[0157] The audio generation device 30 may also include a splicing module for splicing the reference text and the audio text to obtain a compiled text;

[0158] The audio generation device 30 may further include a processing module for performing phoneme conversion processing on the compiled text through a text encoding model to obtain compiled phoneme data, wherein the compiled phoneme data includes a phoneme sequence corresponding to the compiled text;

[0159] The audio generation device 30 may further include a calculation module, for calculating a second phoneme duration of each phoneme according to the first phoneme duration of each phoneme in the phoneme sequence through a phoneme duration prediction model, wherein the second phoneme duration is used to indicate the duration of the phoneme expected to last in the reference audio;

[0160] The audio generating device 30 may further include an adjustment module for adjusting the first phoneme duration of each phoneme in the assembled phoneme data according to the second phoneme duration to obtain phoneme extension data, wherein the number of data frames of the phoneme extension data is greater than the number of data frames of the assembled phoneme data.

[0161] In some embodiments of the present application, the audio generating device 30 may further include a processing module for performing feature extraction on the audio spectrum data of the reference audio through the second data processing model to obtain speech spectrum data when the data processing model includes a second data processing model for processing a reference object, the second data processing model is determined by a spectrum feature encoder, and the spectrum generation object includes speech spectrum data of the reference audio. The data dimension of the speech spectrum data is lower than the data dimension of the audio spectrum data.

[0162] In some embodiments of the present application, the determination module 302 may be specifically configured to align each phoneme in the phoneme expansion data with a spectrum feature in the speech spectrum data of the reference audio according to the second phoneme duration of each phoneme in the phoneme expansion data, to obtain aligned audio spectrum data, where the aligned audio spectrum data includes the speech spectrum data and the mask spectrum data of the mask area;

[0163] Reconstruct the audio spectrum data of the masked area through the emotion data and the phoneme expansion data to obtain the reconstructed audio spectrum data of the masked area;

[0164] Replacing the mask spectrum data in the aligned audio spectrum data with the reconstructed audio spectrum data to obtain expanded mask spectrum data;

[0165] Based on the emotion data, determining the position of the laughter feature data in the expanded mask spectrum data;

[0166] According to the position, the laughter feature data and the expanded mask spectrum data are fused to obtain the laughter speech spectrum data.

[0167] In some embodiments of the present application, the acquisition module 301 may also be used to acquire sample reference objects and sample laughter feature data, where the sample reference objects include sample reference audio and sample reference text;

[0168] The audio generation device 30 may further include a training module for training the first preprocessing model based on the sample reference object and the sample laughter feature data until the first preset training condition is met to obtain the second preprocessing model;

[0169] The audio generating device 30 may further include a processing module, which is used to process the reference object and the sample laughter feature data through a second preprocessing model to obtain a sample spectrum generating object, wherein the sample spectrum generating object includes sample phoneme expansion data corresponding to the sample reference text, sample speech spectrum data of the sample reference audio, and sample emotion data of the sample laughter feature data, wherein the sample phoneme expansion data includes data of the sample assembly phoneme data of the sample assembly text after the phoneme data frame is expanded, and the sample assembly text is a text obtained by splicing the sample reference text and the sample audio text of the sample reference audio;

[0170] The audio generation device 30 may further include a construction module for constructing first sample laughter speech spectrum data according to the sample spectrum generation object through a sample spectrum decoder;

[0171] The determination module 302 may also be used to determine, through a sample spectrum discriminator, a difference value between the first sample laughter speech spectrum data and the second sample laughter speech spectrum data, where the second sample laughter speech spectrum data is reference laughter speech spectrum data corresponding to the sample reference object and the sample laughter feature data;

[0172] The determination module 302 may also be configured to, when the difference value is less than or equal to the first threshold, determine the second preprocessing model as the data processing model, and determine the sample spectrum decoder as the speech spectrum generation model.

[0173] In some embodiments of the present application, the audio generating device 30 may further include a training module, which is used to include a first sample preprocessing model for processing a sample reference object in the first preprocessing model, the first sample preprocessing model includes a text encoder and a phoneme duration predictor, the second sample data processing model includes a first data processing model in the data processing model for processing a reference object, the sample reference text is the text of the sample reference audio after speech recognition processing, and the sample reference audio and the sample reference text are aligned in time sequence at the phoneme level. Through the text encoder, the sample reference text is subjected to phoneme conversion processing to obtain sample phoneme data, and the sample phoneme data includes a sample phoneme sequence corresponding to the text sentence in the sample reference text.

[0174] By means of a phoneme duration predictor, a second sample phoneme duration of each sample phoneme is calculated according to the first sample phoneme duration of each sample phoneme in the sample phoneme sequence, wherein the second sample phoneme duration is used to indicate the duration of the sample phoneme expected to last in the sample reference audio;

[0175] When the difference between the second sample phoneme duration and the third sample phoneme duration is less than or equal to the second threshold, the text encoder is determined as the text encoding model in the first data processing model, and the phoneme duration predictor is determined as the phoneme duration prediction model in the first data processing model, and the third sample phoneme duration is the actual duration of the sample phoneme in the sample reference audio.

[0176] In some embodiments of the present application, the audio generating device 30 may further include a training module for extracting sample laughter feature data from the sample audio when the first preprocessing model includes a laughter emotion recognition model for processing sample laughter feature data and the second preprocessing model includes a third data processing model for processing laughter feature data in the data processing model;

[0177] Using a laughter emotion recognition model, the sample laughter feature data is recognized to obtain first sample emotion data of the sample laughter feature data;

[0178] When the difference value between the first sample emotion data and the second sample emotion data corresponding to the sample laughter feature data is less than or equal to the fourth threshold, the laughter emotion recognition model is determined as the third data processing model.

[0179] In some embodiments of the present application, the audio generation device 30 may further include a construction module for extracting sample audio spectrum data of the sample reference audio through a sample spectrum decoder;

[0180] Performing mask processing on the sample audio spectrum data to obtain sample mask spectrum data, wherein the sample mask spectrum data includes sample mask spectrum data of at least one sample mask area;

[0181] According to the second sample phoneme duration of each sample phoneme in the sample phoneme expansion data, align each sample phoneme in the sample phoneme expansion data with the sample spectrum feature in the sample speech spectrum data of the sample reference audio to obtain sample aligned audio spectrum data, the sample aligned audio spectrum data including the sample speech spectrum data and the sample mask spectrum data of the sample mask area;

[0182] Reconstructing the audio spectrum data of the sample mask area through the sample emotion data and the sample phoneme expansion data to obtain the sample reconstructed audio spectrum data of the sample mask area;

[0183] The sample mask spectrum data in the sample aligned audio spectrum data is replaced with the sample reconstructed audio spectrum data to obtain the sample expanded mask spectrum data;

[0184] Based on the sample emotion data, determining a sample position of the sample laughter feature data in the sample expanded mask spectrum data;

[0185] According to the sample position, the sample laughter feature data and the sample expanded mask spectrum data are fused to obtain the first sample laughter speech spectrum data.

[0186] In some embodiments of the present application, the audio generating device 30 may further include a display module for displaying a laughter synthesis interface, the laughter synthesis interface including a first area and a second area;

[0187] The audio generating device 30 may further include a receiving module for receiving a first input of a user to the first area;

[0188] The determination module 302 may also be used to, in response to the first input, determine the text input by the user in the first area as the reference text;

[0189] The audio generating device 30 may further include a receiving module for receiving a second input of a user to a second area;

[0190] The determination module 302 may also be configured to, in response to a second input, determine a preset audio selected by a user in the second area as a reference audio; or determine a user audio newly added by the user in the second area as a reference audio.

[0191] In some embodiments of the present application, the audio generating device 30 may further include a receiving module for receiving a third input of the user to the third area when the laughter synthesis interface further includes a third area, and the third area is used to display at least one of the following preset options: a preset laughter audio option and a preset laughter label option;

[0192] The determination module 302 may also be configured to, in response to a third input, determine the laughter feature data corresponding to the target preset option selected by the user in the third area as the laughter feature data associated with the reference object.

[0193] In some embodiments of the present application, the audio generating device 30 may further include a generating module for generating laughter feature data related to a reference object according to a first text semantics of a reference text and a second text semantics of a text corresponding to the reference audio.

[0194] The audio generating device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a palmtop computer, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0195] The audio generation device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0196] The audio generation device provided in the embodiment of the present application can achieve Figure 1 to Figure 2 The various processes implemented in the audio generation method embodiment shown achieve the same technical effect and will not be described again here to avoid repetition.

[0197] Based on this, the audio generation device provided in the embodiment of the present application can determine the laughter speech spectrum data according to the reference object and the laughter feature data related to the reference object through the spectrum generation model, wherein the reference object includes a reference text and a reference audio, the reference text is a text used for laughter synthesis, and the reference audio is a reference audio used to indicate the generation of audio of a preset style; then, the laughter speech spectrum data is converted into laughter synthesis audio of a preset style and including the text content of the reference text and the emotion-related laughter represented by the text content. In this way, based on the reference text, the reference audio and the laughter feature data, the laughter synthesis audio can be automatically generated, without the user spending a lot of time on manual editing, simplifying the operation of generating the laughter synthesis audio, reducing the difficulty of generating the laughter synthesis audio, and improving the efficiency and quality of generating the laughter synthesis audio.

[0198] Optional, such as Figure 4 As shown, an embodiment of the present application also provides an electronic device 40, including a processor 401 and a memory 402, wherein the memory 402 stores programs or instructions that can be executed on the processor 401, and when the program or instructions are executed by the processor 401, the various steps of the above-mentioned audio generation method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0199] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0200] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0201] The electronic device 500 includes but is not limited to: a radio frequency unit 501, a network module 502, an audio output unit 503, an input unit 504, a sensor 505, a display unit 506, a user input unit 507, an interface unit 508, a memory 509, a processor 510 and other components.

[0202] Those skilled in the art will appreciate that the electronic device 500 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 510 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system.

[0203] Figure 5 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0204] In the embodiment of the present application, the processor 510 is used to obtain a reference object and laughter feature data related to the reference object, the reference object includes a reference text and a reference audio, the reference text is a text for performing laughter synthesis, and the reference audio is an audio for indicating the generation of a preset style. The processor 510 is also used to determine laughter speech spectrum data according to the reference object and the laughter feature data through a spectrum generation model. The processor 510 is also used to convert the laughter speech spectrum data into laughter synthesis audio, wherein the laughter synthesis audio is an audio of a preset style, and the laughter synthesis audio includes the text content of the reference text and laughter related to the emotion represented by the text content.

[0205] The electronic device 500 is described in detail below, as shown below.

[0206] In some embodiments of the present application, the processor 510 is used to process the reference object and the laughter feature data through the data processing model to obtain a spectrum generation object when the spectrum generation model includes a data processing model and a spectrum generation model; wherein the spectrum generation object includes phoneme expansion data corresponding to the reference text, speech spectrum data of the reference audio, and emotional data of the laughter feature data, the phoneme expansion data includes data of the assembled phoneme data of the assembled text after the phoneme data frame is expanded, and the assembled text is a text obtained by splicing the reference text and the audio text of the reference audio;

[0207] The spectrum generation model is used to generate the object according to the spectrum, and the spectrum data of the laughter speech is determined.

[0208] In some embodiments of the present application, the processor 510 is configured to perform speech recognition processing on the audio spectrum data of the reference audio to obtain an audio text when the data processing model includes a first data processing model for processing a reference object, the first data processing model includes a text encoding model and a phoneme duration prediction model, and the spectrum generation object includes phoneme expansion data;

[0209] splicing the reference text and the audio text to obtain a compilation text;

[0210] Performing phoneme conversion processing on the compiled text through a text encoding model to obtain compiled phoneme data, wherein the compiled phoneme data includes a phoneme sequence corresponding to a text sentence in the compiled text;

[0211] The second phoneme duration of each phoneme in the phoneme sequence is calculated according to the first phoneme duration of each phoneme in the phoneme sequence by using a phoneme duration prediction model, wherein the second phoneme duration is used to indicate the duration of the phoneme that is expected to last in the reference audio;

[0212] According to the second phoneme duration, the first phoneme duration of each phoneme in the assembled phoneme data is adjusted to obtain phoneme extension data, wherein the number of data frames of the phoneme extension data is greater than the number of data frames of the assembled phoneme data.

[0213] In some embodiments of the present application, the processor 510 is used to, when the data processing model includes a second data processing model for processing a reference object, the second data processing model is determined by a spectral feature encoder, and the spectrum generation object includes speech spectrum data of the reference audio, by using the second data processing model, to perform feature extraction on the audio spectrum data of the reference audio to obtain speech spectrum data, and the data dimension of the speech spectrum data is lower than the data dimension of the audio spectrum data.

[0214] In some embodiments of the present application, the processor 510 is configured to align each phoneme in the phoneme expansion data with a spectrum feature in the speech spectrum data of the reference audio according to the second phoneme duration of each phoneme in the phoneme expansion data to obtain aligned audio spectrum data, where the aligned audio spectrum data includes the speech spectrum data and the mask spectrum data of the mask area;

[0215] Reconstruct the audio spectrum data of the masked area through the emotion data and the phoneme expansion data to obtain the reconstructed audio spectrum data of the masked area;

[0216] Replacing the mask spectrum data in the aligned audio spectrum data with the reconstructed audio spectrum data to obtain expanded mask spectrum data;

[0217] Based on the emotion data, determining the position of the laughter feature data in the expanded mask spectrum data;

[0218] According to the position, the laughter feature data and the expanded mask spectrum data are fused to obtain the laughter speech spectrum data.

[0219] In some embodiments of the present application, the processor 510 is used to obtain a sample reference object and sample laughter feature data, where the sample reference object includes a sample reference audio and a sample reference text;

[0220] Based on the sample reference object and the sample laughter feature data, the first preprocessing model is trained until a first preset training condition is met to obtain a second preprocessing model;

[0221] The reference object and the sample laughter feature data are processed by a second preprocessing model to obtain a sample spectrum generation object, wherein the sample spectrum generation object includes sample phoneme expansion data corresponding to the sample reference text, sample speech spectrum data of the sample reference audio, and sample emotion data of the sample laughter feature data, wherein the sample phoneme expansion data includes data of the sample assembly phoneme data of the sample assembly text after the phoneme data frame is expanded, and the sample assembly text is a text obtained by splicing the sample reference text and the sample audio text of the sample reference audio;

[0222] Generate an object according to the sample spectrum through a sample spectrum decoder to construct first sample laughter speech spectrum data;

[0223] Determine the difference between the first sample laughter voice spectrum data and the second sample laughter voice spectrum data by a sample spectrum discriminator, wherein the second sample laughter voice spectrum data is reference laughter voice spectrum data corresponding to the sample reference object and the sample laughter feature data;

[0224] When the difference value is less than or equal to the first threshold, the second pre-processing model is determined as the data processing model, and the sample spectrum decoder is determined as the spectrum generation model.

[0225] In some embodiments of the present application, the processor 510 is used to include a first sample preprocessing model for processing a sample reference object in the first preprocessing model, the first sample preprocessing model includes a text encoder and a phoneme duration predictor, the second sample data processing model includes a first data processing model for processing a reference object in the data processing model, the sample reference text is a text of the sample reference audio after speech recognition processing, and the sample reference audio and the sample reference text are aligned in time sequence at the phoneme level, and the sample reference text is subjected to phoneme conversion processing through the text encoder to obtain sample phoneme data, and the sample phoneme data includes a sample phoneme sequence corresponding to a text sentence in the sample reference text;

[0226] By means of a phoneme duration predictor, a second sample phoneme duration of each sample phoneme is calculated according to the first sample phoneme duration of each sample phoneme in the sample phoneme sequence, wherein the second sample phoneme duration is used to indicate the duration of the sample phoneme expected to last in the sample reference audio;

[0227] When the difference between the second sample phoneme duration and the third sample phoneme duration is less than or equal to the second threshold, the text encoder is determined as the text encoding model in the first data processing model, and the phoneme duration predictor is determined as the phoneme duration prediction model in the first data processing model, and the third sample phoneme duration is the actual duration of the sample phoneme in the sample reference audio.

[0228] In some embodiments of the present application, the processor 510 is configured to extract sample laughter feature data from the sample audio when the first preprocessing model includes a laughter emotion recognition model for processing sample laughter feature data and the second preprocessing model includes a third data processing model for processing laughter feature data in the data processing model;

[0229] Using a laughter emotion recognition model, the sample laughter feature data is recognized to obtain first sample emotion data of the sample laughter feature data;

[0230] When the difference value between the first sample emotion data and the second sample emotion data corresponding to the sample laughter feature data is less than or equal to the fourth threshold, the laughter emotion recognition model is determined as the third data processing model.

[0231] In some embodiments of the present application, the processor 510 is configured to extract sample audio spectrum data of the sample reference audio through a sample spectrum decoder;

[0232] Performing mask processing on the sample audio spectrum data to obtain sample mask spectrum data, wherein the sample mask spectrum data includes sample mask spectrum data of at least one sample mask area;

[0233] According to the second sample phoneme duration of each sample phoneme in the sample phoneme expansion data, align each sample phoneme in the sample phoneme expansion data with the sample spectrum feature in the sample speech spectrum data of the sample reference audio to obtain sample aligned audio spectrum data, the sample aligned audio spectrum data including the sample speech spectrum data and the sample mask spectrum data of the sample mask area;

[0234] Reconstructing the audio spectrum data of the sample mask area through the sample emotion data and the sample phoneme expansion data to obtain the sample reconstructed audio spectrum data of the sample mask area;

[0235] The sample mask spectrum data in the sample aligned audio spectrum data is replaced with the sample reconstructed audio spectrum data to obtain the sample expanded mask spectrum data;

[0236] Based on the sample emotion data, determining a sample position of the sample laughter feature data in the sample expanded mask spectrum data;

[0237] According to the sample position, the sample laughter feature data and the sample expanded mask spectrum data are fused to obtain the first sample laughter speech spectrum data.

[0238] In some embodiments of the present application, the display unit 506 is used to display a laughter synthesis interface, and the laughter synthesis interface includes a first area and a second area;

[0239] A user input unit 507, used to receive a first input of a user to a first area;

[0240] The processor 510 is configured to, in response to a first input, determine the text input by the user in the first area as a reference text;

[0241] A user input unit 507, used to receive a second input of a user to a second area;

[0242] The processor 510 is configured to, in response to the second input, determine the preset audio selected by the user in the second area as the reference audio; or determine the user audio newly added by the user in the second area as the reference audio.

[0243] In some embodiments of the present application, the user input unit 507 is used to receive a third input from the user to the third area when the laughter synthesis interface further includes a third area, and the third area is used to display at least one of the following preset options: a preset laughter audio option and a preset laughter label option;

[0244] The processor 510 may also be configured to, in response to a third input, determine the laughter feature data corresponding to the target preset option selected by the user in the third area as the laughter feature data associated with the reference object.

[0245] In some embodiments of the present application, the processor 510 is configured to generate laughter feature data associated with a reference object according to a first text semantics of a reference text and a second text semantics of a text corresponding to a reference audio.

[0246] It should be understood that the input unit 504 may include a graphics processor (GPU) 5041 and a microphone 5042, and the graphics processor 5041 processes the image data of a static image or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 506 may include a display panel, and the display panel may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 507 includes a touch panel 5071 and at least one of other input devices 5072. The touch panel 5071 is also called a touch screen. The touch panel 5071 may include two parts: a touch detection device and a touch display. Other input devices 5072 may include, but are not limited to, a physical keyboard, function keys (such as a volume display button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0247] The memory 509 can be used to store software programs and various data. The memory 509 can mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area can store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 509 can include a volatile memory or a non-volatile memory, or the memory 509 can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 509 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0248] The processor 510 may include one or more processing units; optionally, the processor 510 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless display signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 510.

[0249] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned audio generation method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0250] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0251] In addition, an embodiment of the present application further provides a chip, which includes a processor and a display interface, the display interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned audio generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0252] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0253] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned audio generation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0254] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0255] In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

[0256] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0257] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An audio generation method, characterized in that: include: Acquire a reference object and laughter feature data related to the reference object, wherein the reference object includes a reference text and a reference audio, wherein the reference text is a text used for performing laughter synthesis, and the reference audio is an audio used for indicating generation of a preset style; Determine the laughter speech spectrum data according to the reference object and the laughter feature data by using a spectrum generation model; The laughter speech spectrum data is converted into laughter synthesis audio, wherein the laughter synthesis audio is the audio of the preset style, and the laughter synthesis audio includes the text content of the reference text and laughter related to the emotion represented by the text content.

2. The method according to claim 1, characterized in that The spectrum generation model includes a data processing model and a speech spectrum generation model; The method of determining the laughter speech spectrum data by using the spectrum generation model according to the reference object and the laughter feature data includes: The reference object and the laughter feature data are processed by the data processing model to obtain a spectrum generation object; wherein the spectrum generation object includes the phoneme expansion data corresponding to the reference text, the speech spectrum data of the reference audio, and the emotion data of the laughter feature data, the phoneme expansion data includes the data of the assembled phoneme data of the assembled text after the phoneme data frame is expanded, and the assembled text is the text obtained by splicing the reference text and the audio text of the reference audio; The laughter speech spectrum data is determined through the speech spectrum generation model according to the spectrum generation object.

3. The method according to claim 2, characterized in that The data processing model comprises a first data processing model for processing the reference object, the first data processing model comprises a text encoding model and a phoneme duration prediction model, and the spectrum generation object comprises the phoneme expansion data; Before the reference object and the laughter feature data are processed by the data processing model to obtain a spectrum generation object, the method further includes: Performing speech recognition processing on the audio spectrum data of the reference audio to obtain the audio text; splicing the reference text and the audio text to obtain the compiled text; The step of processing the reference object and the laughter feature data by the data processing model to obtain a spectrum generation object includes: Performing phoneme conversion processing on the compiled text through the text encoding model to obtain the compiled phoneme data, wherein the compiled phoneme data includes a phoneme sequence corresponding to the compiled text; Calculate the second phoneme duration of each phoneme in the phoneme sequence according to the first phoneme duration of each phoneme in the phoneme sequence by using the phoneme duration prediction model, wherein the second phoneme duration is used to represent the duration of the phoneme expected to last in the reference audio; According to the second phoneme duration, the first phoneme duration of each phoneme in the assembled phoneme data is adjusted to obtain the phoneme extension data, wherein the number of data frames of the phoneme extension data is greater than the number of data frames of the assembled phoneme data.

4. The method according to claim 2, characterized in that: The data processing model includes a second data processing model for processing the reference object, the second data processing model is determined by a spectrum feature encoder, and the spectrum generation object includes speech spectrum data of the reference audio; The step of processing the reference object and the laughter feature data by the data processing model to obtain a spectrum generation object includes: The second data processing model is used to perform feature extraction on the audio spectrum data of the reference audio to obtain the speech spectrum data, wherein the data dimension of the speech spectrum data is lower than the data dimension of the audio spectrum data.

5. The method according to claim 2, characterized in that: The step of determining the laughter speech spectrum data by using the speech spectrum generation model and according to the spectrum generation object comprises: According to the second phoneme duration of each phoneme in the phoneme expansion data, align each phoneme in the phoneme expansion data with the spectrum features in the speech spectrum data of the reference audio to obtain aligned audio spectrum data, wherein the aligned audio spectrum data includes the speech spectrum data and mask spectrum data of the mask area; Reconstructing the audio spectrum data of the masked area by using the emotion data and the phoneme expansion data to obtain the reconstructed audio spectrum data of the masked area; Replacing the mask spectrum data in the aligned audio spectrum data with the reconstructed audio spectrum data to obtain expanded mask spectrum data; Based on the emotion data, determining the position of the laughter feature data in the expanded mask spectrum data; According to the position, the laughter feature data and the expanded mask spectrum data are fused to obtain the laughter speech spectrum data.

6. The method according to claim 2, characterized in that The method further comprises: Acquire a sample reference object and sample laughter feature data, wherein the sample reference object includes a sample reference audio and a sample reference text; Based on the sample reference object and the sample laughter feature data, training the first preprocessing model until a first preset training condition is met to obtain a second preprocessing model; The reference object and the sample laughter feature data are processed by the second preprocessing model to obtain a sample spectrum generation object, wherein the sample spectrum generation object includes sample phoneme expansion data corresponding to the sample reference text, sample speech spectrum data of the sample reference audio, and sample emotion data of the sample laughter feature data, wherein the sample phoneme expansion data includes data of sample assembly phoneme data of a sample assembly text after the phoneme data frame is expanded, and the sample assembly text is a text obtained by splicing the sample reference text and the sample audio text of the sample reference audio; By using a sample spectrum decoder, generating an object according to the sample spectrum, constructing first sample laughter speech spectrum data; Determine, by a sample spectrum discriminator, a difference value between the first sample laughter speech spectrum data and the second sample laughter speech spectrum data, wherein the second sample laughter speech spectrum data is reference laughter speech spectrum data corresponding to the sample reference object and the sample laughter feature data; When the difference value is less than or equal to a first threshold, the second preprocessing model is determined as the data processing model, and the sample spectrum decoder is determined as the speech spectrum generation model.

7. The method according to claim 6, characterized in that The first preprocessing model includes a first sample preprocessing model for processing the sample reference object, the first sample preprocessing model includes a text encoder and a phoneme duration predictor, the second sample data processing model includes a first data processing model in the data processing model for processing the reference object, the sample reference text is the text of the sample reference audio after speech recognition processing, and the sample reference audio and the sample reference text are time-aligned at the phoneme level; The first preprocessing model is trained based on the sample reference object and the sample laughter feature data until a first preset training condition is met to obtain a second preprocessing model, including: The sample reference text is subjected to phoneme conversion processing by the text encoder to obtain sample phoneme data, wherein the sample phoneme data includes a sample phoneme sequence corresponding to a text sentence in the sample reference text; By means of the phoneme duration predictor, a second sample phoneme duration of each sample phoneme in the sample phoneme sequence is calculated according to the first sample phoneme duration of each sample phoneme in the sample phoneme sequence, wherein the second sample phoneme duration is used to represent the duration of the sample phoneme expected to last in the sample reference audio; When the difference between the second sample phoneme duration and the third sample phoneme duration is less than or equal to a second threshold, the text encoder is determined as the text encoding model in the first data processing model, and the phoneme duration predictor is determined as the phoneme duration prediction model in the first data processing model, and the third sample phoneme duration is the actual duration of the sample phoneme in the sample reference audio.

8. The method according to claim 6, characterized in that The first preprocessing model includes a laughter emotion recognition model for processing the sample laughter feature data, and the second preprocessing model includes a third data processing model in the data processing model for processing the laughter feature data; The first preprocessing model is trained based on the sample reference object and the sample laughter feature data until a first preset training condition is met to obtain a second preprocessing model, including: Extracting sample laughter feature data from the sample audio; Using the laughter emotion recognition model, the sample laughter feature data is recognized to obtain first sample emotion data of the sample laughter feature data; When the difference value between the first sample emotion data and the second sample emotion data corresponding to the sample laughter feature data is less than or equal to a fourth threshold, the laughter emotion recognition model is determined as the third data processing model.

9. The method according to claim 6, characterized in that The method of generating an object according to the sample spectrum through a sample spectrum decoder to construct first sample laughter speech spectrum data includes: Extracting the sample audio spectrum data of the sample reference audio through the sample spectrum decoder; Performing mask processing on the sample audio spectrum data to obtain sample mask spectrum data, wherein the sample mask spectrum data includes sample mask spectrum data of at least one sample mask area; According to the second sample phoneme duration of each sample phoneme in the sample phoneme expansion data, align each sample phoneme in the sample phoneme expansion data with the sample spectrum feature in the sample speech spectrum data of the sample reference audio to obtain sample aligned audio spectrum data, wherein the sample aligned audio spectrum data includes the sample speech spectrum data and sample mask spectrum data of the sample mask area; Reconstructing the audio spectrum data of the sample mask area through the sample emotion data and the sample phoneme expansion data to obtain the sample reconstructed audio spectrum data of the sample mask area; Replacing the sample mask spectrum data in the sample aligned audio spectrum data with the sample reconstructed audio spectrum data to obtain sample expanded mask spectrum data; Based on the sample emotion data, determining a sample position of the sample laughter feature data in the sample expanded mask spectrum data; According to the sample position, the sample laughter feature data and the sample expanded mask spectrum data are fused to obtain the first sample laughter speech spectrum data.

10. The method according to claim 1, characterized in that Before obtaining the reference object and the laughter characteristic data related to the reference object, the method further includes: Displaying a laughter synthesis interface, wherein the laughter synthesis interface includes a first area and a second area; receiving a first input from a user on the first area; In response to the first input, determining the text input by the user in the first area as the reference text; receiving a second input from a user on the second area; In response to the second input, a preset audio selected by a user in the second area is determined as the reference audio; or a user audio newly added by the user in the second area is determined as the reference audio.

11. An audio generating device, characterized in that: include: An acquisition module, used for acquiring a reference object and laughter feature data related to the reference object, wherein the reference object includes a reference text and a reference audio, wherein the reference text is a text used for performing laughter synthesis, and the reference audio is an audio used for indicating generation of a preset style; A determination module, used for determining laughter speech spectrum data according to the reference object and the laughter feature data by using a spectrum generation model; A conversion module is used to convert the laughter speech spectrum data into laughter synthesis audio, wherein the laughter synthesis audio is the audio of the preset style, and the laughter synthesis audio includes the text content of the reference text and laughter related to the emotion represented by the text content.

12. The device according to claim 11, characterized in that The determination module is specifically used for, when the spectrum generation model includes a data processing model and a speech spectrum generation model, processing the reference object and the laughter feature data through the data processing model to obtain a spectrum generation object; wherein the spectrum generation object includes the phoneme expansion data corresponding to the reference text, the speech spectrum data of the reference audio, and the emotion data of the laughter feature data, the phoneme expansion data includes the data of the assembled phoneme data of the assembled text after the phoneme data frame is expanded, and the assembled text is the text obtained by splicing the reference text and the audio text of the reference audio; The laughter speech spectrum data is determined through the speech spectrum generation model according to the spectrum generation object.

13. An electronic device, characterized in that: include: A processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the audio generation method as described in any one of claims 1 to 10.

14. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the audio generation method according to any one of claims 1 to 10 are implemented.

Images