Audio generation method and related equipment

Abstract

The invention provides an audio generation method and related equipment, which are suitable for a voice interaction scene. The method comprises the following steps: processing related features of a text by combining a residual quantized correction flow model to obtain quantized acoustic features, and decoding the acoustic features by multiplexing an audio decoder to obtain an audio signal corresponding to the text. Wherein in the training process of the correction flow model, the input data of the correction flow model is obtained according to data encoded by an audio encoder (matched with an audio decoder). Compared with a common Mel spectrum, the quantized acoustic features are lower in dimension, so that compared with a model for generating the Mel spectrum, the diffusion model is smaller in parameter quantity and computing power demand, a multiplexing audio decoder can be considered for decoding without additionally configuring a decoding module, additional parameter quantity is not brought, and computing power is saved. Compared with a traditional diffusion model, the correction flow model can improve the reasoning efficiency. Therefore, the audio generation efficiency can be obviously improved, and the computing power of the electronic equipment is saved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to an audio generation method and related equipment. Background Art

[0002] Smart devices are often used in voice interaction scenarios, which requires voice assistants to provide audio generation capabilities. Currently, a series of speech synthesis methods have been proposed to enable voice assistants to generate audio.

[0003] However, the efficiency of current audio generation is subject to numerous limitations. In one related technology, text is first fed into a feature generation model to generate a mel-spectrogram, which is then converted into a playable time-domain audio signal using a vocoder. Due to the large number of parameters and computing power required, these audio generation models are typically deployed in the cloud, which can lead to a range of issues such as privacy leaks and latency. Furthermore, a vocoder is specifically designed to decode the mel-spectrogram. Due to the high dimensionality of the mel-spectrogram, the vocoder has a large number of parameters and a high computing power requirement.

[0004] Therefore, how to improve the efficiency of audio generation is an urgent problem to be solved. Summary of the Invention

[0005] The present application provides an audio generation method and related equipment, which can significantly improve the efficiency of audio generation and save computing power of electronic devices.

[0006] In a first aspect, an audio generation method is provided, which is applied to an electronic device, and the method includes: obtaining a first text of an audio signal to be generated; obtaining semantic features and phoneme length features of the first text; inputting the semantic features, phoneme length features and a first Gaussian noise obtained by random sampling into a modified flow model to obtain continuous acoustic features, wherein the semantic features and phoneme length features are used as conditional information, and the first Gaussian noise is used as an observation signal; decoding the continuous acoustic features to obtain a time-domain audio signal, wherein the content of the audio signal is the same as the content of the text.

[0007] It can be understood that using the modified flow module to process the relevant features of the text requires significantly fewer iterations than using a general diffusion model to generate acoustic features. This can reduce computing costs and increase inference speed, thereby improving generation efficiency.

[0008] In a possible embodiment, the electronic device includes an audio encoder and an audio decoder, the encoding formats of the audio encoder and the audio decoder match, and the audio decoder is the inverse process of the audio encoder; the training data of the modified flow model includes a first sample signal, and the first sample signal is obtained by processing the audio encoder; the continuous acoustic features are decoded to obtain a time domain audio signal, including: using the audio decoder to decode the continuous acoustic features to obtain the audio signal.

[0009] It's understandable that audio encoders and decoders are modules currently found in almost all electronic devices, while vocoders in related art are generally deployed on electronic devices with higher audio processing requirements. For example, in related art, due to the higher dimensionality of mel-spectrograms, vocoders are specifically designed to decode mel-spectrograms, and the number of vocoder parameters is generally larger than that of audio encoders. Because the quantized acoustic features have lower dimensionality than the mel-spectrograms used in related art, this application considers that there is no need to design a dedicated decoding device for the quantized acoustic features, and further considers reusing the audio decoder already in the electronic device for decoding. On the one hand, compared to the vocoders used in related art, which have larger parameter counts, the audio decoder in the audio codec system used in this application does not introduce additional parameters to the system, and the audio decoder also requires less computing power than the vocoder, thereby saving computing power in the electronic device and improving the efficiency of the audio generation method. On the other hand, since the electronic device does not require additional modules for decoding acoustic features, it is applicable to almost all electronic devices, which can expand the scope of application of the audio generation method compared to related art.

[0010] In a possible embodiment, the semantic feature, the phoneme length feature and the first Gaussian noise obtained by random sampling are input into the modified flow model to obtain a continuous acoustic feature, including: inputting the semantic feature, the phoneme length feature and the first Gaussian noise into the modified flow model to obtain a discrete code vector corresponding to the codebook, the discrete code vector is represented as the vector in the codebook that is closest to the continuous acoustic feature, and the discrete code vector is a discretized representation of the continuous acoustic feature; querying the codebook according to the discrete code vector to obtain the continuous acoustic feature.

[0011] It is understandable that the more complex the data to be generated or processed by the model, the larger the number of parameters of the model, and the greater the computing power required when the model is running. In the audio generation method of the present application, the acoustic features used to generate the audio signal are quantized acoustic features, which are lower in dimension than the Mel spectrum in the related art, that is, the Mel spectrum is more complex than the quantized acoustic features. Therefore, the feature generation model in the related art has a larger number of parameters and requires more computing power than the modified flow model combined with quantization processing. Therefore, the audio generation method provided by the present application can save the computing power of electronic equipment and improve the efficiency of the audio generation method.

[0012] In a possible embodiment, the codebook includes Q first codebooks, the discrete code vector includes Q first discrete code vectors, the Q first codebooks correspond one-to-one to the Q first discrete code vectors, Q ≥ 1 and Q is an integer; when Q ≥ 2, the qth first discrete code vector is represented as the vector in the qth first codebook that is closest to the continuous acoustic feature, 1 ≤ q ≤ Q and q is an integer, and querying the codebook according to the discrete code vector to obtain the continuous acoustic feature includes: querying the vector corresponding to the qth first discrete code vector in the qth codebook, thereby obtaining the vectors corresponding to the Q first discrete code vectors in the Q first codebooks; and performing a sum operation on the vectors corresponding to the Q first discrete code vectors to obtain the continuous acoustic feature.

[0013] It can be understood that the discrete code vector output by the modified flow model is the data after RVQ, which has a smaller dimension and a smaller number of parameters than the data that has not undergone RVQ. Therefore, the modified flow model used to generate discrete code vectors also has a smaller number of parameters and requires less computing power than the model used to generate data that has not undergone RVQ.

[0014] In a possible embodiment, the semantic features and phoneme length features of the first text are obtained, including: inputting the first text into a text encoder to obtain a hidden representation tensor, the hidden representation tensor is used to represent the semantic information of the first text; inputting the hidden representation tensor into a length prediction module to predict the phoneme length corresponding to each character in the first text, and obtaining a length vector based on the phoneme length corresponding to each character, wherein the phoneme length is the duration of the phoneme, the phoneme length corresponding to each character is the duration of the phoneme corresponding to each character, and the length vector indicates the phoneme length corresponding to each character; the method also includes: determining the total time corresponding to the first text based on the length vector, the total time includes N time frames, N>1 and is an integer, wherein the duration of the audio signal is the same as the total time; and randomly sampling N×Q sample points from a Gaussian distribution with a mean of 0 and a variance of 1 to obtain a first Gaussian noise.

[0015] It can be understood that N×Q sample points need to be generated to obtain the first Gaussian noise, that is, the data dimension of the first Gaussian noise is N×Q. In other words, Q sample points are generated in each time frame, where Q is the number of codebooks, so a total of N×Q sample points are generated in N time frames. In common codec systems (for example, with a codec rate of 6kbps), Q is generally 8 or 12. Therefore, the data dimension of the continuous acoustic features output by the modified flow model is also N×Q.

[0016] Traditional speech generation methods typically use mel-spectrograms as acoustic feature data. The data dimensions to be generated for N time frames are typically N × 80, where 80 is the mel-spectrogram dimension. In other words, traditional speech generation methods use 80 data points per time frame to represent the mel-spectrogram information.

[0017] Since N×80 is significantly higher than N×Q, the data dimension of the above scheme is significantly smaller than that of the traditional speech generation method. Therefore, compared with the feature generation module used to generate mel spectrum in the traditional speech generation method, the modified flow model can achieve a smaller number of parameters and a smaller amount of computation.

[0018] In a possible embodiment, the semantic features, phoneme length features and first Gaussian noise are input into the modified flow model to obtain a discrete code vector corresponding to the codebook, including: inputting the semantic features, phoneme length features and first Gaussian noise into the modified flow model to obtain a first derivative of the first Gaussian noise; performing Euler sampling on the first derivative to obtain a first sampling signal; inputting the semantic features, phoneme length features and first sampling signal into the modified flow model to obtain a second derivative of the first sampling signal; performing Euler sampling on the second derivative to obtain a second sampling signal; and obtaining a discrete code vector based on the second sampling signal.

[0019] It is understandable that after using the modified flow network, theoretically, a single-step sampling can be performed to obtain the generated result. Therefore, the sampling process is relatively simple. In this case, the effects achieved by using simple or complex sampling methods are almost the same. Therefore, this application prefers a simple sampling method, such as Euler sampling. Alternatively, other simple sampling methods can be used, which are not limited in this application. Moreover, when K ≥ 2, the final generation effect can be improved compared to sampling only once.

[0020] In a possible embodiment, the method also includes: inputting the sample clean audio signal into an audio encoder to obtain a sample discrete code vector; generating a first sample signal based on a randomly generated second Gaussian noise and the sample discrete code vector; obtaining a sample hidden layer representation tensor and a sample length vector based on the sample text; inputting the first sample signal, the sample hidden layer representation tensor and the sample length vector into a modified flow model to obtain a sample derivative; calculating a loss function based on the sample derivative and target data, the target data being the difference obtained by subtracting the second Gaussian noise from the sample discrete code vector; and updating the parameters of the modified flow model based on the loss function.

[0021] The above scheme uses the difference obtained by subtracting the second Gaussian noise from the sample discrete code vector as the training target to realize the training process of the modified flow model, so that the inference side of the modified flow model can improve the audio generation effect and increase the inference speed compared with the diffusion model.

[0022] In a second aspect, the present application provides an electronic device comprising one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code comprises computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the method described in the first aspect and any possible implementation of the first aspect.

[0023] In a third aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, which are used to call computer instructions to enable the electronic device to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0024] In a fourth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0025] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method described in the first aspect and any possible implementation of the first aspect.

[0026] It is understandable that the electronic device provided in the second aspect, the chip system provided in the third aspect, the computer storage medium provided in the fourth aspect, and the computer program product provided in the fifth aspect are all used to perform the methods provided in this application. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 An example of a voice interaction scenario provided in the embodiments of this application;

[0028] Figure 2 A schematic flow chart of an example of a related technology;

[0029] Figure 3 A schematic diagram of an example of an audio generation method provided in an embodiment of the present application;

[0030] Figure 4 A schematic flow chart of an audio generation method 100 provided in an embodiment of the present application;

[0031] Figure 5 A schematic diagram of an example of a discrete code-vector generation module 204 based on a modified flow model provided in an embodiment of the present application;

[0032] Figure 6 A schematic diagram of an example of a training process for a modified flow network provided in an embodiment of the present application;

[0033] Figure 7 A schematic diagram of the hardware structure of an electronic device 1000 provided in an embodiment of the present application;

[0034] Figure 8 A block diagram of a software system of an electronic device 1000 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0036] In order to more clearly understand the embodiments of the present application, the relevant concepts of residual vector quantization and correction flow involved in the present application are first introduced and explained.

[0037] Audio compression can approximate continuous features using a discretized representation, significantly reducing the size of audio files. Discrete audio representation (tokenization) means that audio can be represented as a sequence of discrete tokens. For example, this can be achieved by applying modeling techniques from natural language processing (e.g., large language models) to speech synthesis tasks.

[0038] Vector quantization (VQ) is a representative audio compression technology. An original vector v uses a codebook (CB) W, which contains A ordered codebook vectors (or codewords) w1, w2, ..., w a ,…,w A , where a∈[1,A] and a and A are both integers, by replacing the vector v with the closest vector (or closest codeword) w in the codebook W. a To represent (it is understandable that there are many ways to evaluate vector similarity, such as Euclidean distance, cosine similarity, etc.), the vector v can be represented as a discrete codebook vector (also referred to as a discrete vector corresponding to the codebook in this application, or simply as a discrete code vector) Token. Token = [Token0, Token1, ..., Token n ,…,Token N-1 ], where N represents the length of the discrete code vector, that is, the number of time frames corresponding to the vector v. In other words, Token n is the discretized representation of the continuous features of the nth time frame of vector v.

[0039] Residual vector quantization (RVQ) is another representative audio compression technology. It uses a clever "layered description" approach to balance performance and storage space. The VQ process is divided into multiple stages (related to the number of quantizers), and each stage (or quantizer) has a corresponding codebook W. q , where q∈[0,Q-1] and q and Q are both integers, Q is the number of stages, that is, the number of codebooks, and the codebook W q There are A ordered codebook vectors w q1 ,w q2 ,…,w qa ,..,w qA , where a∈[1,A] and a and A are both integers. An original input vector v is first generated by the first layer codebook (CB 1 ) in the nearest codebook vector w 1a Representation, then calculate v and w 1a The residual R1 between them is input into the next layer quantizer based on the second layer codebook (CB 2 ) is more accurately expressed, and so on, the Q codebook vectors from different codebooks that are closest to the vector v will be found in the Q codebooks. In this application, the discrete code vector of the nth time frame is in, The codebook vector closest to the acoustic feature of the nth time frame in the qth codebook is quantized as the sum of the Q closest codebook vectors, Feature, which is called the continuous acoustic feature of the compressed audio in this application.

[0040] In this application, Feature = [Feature0, Feature1, ..., Feature n ,…,Feature N-1 ], where N is the number of time frames corresponding to the original vector v. 0≤n≤N and both n and N are integers. That is, Feature n is the acoustic feature of the audio signal corresponding to the text text in the nth time frame. Among them, codebook q represents the qth codebook among the Q codebooks above, Indicates querying and taking out the qth codebook codebook vectors; Indicates that the Q codebooks will be based on The queried Q codebook vectors are summed up.

[0041] Diffusion models are a general generative modeling technique for generating high-dimensional, perceptual data, such as images and videos. By simulating the physical diffusion process, data is gradually converted into noise. The inverse process is then learned to gradually recover the original data from the noise, achieving high-quality generative results. The generative process requires multiple iterations, resulting in high computational cost and slow inference speed. The training and generation processes require significant computational resources, especially when generating high-resolution images or high-quality audio.

[0042] This application also involves Rectified Flow. Rectified Flow is an emerging generative model that constructs a flow-based generative model through linear interpolation, and then draws a DC trajectory through flow operations. It connects data and noise through straight lines. Compared with the diffusion model, it has better theoretical properties and conceptual simplicity. For example, the number of iterations required in the generation process is significantly less. In theory, one iteration (derivative of the straight line) can get the generation result, which can reduce computing costs and increase reasoning speed, thereby improving generation efficiency.

[0043] Smart terminal devices are often used in voice interaction scenarios, which requires voice assistants to provide the ability to generate audio. Figure 1 This is an example of a voice interaction scenario provided by the embodiment of this application. Figure 1 As shown, the electronic device detects that the user says, "Hello YOYO, are there any good movies recently?" The smart assistant panel on interface 1 displays the text "Are there any good movies recently?" that matches the user's voice. Then, interface 2 is displayed, and the audio "I found these great videos. Which one should I choose?" is played to the user. The smart assistant panel on interface 2 displays the text that matches the user's voice, the text that matches the audio "I found these great videos. Which one should I choose?", and search results based on the user's voice in a dialogue format.

[0044] At present, a series of speech synthesis methods have been proposed to realize the function of voice assistants generating audio. Figure 2 FIG. 1 is a schematic flow chart of an example of a related technology. Figure 2 As shown, text (e.g., "I found these wonderful videos. Which one should I choose?") is first fed into a feature generation model to generate acoustic features (e.g., mel-spectrograms). A vocoder is then used to convert the mel-spectrograms into a playable time-domain audio signal. Due to the large number of parameters and computing power required, these audio generation models are typically deployed in the cloud, potentially leading to privacy issues and latency. A vocoder is specifically designed to decode the mel-spectrograms. Due to the high dimensionality of the mel-spectrograms, the vocoder has a large number of parameters and a high computing power requirement.

[0045] Therefore, how to improve the efficiency of audio generation is an urgent problem to be solved. In view of this, the present application provides an audio generation method that can significantly improve the efficiency of the audio generation method.

[0046] Figure 3 This is a schematic diagram of an example of the audio generation method provided in the embodiment of the present application. Figure 3 As shown, the text for generating the audio signal is first obtained. The relevant features of the text are processed using a modified flow model combined with quantization processing to obtain quantized acoustic features. The quantized acoustic features are then decoded by the audio decoder of the multiplexing electronic device to obtain a playable time-domain audio signal. During the training of the modified flow model, the input data for the modified flow model is obtained based on the data encoded by the audio encoder. The encoding formats of the audio encoder and audio decoder match, and the audio decoder is the inverse process of the audio encoder.

[0047] It is understandable that the more complex the data to be generated or processed by the model, the larger the number of parameters of the model, and the greater the computing power required when the model is running. In the audio generation method of the present application, the acoustic features used to generate the audio signal are quantized acoustic features, which have a lower dimension than the Mel spectrum in the related art, that is, the Mel spectrum is more complex than the quantized acoustic features. Therefore, the modified flow model combined with the quantization processing has a smaller number of parameters and a smaller computing power requirement than the feature generation model in the related art. Therefore, the audio generation method provided by the present application can save the computing power of electronic equipment and improve the efficiency of the audio generation method.

[0048] It can also be understood that audio encoders and audio decoders are modules currently available in almost all electronic devices, while vocoders in the related art are generally configured on electronic devices with higher audio processing requirements. For example, in the related art, due to the higher dimensionality of the mel-spectrogram, vocoders are specifically designed to decode the mel-spectrogram, and the number of parameters in the vocoder is generally larger than that of the audio encoder. Because the quantized acoustic features have lower dimensionality than the mel-spectrograms in the related art, the present application considers that there is no need to design a decoding device specifically for the quantized acoustic features, and further considers reusing the audio decoder already in the electronic device for decoding. On the one hand, compared to the vocoders used in the related art, which have larger parameters, the audio decoder in the audio codec system used in the present application does not introduce additional parameters to the system, and the audio decoder also requires less computing power than the vocoder, thereby saving computing power in the electronic device and improving the efficiency of the audio generation method. On the other hand, since the electronic device does not require additional modules for decoding the acoustic features, it is applicable to almost all electronic devices, which can expand the scope of application of the audio generation method compared to the related art.

[0049] It can also be understood that the use of the modified flow module to process the relevant features of the text requires significantly fewer iterations in the generation process compared to the solution of using a general diffusion model to generate acoustic features, which can reduce computing costs and increase inference speed, thereby improving generation efficiency.

[0050] In summary, the audio generation method provided in this application can significantly improve the efficiency of audio generation and save the computing power of electronic devices.

[0051] The following will be combined Figures 4 to 7 , which details the relevant contents of the audio generation method provided in this application.

[0052] Figure 4 This is a schematic flowchart of the audio generation method 100 provided in an embodiment of the present application.

[0053] S101: Obtain a first text of an audio signal to be generated.

[0054] Exemplarily, the first text is uploaded by a user, downloaded from the Internet, or generated by a neural network model configured by the electronic device. Figure 1 In the scenario shown, the first text "I found these wonderful videos, which one should I choose" is generated by the neural network model that the voice assistant application relies on based on the user's voice.

[0055] The following description will be made by taking the text text as an example of the first text and the text acquisition module 201 executing S101 as an example.

[0056] The text acquisition module 201 acquires the text used for synthesizing audio. p ,…text P-1 ], where P represents the length of the text. The length of the text can be understood as the number of characters included in the text. The text acquisition module 201 transmits the text text to the text encoder 202.

[0057] S102: Acquire semantic features and phoneme length features of the first text.

[0058] Next, the latent layer representation tensor latent text As a possible example of semantic features, the length vector L pre A possible example of a phoneme length feature is described below.

[0059] The text encoder 202 encodes the text using a pre-trained model to obtain the latent representation tensor of the text. text Among them, the latent layer representation tensor latent textIt can be used to represent the semantic information of the text. The text encoder 202 converts the latent representation tensor latent text The data is transmitted to the length prediction module 203 and the discrete code vector generation module 204 based on the modified flow model.

[0060] The length prediction module 203 uses a pre-trained model to represent the latent tensor according to the latent text , predict the phoneme length corresponding to each character in the text text, and output the length vector L consisting of the phoneme lengths corresponding to all characters pre .

[0061] It can be understood that a character is the smallest distinctive unit in a writing system; a phoneme is the smallest unit in speech. If speech is to be generated from text, then a character can correspond to one or more phonemes. The phoneme length can be understood as the duration of the phoneme (for example, expressed as the number of time frames or spectral frames). The phoneme length corresponding to each character can be understood as the duration of the phoneme corresponding to each character (for example, expressed as the number of time frames or spectral frames).

[0062] For example, the length vector L pre It includes P numbers arranged in character order. The P numbers are the lengths of the phonemes corresponding to the P characters in the text. In other words, the length vector L pre Indicates the phoneme length corresponding to each character in the text text.

[0063] The length prediction module 203 converts the length vector L pre The data is transmitted to the discrete code vector generation module 204 based on the modified flow model.

[0064] S103: inputting the semantic features, the phoneme length features and the first Gaussian noise obtained by random sampling into the modified flow model to obtain continuous acoustic features.

[0065] Exemplarily, S103 may be implemented by a discrete code-vector generating module 204 based on a modified flow model and a codebook fusion and query module 205 .

[0066] The discrete code vector generation module 204 based on the modified flow model converts the latent layer representation tensor latent text and length vector L pre As conditional information, the modified flow model is used to generate the discrete vector corresponding to the codebook from the randomly sampled white noise (that is, Gaussian noise) samples, that is, the discrete code vector Token = [Token0, Token1, ..., Token n ,……,Token N-1 ]. Where N represents the length of the discrete code vector.

[0067] It can be understood that the discrete code vector generation module 204 based on the modified flow model can generate the code vector according to the length vector L pre Determine the sum of the phoneme lengths of all characters, that is, the total number of time frames of the audio signal corresponding to the text text (that is, the audio signal y(t) finally generated by this method, which is described here uniformly and will not be repeated below). pre The total number of time frames, N, can be obtained by adding up all the values in . In other words, the audio signal corresponding to the text text lasts for N time frames. Since the discrete code vector generation module 204 based on the modified flow model outputs the corresponding discrete code vector for each time frame, the length of the discrete code vector Token is N.

[0068] In the quantization process, Q codebooks are used, where Q ≥ 2 and is an integer. Then, in each time frame, discrete code vectors are output for each of the Q codebooks. in, is the codebook vector (or simply referred to as code vector) closest to the acoustic feature of the nth time frame in the qth codebook included in the Q codebooks, q∈[0,Q-1] and q is an integer.

[0069] The discrete code vector generation module 204 based on the modified flow model transmits the discrete code vector Token to the codebook fusion and query module 205 .

[0070] Specifically, the following will combine Figure 5 A possible implementation of the discrete code-vector generation module 204 based on the modified flow model is introduced in detail.

[0071] It can be understood that using the modified flow module to process the relevant features of the text requires significantly fewer iterations than using a general diffusion model to generate acoustic features. This can reduce computing costs and increase inference speed, thereby improving generation efficiency.

[0072] The codebook fusion and query module 205 queries the discrete code vector Token and obtains the continuous acoustic feature Feature = [Feature0, Feature1, ..., Feature n ,……,Feature N-1 ]. Among them, Feature n is the acoustic feature of the audio signal corresponding to the text text in the nth time frame. Among them, codebook q represents the qth codebook among the Q codebooks above, Indicates querying and taking out the qth residual codebook in the audio codec module. vectors; Indicates that, in the Q codebooks, The queried Q codebook vectors are summed up.

[0073] The codebook fusion and query module 205 transmits the continuous acoustic feature Feature to the audio decoder 206 .

[0074] It can be understood that the discrete code vector output by the modified flow model is the data after RVQ, which has a smaller dimension and a smaller number of parameters than the data that has not undergone RVQ. Therefore, the modified flow model used to generate discrete code vectors also has a smaller number of parameters and requires less computing power than the model used to generate data that has not undergone RVQ.

[0075] S104: Decode the continuous acoustic features to obtain a time-domain audio signal.

[0076] The audio decoder 206 decodes the continuous acoustic feature Feature to obtain an audio signal y(t)=[y(0), y(1), ..., y(T-1)], where T represents the total length of the audio sampling points.

[0077] It is understood that the electronic device includes an audio encoder and an audio decoder, the encoding formats of the audio encoder and audio decoder match, and the audio decoder is the inverse process of the audio encoder. The training data of the modified flow model includes a first sample signal, which is obtained by processing the audio encoder. In other words, the training process of the modified flow model uses the audio signal processed by the audio encoder as the training data of the modified flow model, so that the data output by the modified flow model during inference can be decoded by the audio decoder.

[0078] It can also be understood that audio encoders and audio decoders are modules currently available in almost all electronic devices, while vocoders in the related art are generally configured on electronic devices with higher audio processing requirements. For example, in the related art, due to the higher dimensionality of the mel-spectrogram, vocoders are specifically designed to decode the mel-spectrogram, and the number of parameters in the vocoder is generally larger than that of the audio encoder. Because the quantized acoustic features have lower dimensionality than the mel-spectrograms in the related art, the present application considers that there is no need to design a decoding device specifically for the quantized acoustic features, and further considers reusing the audio decoder already in the electronic device for decoding. On the one hand, compared to the vocoders used in the related art, which have larger parameters, the audio decoder in the audio codec system used in the present application does not introduce additional parameters to the system, and the audio decoder also requires less computing power than the vocoder, thereby saving computing power in the electronic device and improving the efficiency of the audio generation method. On the other hand, since the electronic device does not require additional modules for decoding the acoustic features, it is applicable to almost all electronic devices, which can expand the scope of application of the audio generation method compared to the related art.

[0079] The audio decoder 206 transmits the audio signal y(t) to the audio signal output module 207 .

[0080] The audio signal output module 207 outputs the generated audio signal y(t) to a downstream module or a speaker for playback.

[0081] The above scheme processes the relevant features of the text by combining the modified flow model with quantization processing to obtain quantized acoustic features, and reuses the existing audio decoder to decode the quantized acoustic features, which can significantly improve the efficiency of audio generation and save the computing power of electronic devices.

[0082] Figure 5 This is a schematic diagram of an example of a discrete code-vector generation module 204 based on a modified flow model provided in an embodiment of the present application.

[0083] It is understood that a total of K sampling steps are performed, where K is a preset number of sampling steps, K is an integer greater than 1, preferably K ≥ 2, for example K = 3. Although theoretically the correction flow can be iterated in one step to obtain the generated result, in this application, K ≥ 2 is preferably used to optimize the generation effect.

[0084] Initial execution Figure 5 In the process shown, k = K. Here, 1≤k≤K and k and K are both integers.

[0085] like Figure 5 As shown, the length vector L pre and the latent representation tensor latent text Refer to the above description respectively and be transmitted to the modified flow network inference module 303.

[0086] The Gaussian noise sampling module 301 randomly and independently samples N×Q sample points from a Gaussian distribution with a mean of zero and a variance of 1. Where N and Q refer to the description above. In other words, the sample is randomly and independently taken from a Gaussian distribution with a mean of zero and a variance of 1 on a time frame and code basis. The sampled Gaussian noise (i.e., the first Gaussian noise mentioned above) is defined as x K , or, is defined as x for k = K k .

[0087] The Gaussian noise sampling module 301 converts the Gaussian noise x K Transmitted to the modified flow network inference module 303.

[0088] Sampling time update module 302, according to the formula Generate the sampling time of step k, where t min It can be taken as 0.003, t maxCan be set to 1, which is the first execution Figure 5 In the process shown, k = K, so t k =0.

[0089] The sampling time updating module 302 sets t k Transmitted to the modified flow network inference module 303.

[0090] The input data of the modified flow network inference module 303 includes x k , t k , length vector L pre and the latent representation tensor latent text , where x k and t k As the observation signal or reference signal of the modified flow network inference module 303, the length vector L pre and the latent representation tensor latent text As condition information. The output data of the modified flow network inference module 303 is x k t k The derivative v k .

[0091] Exemplarily, the modified flow network inference module 303 can be implemented based on a network model or a combination of multiple network models such as noise conditional scoring network (NCSN)++ network, convolutional neural networks (CNN), convolutional recurrent neural networks (CRNN), U-type neural network pooling subgraph structure (U-Net), etc.

[0092] Specifically, the training process of the modified flow network will be combined with Figure 6 Provide a detailed introduction.

[0093] The modified flow network inference module 303 converts the derivative v k Transmitted to the Euler sampling module 304.

[0094] Euler sampling module 304, through the formula Sampling is performed, and the sampling number k=k-1 is updated.

[0095] It is understandable that after using the modified flow network, theoretically, a single-step sampling can be performed to obtain the generated result. Therefore, the sampling process is relatively simple. In this case, the effects achieved by using simple or complex sampling methods are almost the same. Therefore, this application prefers a simple sampling method, such as Euler sampling. Alternatively, other simple sampling methods can be used, and this application does not limit them.

[0096] The judgment module 305 judges whether the preset number of sampling steps is reached.

[0097] For example, it is determined whether the preset number of sampling steps has been reached by judging whether the sampling number k is equal to 0. If k = 0, it is determined that the preset number of sampling steps has been reached, and x k+1 Output as discrete code vector Token; if k≠0, then x k Continue sampling, that is, continue to generate updated t according to the updated k (updated by the Euler sampling module 304) by the sampling time updating module 302. k , and the updated x k and t k as the observation signal or reference signal, and the length vector L pre and the latent representation tensor latent text As condition information input to the modified flow network reasoning module 303, so as to continue to execute Figure 5 The process shown.

[0098] That is, when K≥2 and k>1, starting from the second time the sampling time is generated by the sampling time updating module 302 (including the second time), each subsequent sampling will not be K Input correction flow network inference module 303, instead of the x obtained after the last sampling k Input the modified flow network inference module 303.

[0099] It can be understood that multiple sampling of the data output by the modified flow model can improve the final generation effect.

[0100] It can be understood that the Gaussian noise sampling module 301 needs to generate N×Q sample points, that is, the data dimension is N×Q. In other words, Q sample points are generated in each time frame, where Q is the number of codebooks, so N time frames generate a total of N×Q sample points. In a commonly used codec system (for example, a codec rate of 6kbps), Q is generally 8 or 12. Therefore, the derivative v output by the modified flow network inference module 303 is k The data dimension of is also N×Q, and the data dimension of the discrete code vector finally output by the discrete code vector generation module 204 based on the modified flow model is also N×Q.

[0101] In traditional speech generation methods, mel-spectrograms are used as acoustic feature data. The data dimensions to be generated for N time frames are generally N × 80, where 80 is the dimension of the mel-spectrogram. In other words, in traditional speech generation methods, 80 data points are used to represent the mel-spectrogram information in each time frame.

[0102] Since N×80 is significantly higher than N×Q, the data dimension of the above scheme is significantly smaller than that of the traditional speech generation method. Therefore, the discrete code vector generation module 204 based on the modified flow model can achieve a smaller number of parameters and a smaller amount of calculation compared to the feature generation module in the traditional speech generation method.

[0103] It can be understood that when k=K, the derivative v k For the example of the first derivative mentioned above, the x obtained by Euler sampling is k-1 is an example of the first sampling signal mentioned above. In the case of k<K, the derivative v k For the example of the second derivative mentioned above, the x obtained by Euler sampling is k-1 is an example of the second sampling signal mentioned above.

[0104] Figure 6 A schematic diagram of an example of a training process for a modified flow network provided in an embodiment of the present application.

[0105] The clean speech acquisition module 401 is configured to acquire a sample clean audio signal Yx(t) from a clean sample audio data set.

[0106] The clean speech module 401 transmits the sample clean audio signal Yx(t) to the audio encoder 403 .

[0107] The text acquisition module 402 acquires a sample text Ytext corresponding to the sample clean audio signal.

[0108] For example, the sample text dataset includes sample texts that can form an audio-text data pair with the sample clean audio signal, and the sample text Ytext corresponding to the sample clean audio signal is obtained from the sample text dataset.

[0109] For another example, if the sample text dataset does not include sample text corresponding to the sample clean audio signal, an automatic speech recognition system may be used to recognize the text corresponding to the sample clean audio signal as the sample text Ytext.

[0110] The text acquisition module 402 transmits the sample text Ytext to the text encoder 408 .

[0111] The audio encoder 403 encodes the sample clean audio signal to obtain a sample discrete code vector YToken.

[0112] YToken=[YToken0, YToken1,…, YToken m ,……,YToken M-1 ].

[0113] It can be understood that the audio encoder and the audio decoder are two submodules in an audio codec module of an electronic device that appear in pairs, have matching encoding formats, and perform inverse operations.

[0114] The audio encoder 403 transmits the sample discrete code vector YToken to the sample generation module 406 .

[0115] Sampling time generation module 404, which is used to randomly generate a Yt min To Yt max The decimal between is the sample time Yt, where Yt min is the minimum value, such as 0.003, Yt max The maximum value, such as 1.

[0116] The sampling time generation module 404 transmits the sample sampling time Yt to the Gaussian noise generation module 405 and the correction flow network 407 .

[0117] Gaussian noise generation module 405 generates sample Gaussian noise Ynoise with a mean of zero and a standard deviation of σ(Yt) according to time t. The standard deviation can be calculated in various ways, for example,

[0118] The Gaussian noise generation module 405 outputs the generated Gaussian noise Ynoise to the sample generation module 406 .

[0119] The sample generation module 406 adds the sample Gaussian noise Ynoise and the sample discrete code vector YToken to obtain a noisy sample Ynoisy_Token.

[0120] The sample generation module 406 transmits the noisy sample Ynoisy_Token to the correction flow network 407 .

[0121] The text encoder 408 encodes the sample text Ytext using a pre-trained model to obtain the sample latent layer representation tensor Ylatent of the sample text Ytext text Among them, the sample hidden layer representation tensor Ylatent text It can be used to represent the semantic information of the sample text Ytext.

[0122] The text encoder 408 converts the sample hidden layer representation tensor Ylatent text Transmitted to the length prediction module 409, and based on the modified flow network 407.

[0123] The length prediction module 409 uses a pre-trained model to represent the sample hidden layer tensor Ylatent text, predict the phoneme length of each character in the sample text Ytext, and output the sample length vector YL consisting of the phoneme lengths of all characters pre .

[0124] For an understanding of the phoneme length and sample length vector, please refer to the above description of the length vector of the phoneme length.

[0125] The length prediction module 409 converts the sample length vector YL pre Transmitted to the correction flow network 407.

[0126] Modify the flow network 407 and represent the sample hidden layer tensor Ylatent text and the sample length vector YL pre As conditional information, a sample derivative Yv is generated according to the noisy sample Ynoisy_Token and the sample time Yt, and the sample derivative Yv is the corrected stream.

[0127] Exemplarily, the correction flow network 407 can be implemented based on one network model or a combination of multiple network models such as NCSN, CNN, CRNN, U-Net, etc.

[0128] The modified flow network 407 transmits the sample derivative Yv to the loss function calculation module 410.

[0129] The loss function calculation module 410 calculates the loss function corresponding to the sample derivative Yv: loss = mse(Yv, YToken - Ynoise). Mean squared error (MSE) is a commonly used loss function in regression tasks. It is defined as the average of the squares of the differences between the predicted value and the true value. YToken - Ynoise is the target data, or the training target.

[0130] The loss function calculation module 410 transmits the loss function loss to the network parameter updating module 411 .

[0131] The network parameter updating module 411 updates the parameters of the correction flow network 407 according to the loss function loss.

[0132] The above scheme uses YToken-Ynoise as the training target to implement the training process of the modified flow model, so that the inference side of the modified flow model can improve the audio generation effect and increase the inference speed compared to the diffusion model.

[0133] Figure 7 This is a hardware structure diagram of an electronic device 1000 provided in an embodiment of the present application. Figure 7The electronic device 1000 may include a processor 1010, an external memory interface 1020, an internal memory 1021, a universal serial bus (USB) interface 1030, a charging management module 1040, a power management module 1041, a battery 1042, an antenna 1, an antenna 2, a mobile communication module 1050, a wireless communication module 1060, an audio module 1070, a speaker 1070A, a receiver 1070B, a microphone 1070C, an earphone interface 1070D, a sensor module 1080, a button 1090, a motor 1091, an indicator 1092, a camera 1093, a display screen 1094, and a subscriber identification module (SIM) card interface 1095, etc. The sensor module 1080 may include a pressure sensor 1080A, a gyroscope sensor 1080B, an air pressure sensor 1080C, a magnetic sensor 1080D, an acceleration sensor 1080E, a distance sensor 1080F, a proximity light sensor 1080G, a fingerprint sensor 1080H, a temperature sensor 1080J, a touch sensor 1080K, an ambient light sensor 1080L, a bone conduction sensor 1080M, etc.

[0134] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 1000. In other embodiments of the present application, the electronic device 1000 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0135] The processor 1010 may include one or more processing units, for example: the processor 1010 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc.

[0136] The controller may be the nerve center and command center of the electronic device 1000. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0137] Processor 1010 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 1010 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 1010. If processor 1010 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 1010 latency, and thus improves system efficiency.

[0138] In some embodiments, the processor 1010 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0139] The MIPI interface can be used to connect the processor 1010 to peripheral devices such as the display 1094 and the camera 1093. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 1010 and the camera 1093 communicate via the CSI interface to implement the camera function of the electronic device 1000. The processor 1010 and the display 1094 communicate via the DSI interface to implement the display function of the electronic device 1000.

[0140] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 1000. In other embodiments of the present application, the electronic device 1000 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0141] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 1000 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0142] Video codecs are used to compress or decompress digital video. Electronic device 1000 may support one or more video codecs. This allows electronic device 1000 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0143] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 1000, such as image recognition, face recognition, voice recognition, and text comprehension.

[0144] Electronic device 1000 can implement audio functions, such as music playback and recording, through audio module 1070, speaker 1070A, receiver 1070B, microphone 1070C, headphone jack 1070D, and application processor. Keys 1090 include a power button, volume button, and the like. Keys 1090 can be mechanical or touch-sensitive. Electronic device 1000 can receive key inputs and generate key signal inputs related to user settings and function control of electronic device 1000.

[0145] Next, the software system of the electronic device 1000 will be described.

[0146] For example, the electronic device 1000 may be a mobile phone. The software system of the electronic device 1000 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a microservice architecture, or a cloud architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to illustrate the software system of the electronic device 1000.

[0147] Figure 8 This is a block diagram of a software system of an electronic device 1000 provided in an embodiment of the present application. Figure 8The layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers: from top to bottom, the application layer (application layer), the application framework layer (framework layer), the hardware abstraction layer (HAL), the driver layer, and the hardware layer.

[0148] The application layer may include a series of application packages, such as a dialing application, a gallery application, etc. (not shown in the figure). In the embodiment of the present application, the application package may include applications such as video, music, and voice assistants, all of which play audio to the user. In particular, the voice assistant application can generate audio based on the text and play the generated audio signal to the user.

[0149] Alternatively, the application layer may also include other applications that need to play audio signals generated based on text to users, which is not limited in this application.

[0150] The framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions. In the embodiment of the present application, the framework layer includes an audio generation service interface. The audio generation service interface can provide an API and programming framework for applications that obtain audio generation services.

[0151] The Hardware Abstraction Layer (HAL) is an interface layer between the operating system kernel and upper-layer software, providing a virtual hardware platform for the operating system. In embodiments of the present application, the Hardware Abstraction Layer may include an audio generation algorithm. The audio generation algorithm may include the operating code and data that implement the audio generation method provided in embodiments of the present application.

[0152] The driver layer is the layer between hardware and software. It includes drivers for various hardware components. This layer can include drivers for audio digital signal processors (DSPs). These drivers are used to drive the DSPs to process audio digital signals.

[0153] The hardware layer includes an audio digital signal processor. The audio digital signal processor can be used to process audio digital signals. It should be noted that the application provides Figure 8 The software structure diagram of the electronic device shown is only an example and does not limit the specific module division in different layers of the Android operating system. For details, please refer to the introduction of the Android operating system software structure in conventional technology.

[0154] The following describes the method in the embodiment of the present application in detail in combination with the above hardware structure and system structure:

[0155] In response to enabling applications such as video, music, or voice assistants, such applications can call the audio generation service interface to obtain the application programming interface and programming framework provided by the audio generation service.

[0156] The audio generation service initializes the audio generation algorithm. The algorithm retrieves text for generating an audio signal, generates continuous acoustic features based on a stored audio generation method, and processes the continuous acoustic features using an audio signal processor to produce a time-domain audio signal for output.

[0157] Specifically, how to process the audio signal can be found in the previous section. Figures 4 to 6 The method flow chart shown.

[0158] Finally, the audio generation algorithm can pass the generated audio signal back to the audio generation service and then back to the application layer.

[0159] The present invention provides a chip system comprising one or more processors configured to retrieve and execute instructions stored in a memory, thereby executing the method of the present invention. The chip system may be composed of a chip or may include a chip and other discrete devices.

[0160] The chip system may include an input circuit or interface for sending information or data, and an output circuit or interface for receiving information or data.

[0161] The present application also provides a computer program product, which, when executed by a processor, implements the method described in any method embodiment of the present application.

[0162] The computer program product can be stored in a memory and finally converted into an executable target file that can be executed by a processor through preprocessing, compilation, assembly and linking.

[0163] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer, implements the method described in any method embodiment of the present application. The computer program can be a high-level language program or an executable target program.

[0164] The computer-readable storage medium may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0165] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and equipment and the technical effects produced can refer to the corresponding processes and technical effects in the aforementioned method embodiments, and will not be repeated here.

[0166] In the several embodiments provided in this application, the disclosed systems, devices and methods can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not executed. The device embodiments described above are merely schematic, and the division of units is only a logical function division. There may be other division methods in actual implementation, and multiple units or components may be combined or integrated into another system. In addition, the coupling between the units or the coupling between the components may be direct coupling or indirect coupling, and the above coupling includes electrical, mechanical or other forms of connection.

[0167] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0168] It should be understood that the term "plurality" used herein refers to two or more. The term "and / or" in this document simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the related objects are in an "or" relationship.

[0169] The terms (or numbers) "first", "second", ... etc. that appear in the embodiments of the present application are only used for descriptive purposes, that is, they are only used to distinguish different objects, such as different "coordinates", etc., and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first", "second", ... etc. may explicitly or implicitly include one or more features. In the description of the embodiments of the present application, "at least one (item)" refers to one or more. "Multiple" means two or more. "At least one of the following (item)" or similar expressions refers to any combination of these items, including any combination of a single (item) or plural (items).

[0170] In short, the above description is only a preferred embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application shall be included in the scope of protection of this application.

Claims

1. An audio generation method, applied to electronic equipment, characterized in that: The method comprises: Obtaining a first text of an audio signal to be generated; Acquiring semantic features and phoneme length features of the first text; Inputting the semantic feature, the phoneme length feature, and a first Gaussian noise obtained by random sampling into a modified flow model to obtain a continuous acoustic feature, wherein the semantic feature and the phoneme length feature are used as conditional information, and the first Gaussian noise is used as an observation signal; The continuous acoustic features are decoded to obtain a time-domain audio signal, where the content of the audio signal is the same as the content of the text.

2. The method according to claim 1, wherein The electronic device comprises an audio encoder and an audio decoder, wherein the encoding formats of the audio encoder and the audio decoder match and the audio decoder is an inverse process of the audio encoder; The training data of the modified flow model includes a first sample signal, where the first sample signal is obtained by processing the audio encoder; The decoding of the continuous acoustic features to obtain a time-domain audio signal includes: The continuous acoustic features are decoded using the audio decoder to obtain the audio signal.

3. The method according to claim 1 or 2, wherein: The step of inputting the semantic feature, the phoneme length feature, and the first Gaussian noise obtained by random sampling into a modified flow model to obtain a continuous acoustic feature includes: Inputting the semantic feature, the phoneme length feature, and the first Gaussian noise into a modified flow model to obtain a discrete code vector corresponding to a codebook, wherein the discrete code vector is represented as a vector in the codebook that is closest to the continuous acoustic feature and is a discretized representation of the continuous acoustic feature; The codebook is queried according to the discrete code vector to obtain the continuous acoustic feature.

4. The method according to claim 3, wherein The codebook includes Q first codebooks, the discrete code vectors include Q first discrete code vectors, the Q first codebooks correspond to the Q first discrete code vectors one-to-one, Q ≥ 1 and Q is an integer; When Q≥2, the qth first discrete code vector is represented as the vector in the qth first codebook that is closest to the continuous acoustic feature, 1≤q≤Q and q is an integer, and querying the codebook according to the discrete code vector to obtain the continuous acoustic feature includes: Searching the qth codebook for a vector corresponding to the qth first discrete code vector, thereby obtaining vectors in Q first codebooks corresponding to the Q first discrete code vectors respectively; A sum operation is performed on the vectors corresponding to the Q first discrete code vectors to obtain the continuous acoustic feature.

5. The method according to claim 4, wherein The obtaining of the semantic features and phoneme length features of the first text includes: Inputting the first text into a text encoder to obtain a hidden layer representation tensor, where the hidden layer representation tensor is used to represent semantic information of the first text; Inputting the hidden layer representation tensor into a length prediction module, predicting the phoneme length corresponding to each character in the first text, and obtaining a length vector based on the phoneme length corresponding to each character, wherein the phoneme length is the duration of the phoneme, the phoneme length corresponding to each character is the duration of the phoneme corresponding to each character, and the length vector indicates the phoneme length corresponding to each character; The method further comprises: Determining a total time corresponding to the first text according to the length vector, the total time including N time frames, where N>1 and is an integer, wherein the duration of the audio signal is the same as the total time; The first Gaussian noise is obtained by randomly sampling N×Q sample points from a Gaussian distribution with a mean of 0 and a variance of 1.

6. The method according to claim 3 or 4, wherein: The step of inputting the semantic feature, the phoneme length feature, and the first Gaussian noise into a modified stream model to obtain a discrete code vector corresponding to a codebook includes: Inputting the semantic feature, the phoneme length feature, and the first Gaussian noise into the modified flow model to obtain a first derivative of the first Gaussian noise; Performing Euler sampling on the first derivative to obtain a first sampling signal; Inputting the semantic feature, the phoneme length feature, and the first sampled signal into the modified flow model to obtain a second derivative of the first sampled signal; Performing Euler sampling on the second derivative to obtain a second sampling signal; The discrete code vector is obtained according to the second sampling signal.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Input the sample clean audio signal into the audio encoder to obtain a sample discrete code vector; generating a first sample signal according to a randomly generated second Gaussian noise and the sample discrete code vector; Obtain the sample hidden layer representation tensor and sample length vector based on the sample text; Inputting the first sample signal, the sample hidden layer representation tensor and the sample length vector into the modified flow model to obtain a sample derivative; Calculating a loss function based on the sample derivative and target data, where the target data is a difference between the sample discrete code vector and the second Gaussian noise; Parameters of the modified flow model are updated according to the loss function.

8. An electronic device, characterized in that: The electronic device includes: one or more processors, and a memory; The memory is coupled to the one or more processors, and the memory is used to store computer program code, where the computer program code includes computer instructions. The one or more processors call the computer instructions to enable the electronic device to execute the method according to any one of claims 1 to 7.

9. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the one or more processors are used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium comprises instructions, which, when executed on an electronic device, enable the electronic device to perform the method according to any one of claims 1 to 7.

Images