An audio generation method and related device

Abstract

The application provides an audio generation method and related equipment suitable for a voice interaction scene. By combining a residual quantization correction flow model to process related features of the text, an acoustic feature after quantization is obtained, and an audio decoder is reused to decode the acoustic feature to obtain an audio signal corresponding to the text. In the training process of the correction flow model, the input data of the correction flow model is obtained according to the data after encoding by an audio encoder (matching the audio decoder). The acoustic feature after quantization has a lower dimension than a commonly used mel spectrum, so the diffusion model has smaller parameter quantity and power requirement than a model for generating a mel spectrum, and can consider reusing the audio decoder for decoding without configuring an additional decoding module, thereby not bringing additional parameter quantity and saving power. The correction flow model can improve inference efficiency compared with a traditional diffusion model. Therefore, the above scheme can significantly improve the efficiency of audio generation and save the power of an electronic device.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an audio generation method and related equipment. Background Technology

[0002] Smart terminal devices are frequently used in voice interaction scenarios, which requires voice assistants to have the ability to generate audio. Currently, a series of speech synthesis methods have been proposed to enable voice assistants to generate audio.

[0003] However, the efficiency of audio generation is currently limited by many factors. One related technique first uses text input to generate a feature model, obtaining a Mel spectrum, and then uses a vocoder to convert the Mel spectrum into a playable time-domain audio signal. These audio generation models, due to their large number of parameters and computational requirements, are generally deployed in the cloud, potentially leading to problems such as privacy leaks and latency. Furthermore, specifically designed vocoders for decoding Mel spectra also require significant computational power due to the high dimensionality of the Mel spectrum and the large number of parameters in the vocoder.

[0004] Therefore, improving the efficiency of audio generation has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides an audio generation method and related equipment, which can significantly improve the efficiency of audio generation and save computing power of electronic devices.

[0006] In a first aspect, an audio generation method is provided for use in an electronic device. The method includes: acquiring a first text to be generated as an audio signal; acquiring semantic features and phoneme length features of the first text; inputting the semantic features, phoneme length features, and a first Gaussian noise obtained by random sampling into a modified flow model to obtain continuous acoustic features, wherein the semantic features and phoneme length features are used as conditional information, and the first Gaussian noise is used as an observation signal; decoding the continuous acoustic features to obtain a time-domain audio signal, the content of which is the same as the content of the text.

[0007] It is understandable that using the modified flow module to process the relevant features of the text, compared with the general diffusion model to generate acoustic features, requires significantly fewer iterations in the generation process, which can reduce computational costs and improve inference speed, thereby improving generation efficiency.

[0008] In one possible embodiment, the electronic device includes an audio encoder and an audio decoder, the encoding formats of the audio encoder and the audio decoder are matched and the audio decoder is the inverse process of the audio encoder; the training data of the modified stream model includes a first sample signal, which is obtained by processing the audio encoder; decoding the continuous acoustic features to obtain a time-domain audio signal includes: decoding the continuous acoustic features using the audio decoder to obtain the audio signal.

[0009] It is understandable that audio encoders and decoders are modules present in almost all electronic devices, while vocoders in related technologies are generally configured on electronic devices with higher audio processing requirements. For example, in related technologies, due to the high dimensionality of the Mel spectrum, vocoders are specifically designed to decode the Mel spectrum, and the number of parameters in a vocoder is generally larger than that in an audio encoder. Since the quantized acoustic features have a lower dimensionality than the Mel spectrum in related technologies, this application considers that it is not necessary to design a dedicated decoding device for the quantized acoustic features, and further considers reusing the existing audio decoder of the electronic device for decoding. On the one hand, compared with the vocoders with a large number of parameters used in related technologies, the audio decoder in the audio encoding and decoding system used in this application does not bring additional parameters to the system, and the audio decoder has a lower computational power requirement than the vocoder, thereby saving the computational power of the electronic device and improving the efficiency of the audio generation method. On the other hand, it does not require the electronic device to be configured with other modules for decoding acoustic features, and can be applied to almost all electronic devices, thus improving the applicability of the audio generation method compared to related technologies.

[0010] In one possible embodiment, semantic features, phoneme length features, and a first Gaussian noise obtained by random sampling are input into a modified flow model to obtain continuous acoustic features. This includes: inputting semantic features, phoneme length features, and the first Gaussian noise into the modified flow model to obtain a discrete code vector corresponding to the codebook. The discrete code vector is represented as the vector in the codebook that is closest to the continuous acoustic features. The discrete code vector is a discretized representation of the continuous acoustic features. The codebook is then queried based on the discrete code vector to obtain the continuous acoustic features.

[0011] It is understandable that the more complex the data that a model needs to generate or process, the larger the number of parameters in the model, and the greater the computational power required for the model to run. In the audio generation method of this application, the acoustic features used to generate the audio signal are quantized acoustic features, which have a lower dimension than the Mel spectrum in related technologies. In other words, the Mel spectrum is more complex than the quantized acoustic features. Therefore, the feature generation models in related technologies have a larger number of parameters and require more computational power than the modified flow model that combines quantization processing. Therefore, the audio generation method provided in this application can save the computational power of electronic devices and improve the efficiency of audio generation methods.

[0012] In one possible embodiment, the codebook includes Q first codebooks, and the discrete code vector includes Q first discrete code vectors. There is a one-to-one correspondence between the Q first codebooks and the Q first discrete code vectors, where Q ≥ 1 and Q is an integer. When Q ≥ 2, the q-th first discrete code vector is represented as the vector in the q-th first codebook that is closest to the continuous acoustic feature, where 1 ≤ q ≤ Q and q is an integer. The continuous acoustic feature is obtained by querying the codebook based on the discrete code vectors, including: querying the vector in the q-th codebook that corresponds to the q-th first discrete code vector, thereby obtaining the vectors in the Q first codebooks that correspond to the Q first discrete code vectors respectively; and summing the vectors corresponding to the Q first discrete code vectors to obtain the continuous acoustic feature.

[0013] It is understandable that the discrete code vectors output by the modified stream model are data after RVQ, which have smaller dimensions and fewer parameters compared to data without RVQ. Therefore, the modified stream model used to generate discrete code vectors also has fewer parameters and requires less computing power compared to the model used to generate data without RVQ.

[0014] In one possible embodiment, obtaining the semantic features and phoneme length features of the first text includes: inputting the first text into a text encoder to obtain a hidden layer representation tensor, the hidden layer representation tensor being used to represent the semantic information of the first text; inputting the hidden layer representation tensor into a length prediction module to predict the phoneme length corresponding to each character in the first text, and obtaining a length vector based on the phoneme length corresponding to each character, wherein the phoneme length is the duration of the phoneme, the phoneme length corresponding to each character is the duration of the phoneme corresponding to each character, and the length vector indicates the phoneme length corresponding to each character; the method further includes: determining the total time corresponding to the first text based on the length vector, the total time including N time frames, N>1 and being an integer, wherein the duration of the audio signal is the same as the total time; randomly sampling N×Q sample points from a Gaussian distribution with a mean of 0 and a variance of 1 to obtain the first Gaussian noise.

[0015] Understandably, N×Q sample points need to be generated to obtain the first Gaussian noise, meaning the data dimension of the first Gaussian noise is N×Q. That is, Q sample points are generated for each time frame, where Q is the number of codebooks; therefore, N time frames generate a total of N×Q sample points. In commonly used codec systems (e.g., with a codec rate of 6kbps), Q is typically 8 or 12. Consequently, the data dimension of the continuous acoustic features output by the corrected streaming model is also N×Q.

[0016] In traditional speech generation methods, Mel spectrum is generally used as acoustic feature data. The data dimension to be generated over N time frames is typically N×80, where 80 represents the dimension of the Mel spectrum. In other words, traditional speech methods use 80 data points per time frame to represent the information of the Mel spectrum.

[0017] Since N×80 is significantly higher than N×Q, the data dimension of the above scheme is significantly smaller than that of traditional speech generation methods. Therefore, compared with the feature generation module used to generate Mel spectrum in traditional speech generation methods, the modified flow model can achieve a smaller number of parameters and a smaller amount of computation.

[0018] In one possible embodiment, semantic features, phoneme length features, and a first Gaussian noise are input into a modified flow model to obtain a discrete code vector corresponding to the codebook. This includes: inputting semantic features, phoneme length features, and the first Gaussian noise into the modified flow model to obtain a first derivative of the first Gaussian noise; performing Euler sampling on the first derivative to obtain a first sampled signal; inputting semantic features, phoneme length features, and the first sampled signal into the modified flow model to obtain a second derivative of the first sampled signal; performing Euler sampling on the second derivative to obtain a second sampled signal; and obtaining a discrete code vector based on the second sampled signal.

[0019] It is understandable that, theoretically, the generated result can be obtained through single-step sampling after using the modified flow network. Therefore, the sampling process is relatively simple. In this case, the effect achieved by using simple or complex sampling methods is almost the same. Therefore, this application prefers a simple sampling method, such as Eulerian sampling. Alternatively, other simple sampling methods can also be used, and this application is not limited to them. Furthermore, when K≥2, the final generation effect can be improved compared to sampling only once.

[0020] In one possible embodiment, the method further includes: inputting a clean audio signal of the sample into an audio encoder to obtain a discrete code vector of the sample; generating a first sample signal based on a randomly generated second Gaussian noise and the discrete code vector of the sample; obtaining a hidden layer representation tensor and a sample length vector based on the sample text; inputting the first sample signal, the hidden layer representation tensor, and the sample length vector into a modified flow model to obtain a sample derivative; calculating a loss function based on the sample derivative and target data, where the target data is the difference between the discrete code vector of the sample and the second Gaussian noise; and updating the parameters of the modified flow model based on the loss function.

[0021] The above scheme uses the difference obtained by subtracting the second Gaussian noise from the discrete code vector of the sample as the training target to realize the training process of the modified flow model, so that the inference side of the modified flow model can improve the audio generation effect and the inference speed compared with the diffusion model.

[0022] In a second aspect, this application provides an electronic device including one or more processors and one or more memories; wherein the one or more memories are coupled to one or more processors, and the one or more memories are used to store computer program code, the computer program code including computer instructions, which, when executed by one or more processors, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0023] Thirdly, embodiments of this application provide a chip system applied to an electronic device. The chip system includes one or more processors, which are used to invoke computer instructions to cause the electronic device to perform the methods described in the first aspect and any possible implementation thereof.

[0024] Fourthly, this application provides a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0025] Fifthly, this application provides a computer program product containing instructions that, when the computer program product is run on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0026] Understandably, the electronic device provided in the second aspect, the chip system provided in the third aspect, the computer storage medium provided in the fourth aspect, and the computer program product provided in the fifth aspect are all used to execute the method provided in this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0027] Figure 1 Examples of voice interaction scenarios provided in embodiments of this application;

[0028] Figure 2 A schematic flowchart illustrating a related technology example;

[0029] Figure 3 A schematic diagram illustrating an example of an audio generation method provided in an embodiment of this application;

[0030] Figure 4 A schematic flowchart of the audio generation method 100 provided in the embodiments of this application;

[0031] Figure 5 A schematic diagram illustrating an example of the discrete code vector generation module 204 based on the modified flow model provided in this application embodiment;

[0032] Figure 6 A schematic diagram illustrating an example of the training process of a modified stream network provided in an embodiment of this application;

[0033] Figure 7 A schematic diagram of the hardware structure of an electronic device 1000 provided in an embodiment of this application;

[0034] Figure 8 A block diagram of a software system for an electronic device 1000 provided in an embodiment of this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0036] To better understand the embodiments of this application, the relevant concepts of residual vector quantization and correction flow involved in this application will be introduced and explained below.

[0037] Audio compression, by approximating continuous features using a discretized method, can significantly reduce the size of audio files. Discretized audio representation (tokens) means that audio can be characterized using discrete sequences of tokens, for example, by employing modeling techniques from natural language processing (such as large language models) to achieve speech synthesis tasks.

[0038] Vector quantization (VQ) is a representative audio compression technique. A raw vector v is processed using a codebook (CB) W, which contains A ordered codebook vectors (or codewords) w1, w2, ..., w... a ,…,w A Where a∈[1,A] and a and A are both integers, the vector v is obtained by using the vector (or codeword) w in the codebook W that is closest to it. a Let Token be used to represent (understandably, there are many methods for evaluating vector similarity, such as Euclidean distance, cosine similarity, etc.), so that vector v can be represented as a discrete codebook vector (in this application, it can also be called the discrete vector corresponding to the codebook, or simply the discrete code vector) Token. Token = [Token0, Token1, ..., Token...] n , ..., Token N-1 ], where N represents the length of the discrete code vector, which is the number of time frames corresponding to vector v. In other words, Token n It is a discretized representation of the continuous features of vector v in the nth time frame.

[0039] Residual vector quantization (RVQ) is another representative audio compression technique. It cleverly uses a "layered description" approach to balance effects and storage space. The VQ process is divided into multiple stages (related to the number of quantizers), each stage (or quantizer) having a corresponding codebook W. q Where q∈[0,Q-1] and both q and Q are integers, Q is the number of stages, which is also the number of codebooks, and W is the number of codebooks. q It includes A ordered codebook vectors w q1 ,w q2 ,…,w qa ,..,w qA , where a∈[1,A] and a and A are both integers. A raw input vector v is first generated by the first-level codebook (CB). 1 The nearest codebook vector w in ) 1a Then calculate v and w. 1a The residual R1 between the two layers is input into the next layer quantizer based on the second layer codebook (CB). 2 To achieve a more precise representation, and so on, the Q codebook vectors from different codebooks that are closest to vector v will be found among the Q codebooks. In this application, the discrete code vector of the nth time frame... in, The acoustic feature of the nth time frame is the codebook vector closest to the codebook in the qth codebook. Finally, the vector v is quantized into the sum of the Q closest codebook vectors. Feature, referred to in this application as the continuous acoustic feature of the compressed audio.

[0040] In this application, Feature = [Feature0, Feature1, ..., Feature...] n , ..., Feature N-1 ], where N is the number of time frames corresponding to the original vector v. 0 ≤ n ≤ N and both n and N are integers. That is to say, Feature n Let be the acoustic features of the audio signal corresponding to the text at the nth time frame. Among them, codebook q This represents the q-th codebook among the Q codebooks mentioned above. This means querying and retrieving the qth codebook. Each codebook vector; This indicates that the codebooks will be based on... The Q codebook vectors retrieved are summed.

[0041] Diffusion models are a general generative modeling technique used to create high-dimensional, perceptual data such as images and videos. By simulating the physical diffusion process, data is progressively transformed into noise, and then the inverse process is learned to gradually recover the original data from the noise, achieving high-quality generation results. The generation process requires multiple iterations, resulting in high computational costs and slow inference speeds. Significant computational resources are needed during training and generation, especially when generating high-resolution images or high-quality audio.

[0042] This application also relates to Rectified Flow. Rectified Flow is an emerging generative model that constructs a flow-based generative model through linear interpolation and then uses flow operations to draw the trajectory of a straight line. It connects data and noise with a straight line, offering better theoretical properties and conceptual simplicity compared to diffusion models. For example, it requires significantly fewer iterations during the generation process; theoretically, one iteration (differentiating the line) is sufficient to obtain the generated result, reducing computational costs and increasing inference speed, thereby improving generation efficiency.

[0043] Smart terminal devices are often used in voice interaction scenarios, which requires voice assistants to provide the ability to generate audio. Figure 1 Examples of voice interaction scenarios provided in embodiments of this application. For example... Figure 1 As shown, the electronic device detects the user saying, "Hello YOYO, are there any good movies lately?", and displays the text matching the user's voice, "Are there any good movies lately?", on the smart assistant panel of interface 1. Then, interface 2 is displayed, and the audio "I found these great videos, which one do you want to choose?" is played to the user. The smart assistant panel of interface 2 displays, in a dialogue format, the text matching the user's voice, the text matching the audio, "I found these great videos, which one do you want to choose?", and search results based on the user's voice.

[0044] Currently, a series of speech synthesis methods have been proposed to enable voice assistants to generate audio. Figure 2 This is a schematic flowchart illustrating a related technology example. Figure 2 As shown, the text (e.g., "Found these amazing videos, which one to choose?") is first input into a feature generation model to obtain acoustic features (e.g., Mel spectrum). Then, a vocoder is used to convert the Mel spectrum into a playable time-domain audio signal. These audio generation models, due to their large number of parameters and computational requirements, are generally deployed in the cloud, which can potentially lead to problems such as privacy leaks and latency. Furthermore, a dedicated vocoder is designed to decode the Mel spectrum; because the Mel spectrum has high dimensionality, the vocoder has a large number of parameters, resulting in significant computational demands.

[0045] Therefore, improving the efficiency of audio generation is an urgent problem to be solved. In view of this, this application provides an audio generation method that can significantly improve the efficiency of audio generation methods.

[0046] Figure 3 This is a schematic diagram illustrating an example of an audio generation method provided in an embodiment of this application. For example... Figure 3 As shown, the text to be generated as an audio signal is first acquired. The relevant features of the text are then processed using a modified flow model that incorporates quantization, resulting in quantized acoustic features. These quantized acoustic features are then decoded using an electronic device's audio decoder to obtain a playable time-domain audio signal. During the training of the modified flow model, the input data is obtained from the data encoded by the audio encoder. The encoding formats of the audio encoder and decoder are matched, and the audio decoder is the inverse process of the audio encoder.

[0047] Understandably, the more complex the data a model generates or processes, the larger the number of model parameters, and the greater the computational power required for model operation. In the audio generation method of this application, the acoustic features used to generate the audio signal are quantized acoustic features, which have a lower dimension than the Mel spectrum in related technologies. In other words, the Mel spectrum is more complex than the quantized acoustic features. Therefore, the modified flow model combined with quantization processing has fewer parameters and requires less computational power compared to the feature generation models in related technologies. Therefore, the audio generation method provided by this application can save the computational power of electronic devices and improve the efficiency of audio generation methods.

[0048] It is also understandable that audio encoders and decoders are modules present in almost all electronic devices, while vocoders in related technologies are generally configured on electronic devices with higher demands for audio processing. For example, in related technologies, due to the high dimensionality of the Mel spectrum, vocoders are specifically designed to decode the Mel spectrum, and the number of parameters in a vocoder is generally larger than that in an audio encoder. Since the quantized acoustic features have a lower dimensionality than the Mel spectrum in related technologies, this application considers that it is not necessary to design a dedicated decoding device for the quantized acoustic features, and further considers reusing the existing audio decoder of the electronic device for decoding. On the one hand, compared with the vocoders with a large number of parameters used in related technologies, the audio decoder in the audio encoding and decoding system used in this application does not bring additional parameters to the system, and the audio decoder has a lower computational power requirement than the vocoder, thereby saving the computational power of the electronic device and improving the efficiency of the audio generation method. On the other hand, it does not require the electronic device to be configured with other modules for decoding acoustic features, and it can be applied to almost all electronic devices, thus improving the applicability of the audio generation method compared to related technologies.

[0049] It is also understandable that using the modified flow module to process the relevant features of the text, compared with the general diffusion model to generate acoustic features, requires significantly fewer iterations in the generation process, which can reduce computational costs and improve inference speed, thereby improving generation efficiency.

[0050] In summary, the audio generation method provided in this application can significantly improve the efficiency of audio generation and save computing power of electronic devices.

[0051] The following will combine Figures 4 to 7 This application provides a detailed description of the audio generation method.

[0052] Figure 4 This is a schematic flowchart of an audio generation method 100 provided in an embodiment of this application.

[0053] S101, Obtain the first text of the audio signal to be generated.

[0054] For example, the first text may be uploaded by a user, downloaded from the network, or generated by a neural network model configured on an electronic device. For instance, Figure 1 In the scenario shown, the first text, "These amazing videos have been found, which one should I choose?", is generated by the neural network model upon which the voice assistant application relies based on the user's voice.

[0055] The following example uses text as the first text and the text acquisition module 201 executing S101 as an example for illustration.

[0056] Text acquisition module 201 acquires the text used for audio synthesis: text = [text0, text1, ..., text...]. p ,…text P-1 ], where P represents the length of the text. The length of the text can be understood as the number of characters included in the text. The text acquisition module 201 transmits the text to the text encoder 202.

[0057] S102, Obtain the semantic features and phoneme length features of the first text.

[0058] The hidden layer representation tensor will be discussed below. text As a possible example of semantic features, the length vector L pre This is an example of a possible phoneme length feature, which will be explained below.

[0059] Text encoder 202 encodes text using a pre-trained model to obtain the text's hidden layer representation tensor. text Among them, the hidden layer representation tensor is... textIt can be used to represent the semantic information of text. The text encoder 202 represents the hidden layer tensor. text The data is transmitted to the length prediction module 203 and the discrete code vector generation module 204 based on the modified flow model.

[0060] Length prediction module 203 uses a pre-trained model based on the hidden layer representation tensor. text Predict the phoneme length of each character in the text and output a length vector L consisting of the phoneme lengths of all characters. pre .

[0061] It is understandable that a character is the smallest distinguishable unit in a writing system; a phoneme is the smallest unit in speech. If speech is to be generated from text, then one character can correspond to one or more phonemes. Phoneme length can be understood as the duration of a phoneme (e.g., expressed as the number of time frames or spectrum frames). The phoneme length corresponding to each character can be understood as the duration of the phoneme corresponding to that character (e.g., expressed as the number of time frames or spectrum frames).

[0062] For example, the length vector L pre This includes P numbers arranged in character order. These P numbers represent the phoneme lengths corresponding to the P characters in the text. In other words, the length vector L... pre Indicates the phoneme length corresponding to each character in the text.

[0063] Length prediction module 203 will predict the length vector L pre Transmitted to the discrete code vector generation module 204 based on the modified flow model.

[0064] S103 inputs the semantic features, phoneme length features, and the first Gaussian noise obtained by random sampling into the modified flow model to obtain continuous acoustic features.

[0065] For example, S103 can be implemented by a discrete code vector generation module 204 based on a modified flow model and a codebook fusion and query module 205.

[0066] The discrete code vector generation module 204 based on the modified flow model generates the hidden layer representation tensor. text and length vector L pre As conditional information, the discrete vector corresponding to the codebook is generated from randomly sampled white noise (i.e., Gaussian noise) samples using the modified flow model, namely the discrete code vector Token = [Token0, Token1, ..., Token2]. n , ..., Token N-1 ]. Where N represents the length of the discrete code vector.

[0067] It is understandable that the discrete code vector generation module 204 based on the modified flow model can generate code vectors according to the length vector L. pre Determining the sum of the phoneme lengths of all characters means determining the total number of time frames of the audio signal corresponding to the text (i.e., the final audio signal y(t) generated by this method, which will be explained uniformly here and will not be elaborated further below). The length vector L... pre Adding all the values ​​together gives the total number of time frames, N. In other words, the audio signal corresponding to the text lasts for N time frames. Since the discrete code vector generation module 204 based on the modified flow model outputs a corresponding discrete code vector for each time frame, the length of the discrete code vector Token is N.

[0068] In the quantization process, Q codebooks are used, where Q ≥ 2 and are integers. Therefore, for each time frame, discrete code vectors are output for each of the Q codebooks. in, Let q be the closest codebook vector (or simply code vector) among the Q codebooks containing the acoustic features of the nth time frame, where q∈[0,Q-1] and q is an integer.

[0069] The discrete code vector generation module 204 based on the modified flow model transmits the discrete code vector Token to the codebook fusion and query module 205.

[0070] Specifically, the following will combine Figure 5 This paper details the possible implementations of the discrete code vector generation module 204 based on the modified flow model.

[0071] It is understandable that using the modified flow module to process the relevant features of the text, compared with the general diffusion model to generate acoustic features, requires significantly fewer iterations in the generation process, which can reduce computational costs and improve inference speed, thereby improving generation efficiency.

[0072] The codebook fusion and query module 205 queries the discrete code vector Token to obtain the continuous acoustic feature Feature = [Feature0, Feature1, ..., Feature...]. n ...Feature N-1 Among them, Feature n Let be the acoustic features of the audio signal corresponding to the text at the nth time frame. Among them, codebook q This represents the q-th codebook among the Q codebooks mentioned above. This indicates querying and retrieving the q-th residual codebook from the audio codec module. One vector; This indicates that, based on Q codebooks respectively The Q codebook vectors retrieved are summed.

[0073] The codebook fusion and query module 205 transmits the continuous acoustic feature to the audio decoder 206.

[0074] It is understandable that the discrete code vectors output by the modified stream model are data after RVQ, which have smaller dimensions and fewer parameters compared to data without RVQ. Therefore, the modified stream model used to generate discrete code vectors also has fewer parameters and requires less computing power compared to the model used to generate data without RVQ.

[0075] S104 decodes the continuous acoustic features to obtain the audio signal in the time domain.

[0076] The audio decoder 206 decodes the continuous acoustic features to obtain the audio signal y(t)=[y(0),y(1),…,y(T-1)], where T represents the total length of the audio sampling points.

[0077] It is understandable that electronic devices include audio encoders and audio decoders, with matching encoding formats and the audio decoder being the inverse process of the audio encoder. The training data for the modified streaming model includes a first sample signal, which is obtained by processing the audio encoder. In other words, the training process of the modified streaming model uses the audio signal processed by the audio encoder as its training data, so that the data output by the modified streaming model during inference can be decoded by the audio decoder.

[0078] It is also understandable that audio encoders and decoders are modules present in almost all electronic devices, while vocoders in related technologies are generally configured on electronic devices with higher demands for audio processing. For example, in related technologies, due to the high dimensionality of the Mel spectrum, vocoders are specifically designed to decode the Mel spectrum, and the number of parameters in a vocoder is generally larger than that in an audio encoder. Since the quantized acoustic features have a lower dimensionality than the Mel spectrum in related technologies, this application considers that it is not necessary to design a dedicated decoding device for the quantized acoustic features, and further considers reusing the existing audio decoder of the electronic device for decoding. On the one hand, compared with the vocoders with a large number of parameters used in related technologies, the audio decoder in the audio encoding and decoding system used in this application does not bring additional parameters to the system, and the audio decoder has a lower computational power requirement than the vocoder, thereby saving the computational power of the electronic device and improving the efficiency of the audio generation method. On the other hand, it does not require the electronic device to be configured with other modules for decoding acoustic features, and it can be applied to almost all electronic devices, thus improving the applicability of the audio generation method compared to related technologies.

[0079] The audio decoder 206 transmits the audio signal y(t) to the audio signal output module 207.

[0080] The audio signal output module 207 outputs the generated audio signal y(t) to the downstream module or speaker for playback.

[0081] The above scheme, by combining a modified flow model for quantization processing to process relevant features from the text to obtain quantized acoustic features, and by reusing existing audio decoders to decode the quantized acoustic features, can significantly improve the efficiency of audio generation and save computing power of electronic devices.

[0082] Figure 5 This is a schematic diagram of an example of a discrete code vector generation module 204 based on a modified flow model provided in an embodiment of this application.

[0083] It is understandable that a total of K sampling steps are performed, where K is a pre-set number of sampling steps and is an integer greater than 1, preferably K ≥ 2, for example, K = 3. Although theoretically the modified flow can obtain the generated result in one iteration, this application prefers K ≥ 2 to optimize the generation effect.

[0084] First execution Figure 5 In the process shown, k = K. Where 1 ≤ k ≤ K and both k and K are integers.

[0085] like Figure 5 As shown, the length vector L pre and hidden layer representation tensor text Referring to the descriptions above, it is transmitted to the modified stream network inference module 303.

[0086] The Gaussian noise sampling module 301 randomly and independently samples N×Q sample points from a Gaussian distribution with zero mean and variance of 1. Here, N and Q are described above. That is, it randomly and independently samples from the Gaussian distribution with zero mean and variance of 1 on a time-frame, code-by-code basis. The sampled Gaussian noise (i.e., the first Gaussian noise mentioned above) is defined as x. K Or, defined as x when k = K k .

[0087] Gaussian noise sampling module 301 will sample Gaussian noise x K Transmitted to the corrected stream network inference module 303.

[0088] Sampling time update module 302, according to the formula Generate the sampling time for the k-th step, where t min It can be 0.003, t maxThe value can be 1, which indicates the initial execution. Figure 5 The process shown is such that k = K, therefore t k =0.

[0089] The sampling time update module 302 will update t k Transmitted to the corrected stream network inference module 303.

[0090] The input data for the modified stream network inference module 303 includes x k t k Length vector L pre and hidden layer representation tensor text , where x k and t k The length vector L serves as the observation or reference signal for the corrected flow network inference module 303. pre and hidden layer representation tensor text As conditional information, the output data of the corrected stream network inference module 303 is x. k For t k The derivative v k .

[0091] For example, the modified flow network inference module 303 can be implemented based on one or a combination of network models such as Noise Conditional Scoring Network (NCSN)++, Convolutional Neural Network (CNN), Convolutional Recurrent Neural Network (CRNN), U-shaped Neural Network Pooled Subgraph Structure (U-Net).

[0092] Specifically, the training process of the modified flow network will be discussed below. Figure 6 A detailed introduction will be provided.

[0093] The modified stream network inference module 303 will convert the derivative v k Transmitted to Euler sampling module 304.

[0094] Euler sampling module 304, through formula Perform sampling and update the sampling count k = k-1.

[0095] It is understandable that, theoretically, the generated result can be obtained through single-step sampling after using the modified flow network. Therefore, the sampling process is relatively simple. In this case, the effect achieved by using simple or complex sampling methods is almost the same. Therefore, this application prefers a simple sampling method, such as Eulerian sampling. Alternatively, other simple sampling methods can also be used, and this application is not limited to any particular method.

[0096] Module 305 determines: Has the preset number of sampling steps been reached?

[0097] For example, determining whether the preset number of sampling steps has been reached can be done by checking if the number of sampling steps, k, is equal to 0. If k = 0, it is determined that the preset number of sampling steps has been reached, and then x... k+1 As a discrete code vector Token output; if k≠0, then for x k Sampling continues, meaning that the sampling time update module 302 continues to generate the updated t based on the updated k (updated by the Euler sampling module 304). k And will update x k and t k As an observation signal or reference signal, and the length vector L pre and hidden layer representation tensor text As conditional information input to the modified flow network inference module 303, execution continues. Figure 5 The process is shown below.

[0098] In other words, when K≥2 and k>1, starting from the second generation of the sampling time by the sampling time update module 302 (including the second generation), in each subsequent sampling, x will not be changed. K Instead of inputting the corrected flow network inference module 303, it uses the x obtained after the previous sampling. k Input correction flow network inference module 303.

[0099] It is understandable that sampling the data output by the modified flow model multiple times can improve the final generation effect.

[0100] It is understandable that the Gaussian noise sampling module 301 needs to generate N×Q sample points, that is, the data dimension is N×Q. In other words, Q sample points are generated for each time frame, where Q is the number of codebooks; therefore, N time frames generate a total of N×Q sample points. In commonly used encoding / decoding systems (e.g., with a codec rate of 6kbps), Q is typically 8 or 12. Thus, the derivative v output by the corrected stream network inference module 303... k The data dimension is also N×Q, and the data dimension of the discrete code vector finally output by the discrete code vector generation module 204 based on the modified flow model is also N×Q.

[0101] In traditional speech generation methods, the Mel spectrum is used as acoustic feature data. The data dimension to be generated over N time frames is typically N×80, where 80 represents the dimension of the Mel spectrum. In other words, traditional speech methods use 80 data points per time frame to represent the information of the Mel spectrum.

[0102] Since N×80 is significantly higher than N×Q, the data dimension of the above scheme is significantly smaller than that of traditional speech generation methods. Therefore, the discrete code vector generation module 204 based on the modified flow model can achieve a smaller number of parameters and a smaller amount of computation compared to the feature generation module in traditional speech generation methods.

[0103] It is understandable that, when k = K, the derivative v k As an example of the first derivative mentioned above, x obtained by Euler sampling k-1 This is an example of the first sampled signal described above. When k < K, the derivative v k As an example of the second derivative mentioned above, x obtained by Euler sampling k-1 This is an example of the second sampled signal described above.

[0104] Figure 6 This is a schematic diagram illustrating an example of the training process of a modified stream network provided in an embodiment of this application.

[0105] The clean speech acquisition module 401 is used to acquire sample clean audio signals Yx(t) from a clean sample audio dataset.

[0106] The clean audio module 401 transmits the sample clean audio signal Yx(t) to the audio encoder 403.

[0107] The text acquisition module 402 acquires the sample text Ytext corresponding to the clean audio signal of the sample.

[0108] For example, if the sample text dataset includes sample texts that can form audio-text data pairs with the sample clean audio signals, then the sample text Ytext corresponding to the sample clean audio signals is obtained from the sample text dataset.

[0109] For example, if the sample text dataset does not include sample text corresponding to the clean audio signal, an automatic speech recognition system can be used to identify the text corresponding to the clean audio signal and use it as the sample text Ytext.

[0110] The text acquisition module 402 transmits the sample text Ytext to the text encoder 408.

[0111] The audio encoder 403 encodes the clean audio signal of the sample to obtain the sample discrete code vector YToken.

[0112] YToken=[YToken0, YToken1,…, YToken m ..., YToken M-1 ].

[0113] It is understandable that audio encoders and audio decoders are two sub-modules in the audio encoding and decoding module of an electronic device that appear in pairs, have matching encoding formats, and perform inverse operations.

[0114] The audio encoder 403 transmits the sample discrete code vector YToken to the sample generation module 406.

[0115] Sampling time generation module 404, this module is used to randomly generate a Yt min To Yt max The decimal between these two values ​​is taken as the sample sampling time Yt, where Yt min For the minimum value, such as 0.003, Yt max It is the maximum value, such as 1.

[0116] The sampling time generation module 404 transmits the sample sampling time Yt to the Gaussian noise generation module 405 and the corrected flow network 407.

[0117] Gaussian noise generation module 405 generates sample Gaussian noise Ynoise with zero mean and standard deviation σ(Yt) based on time t. The standard deviation can be calculated in various ways, such as...

[0118] The Gaussian noise generation module 405 outputs the generated Gaussian noise Ynoise to the sample generation module 406.

[0119] The sample generation module 406 adds the sample Gaussian noise Ynoise and the sample discrete code vector YToken to obtain the noisy sample Ynoisy_Token.

[0120] The sample generation module 406 transmits the noisy sample Ynoisy_Token to the corrected stream network 407.

[0121] Text encoder 408 encodes the sample text Ytext using a pre-trained model to obtain the sample hidden layer representation tensor Ylatent of the sample text Ytext. text Among them, the sample hidden layer representation tensor Ylatent text It can be used to characterize the semantic information of sample text Ytext.

[0122] The text encoder 408 represents the sample hidden layer tensor Ylatent. text The data is transmitted to the length prediction module 409, and to the modified flow network 407.

[0123] Length prediction module 409 uses a pre-trained model to predict the Ylatent tensor based on the hidden layer representation of the samples. textPredict the phoneme length of each character in the sample text Ytext, and output the sample length vector YL composed of the phoneme lengths of all characters. pre .

[0124] For an understanding of phoneme length and sample length vector, please refer to the description of the length vector of phoneme length above.

[0125] The length prediction module 409 will generate the sample length vector YL pre Transmitted to the correction stream network 407.

[0126] Modify the flow network 407 to represent the sample hidden layer tensor Ylatent text and sample length vector YL pre As conditional information, the sample derivative Yv is generated based on the noisy sample Ynoisy_Token and the sample time Yt. The sample derivative Yv is the corrected stream.

[0127] For example, the modified flow network 407 can be implemented based on one or a combination of network models such as NCSN, CNN, CRNN, U-Net, etc.

[0128] The modified flow network 407 transmits the sample derivative Yv to the loss function calculation module 410.

[0129] The loss function calculation module 410 calculates the loss function corresponding to the sample derivative Yv: loss = mse(Yv, YToken - Ynoise), where the mean squared error (mse) is a commonly used loss function in regression tasks. It is defined as the average of the squared differences between the predicted and true values. YToken - Ynoise represents the target data, i.e., the training objective.

[0130] The loss function calculation module 410 transmits the loss function loss to the network parameter update module 411.

[0131] The network parameter update module 411 updates the parameters of the corrected flow network 407 according to the loss function loss.

[0132] The above scheme uses YToken-Ynoise as the training target to implement the training process of the modified flow model, so that the inference side of the modified flow model can improve the audio generation effect and the inference speed compared with the diffusion model.

[0133] Figure 7 This is a schematic diagram of the hardware structure of an electronic device 1000 provided in an embodiment of this application. See also... Figure 7The electronic device 1000 may include a processor 1010, an external memory interface 1020, an internal memory 1021, a universal serial bus (USB) interface 1030, a charging management module 1040, a power management module 1041, a battery 1042, an antenna 1, an antenna 2, a mobile communication module 1050, a wireless communication module 1060, an audio module 1070, a speaker 1070A, a receiver 1070B, a microphone 1070C, a headphone jack 1070D, a sensor module 1080, buttons 1090, a motor 1091, an indicator 1092, a camera 1093, a display screen 1094, and a subscriber identification module (SIM) card interface 1095, etc. The sensor module 1080 may include a pressure sensor 1080A, a gyroscope sensor 1080B, a barometric pressure sensor 1080C, a magnetic sensor 1080D, an accelerometer sensor 1080E, a distance sensor 1080F, a proximity sensor 1080G, a fingerprint sensor 1080H, a temperature sensor 1080J, a touch sensor 1080K, an ambient light sensor 1080L, a bone conduction sensor 1080M, etc.

[0134] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 1000. In other embodiments of this application, the electronic device 1000 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0135] The processor 1010 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU).

[0136] The controller can be the nerve center and command center of the electronic device 1000. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0137] The processor 1010 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 1010 is a cache memory. This memory can store instructions or data that the processor 1010 has just used or that are used repeatedly. If the processor 1010 needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces the waiting time of the processor 1010, and thus improves the efficiency of the system.

[0138] In some embodiments, the processor 1010 may include one or more interfaces, such as an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0139] The MIPI interface can be used to connect the processor 1010 to peripheral devices such as the display screen 1094 and the camera 1093. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 1010 and the camera 1093 communicate via the CSI interface to enable the electronic device 1000 to perform its shooting function. The processor 1010 and the display screen 1094 communicate via the DSI interface to enable the electronic device 1000 to perform its display function.

[0140] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 1000. In other embodiments of this application, the electronic device 1000 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0141] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 1000 is selecting a frequency, the DSP is used to perform Fourier transforms on the frequency energy.

[0142] Video codecs are used to compress or decompress digital video. Electronic device 1000 can support one or more video codecs. Thus, electronic device 1000 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0143] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs can enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0144] Electronic device 1000 can implement audio functions, such as music playback and recording, through audio module 1070, speaker 1070A, receiver 1070B, microphone 1070C, headphone jack 1070D, and application processor. Buttons 1090 include a power button and volume buttons. Buttons 1090 can be mechanical buttons or touch buttons. Electronic device 1000 can receive button input and generate key signal inputs related to user settings and function control of electronic device 1000.

[0145] The software system of electronic device 1000 will be described next.

[0146] For example, electronic device 1000 may be a mobile phone. The software system of electronic device 1000 may adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment takes the layered architecture Android system as an example to illustrate the software system of electronic device 1000.

[0147] Figure 8 A block diagram of a software system for an electronic device 1000 provided in an embodiment of this application. See also... Figure 8A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: the application layer, the application framework layer, the hardware abstraction layer (HAL), the driver layer, and the hardware layer.

[0148] The application layer may include a series of application packages, such as dialer applications, gallery applications, etc. (not shown in the figure). In this embodiment, the application packages may include applications such as video, music, and voice assistants, all of which play audio to the user. In particular, voice assistant applications allow the terminal device to generate audio based on text and play the generated audio signal to the user.

[0149] Alternatively, the application layer may also include other applications that need to play audio signals generated from the text to the user, which is not limited in this application.

[0150] The framework layer provides an application programming interface (API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. In this embodiment, the framework layer includes an audio generation service interface. This audio generation service interface provides an API and a programming framework for applications that obtain audio generation services.

[0151] The Hardware Abstraction Layer (HAL) is an interface layer located between the operating system kernel and upper-layer software, providing a virtual hardware platform for the operating system. In this embodiment, the HAL may include an audio generation algorithm. The audio generation algorithm may include runtime code and data that implement the audio generation method provided in this embodiment.

[0152] The driver layer is the layer between hardware and software. It includes drivers for various hardware components. The driver layer can include drivers for audio digital signal processors (DSPs), such as those for audio DSPs. Audio DSP drivers are used to drive audio DSPs to process audio digital signals.

[0153] The hardware layer includes an audio digital signal processor. The audio digital signal processor can be used to process audio digital signals. It should be noted that the application provided... Figure 8 The software architecture diagram of the electronic device shown is only an example and does not limit the specific module division in different layers of the Android operating system. For details, please refer to the introduction of the Android operating system software architecture in conventional technology.

[0154] The method in the embodiments of this application will be described in detail below with reference to the above hardware structure and system structure:

[0155] In response to applications that enable video, music, or voice assistants, these applications can call the audio generation service interface to obtain the application programming interface and programming framework provided by the audio generation service.

[0156] The audio generation service initializes the audio generation algorithm. The audio generation algorithm acquires the text used to generate the audio signal, generates continuous acoustic features according to the stored audio generation method, and processes the continuous acoustic features using an audio signal processor to obtain an output time-domain audio signal.

[0157] For details on how to process the audio signal, please refer to the previous section. Figures 4 to 6 The flowchart of the method is shown.

[0158] Finally, the audio generation algorithm can send the generated audio signal back to the audio generation service, and then back to the application layer.

[0159] This application provides a chip system including one or more processors for calling and executing instructions stored in memory, thereby performing the methods described in this application. The chip system may be composed of chips or may include chips and other discrete devices.

[0160] The chip system may include input circuits or interfaces for transmitting information or data, and output circuits or interfaces for receiving information or data.

[0161] This application also provides a computer program product that, when executed by a processor, implements the methods described in any of the method embodiments of this application.

[0162] The computer program product can be stored in memory and, after processes such as preprocessing, compilation, assembly, and linking, is finally converted into an executable object file that can be executed by a processor.

[0163] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer, implements the methods described in any of the method embodiments of this application. The computer program may be a high-level language program or an executable object program.

[0164] The computer-readable storage medium can be volatile memory or non-volatile memory, or it can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0165] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process and technical effects of the above-described apparatus and equipment can be referred to the corresponding processes and technical effects in the foregoing method embodiments, and will not be repeated here.

[0166] In the several embodiments provided in this application, the systems, apparatuses, and methods disclosed can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not performed. The apparatus embodiments described above are merely illustrative; the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Multiple units or components can be combined or integrated into another system. Furthermore, the coupling between units or components can be direct coupling or indirect coupling, including electrical, mechanical, or other forms of connection.

[0167] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0168] It should be understood that "multiple" as used in this application refers to two or more. The term "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Furthermore, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0169] The terms (or numbers) "first," "second," etc., appearing in the embodiments of this application are for descriptive purposes only, that is, only to distinguish different objects, such as different "coordinates," etc., and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first," "second," etc., may explicitly or implicitly include one or more features. In the description of the embodiments of this application, "at least one (item)" refers to one or more. "Multiple" means two or more. "At least one (item) below" or similar expressions refer to any combination of these items, including any combination of a single (item) or a plurality of (items).

[0170] In summary, the above description is merely a preferred embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An audio generation method, applied to electronic devices, characterized in that, The method includes: Obtain the first text of the audio signal to be generated; Obtain the semantic features and phoneme length features of the first text; The semantic features, the phoneme length features, and the first Gaussian noise obtained by random sampling are input into the modified flow model to obtain the discrete code vector corresponding to the codebook. The codebook is queried according to the discrete code vector, and the queried codebook vectors are fused to obtain continuous acoustic features. The discrete code vector is the vector in the codebook that is closest to the continuous acoustic features. The discrete code vector is the discretized representation of the continuous acoustic features. The semantic features and the phoneme length features are used as conditional information, and the first Gaussian noise is used as the observation signal. The continuous acoustic features are decoded using an audio decoder to obtain a time-domain audio signal, the content of which is the same as the content of the text. The audio decoder and the audio encoder in the electronic device have the same encoding format and are the inverse of each other. The sample signals in the training data of the modified stream model are the quantized features of the sample audio signals after being processed by the audio encoder.

2. The method as described in claim 1, characterized in that, The codebook includes Q first codebooks, and the discrete code vector includes Q first discrete code vectors. The Q first codebooks and the Q first discrete code vectors correspond one-to-one, and Q≥1 and Q is an integer. When Q≥2, the q-th first discrete code vector is represented as the vector in the q-th first codebook that is closest to the continuous acoustic feature, 1≤q≤Q and q is an integer. The step of querying the codebook based on the discrete code vector to obtain the continuous acoustic feature includes: Query the vector in the qth codebook that corresponds to the qth first discrete code vector, thereby obtaining the vectors in the Q first codebooks that correspond to the Q first discrete code vectors respectively; The continuous acoustic features are obtained by summing the vectors corresponding to the Q first discrete code vectors.

3. The method as described in claim 2, characterized in that, The acquisition of semantic features and phoneme length features of the first text includes: The first text is input into a text encoder to obtain a hidden layer representation tensor, which is used to represent the semantic information of the first text. The hidden layer representation tensor is input into the length prediction module to predict the phoneme length corresponding to each character in the first text, and a length vector is obtained based on the phoneme length corresponding to each character. The phoneme length is the duration of the phoneme, the phoneme length corresponding to each character is the duration of the phoneme corresponding to each character, and the length vector indicates the phoneme length corresponding to each character. The method further includes: Based on the length vector, the total time corresponding to the first text is determined. The total time includes N time frames, where N > 1 and is an integer. The duration of the audio signal is the same as the total time. The first Gaussian noise is obtained by randomly sampling N×Q sample points from a Gaussian distribution with a mean of 0 and a variance of 1.

4. The method as described in claim 1, characterized in that, The step of inputting the semantic features, the phoneme length features, and the first Gaussian noise into the modified flow model to obtain the discrete code vector corresponding to the codebook includes: The semantic features, the phoneme length features, and the first Gaussian noise are input into the modified flow model to obtain the first derivative of the first Gaussian noise. Euler sampling is performed on the first derivative to obtain the first sampled signal; The semantic features, the phoneme length features, and the first sampled signal are input into the modified flow model to obtain the second derivative of the first sampled signal. Euler sampling is performed on the second derivative to obtain the second sampled signal; The discrete code vector is obtained based on the second sampled signal.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: The clean audio signal of the sample is input into the audio encoder to obtain the discrete code vector of the sample; A first sample signal is generated based on the randomly generated second Gaussian noise and the sample discrete code vector; The hidden layer representation tensor and sample length vector are obtained from the sample text. The first sample signal, the sample hidden layer representation tensor, and the sample length vector are input into the modified flow model to obtain the sample derivative. The loss function is calculated based on the sample derivative and the target data, wherein the target data is the difference obtained by subtracting the second Gaussian noise from the sample discrete code vector; The parameters of the corrected flow model are updated based on the loss function.

6. An electronic device, characterized in that, The electronic device includes: one or more processors, and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 5.

7. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 5.

Images