Method, device, electronic device and storage medium for generating audio data

Abstract

The present disclosure relates to a method, apparatus, electronic device and storage medium for generating audio data. A method for generating audio data is provided. In a generator for converting text feature data to audio data, in response to receiving the text feature data, an inverse convolution operation is performed on the text feature data to determine inverse convolution data of the text feature data. The inverse convolution data is enhanced to generate enhanced inverse convolution data by using up-sampling data associated with the text feature data. The enhanced inverse convolution data is processed by using a shuffle network model to generate audio data corresponding to the text feature data. In this way, an effective balance between reducing the computational resource overhead and ensuring the signal processing accuracy can be achieved, thereby improving the efficiency of the audio data generation.

Description

[0001] Implementations of this disclosure relate to audio generation, and more specifically, to methods, apparatus, electronic devices, and storage media for converting text feature data into audio data. Background Technology

[0002] In the field of speech synthesis technology, various text-to-speech conversion schemes have been developed. For example, vocoders can convert the features of text into speech signals that humans can recognize. Neural network technology has been widely applied in speech synthesis. For instance, vocoders based on various neural network structures have been proposed, striving to improve real-time performance while maintaining sound quality. However, existing solutions struggle to simultaneously achieve both sound quality and real-time performance, making it difficult to perform real-time speech generation on terminal devices with limited computing resources (e.g., mobile terminals). Therefore, how to reduce the computational resources required for speech generation while maintaining sound quality has become a research hotspot. Summary of the Invention

[0003] The goal is to develop and implement a more efficient technology for generating audio data. This technology should be compatible with existing application environments and reduce the computational resources required for speech generation while maintaining sound quality.

[0004] According to a first aspect of this disclosure, a method for generating audio data is provided. In a generator for converting text feature data into audio data, in response to receiving the text feature data, a deconvolution operation is performed on the text feature data to determine deconvolution data of the text feature data. The deconvolution data is enhanced using upsampled data associated with the text feature data to generate enhanced deconvolution data. The enhanced deconvolution data is processed using a ShuffleNet model to generate audio data corresponding to the text feature data.

[0005] According to a second aspect of this disclosure, an apparatus for generating audio data is provided, comprising a generator for converting text feature data into audio data, the generator comprising: a deconvolution module configured to perform a deconvolution operation on the text feature data in response to receiving the text feature data to determine deconvolution data of the text feature data; an enhancement module configured to enhance the deconvolution data using upsampled data associated with the text feature data to generate enhanced deconvolution data; and a generation module configured to process the enhanced deconvolution data using a shuffling network model to generate audio data corresponding to the text feature data.

[0006] According to a third aspect of this disclosure, an electronic device is provided, comprising: a memory and a processor; wherein the memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to a first aspect of this disclosure.

[0007] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided that stores one or more computer instructions thereon, wherein the one or more computer instructions are executed by a processor to implement the method according to a first aspect of this disclosure.

[0008] According to a fifth aspect of this disclosure, a computer program product is provided, including a computer program / instructions, wherein when the computer program / instructions are executed by a processor, they implement the method according to a first aspect of this disclosure. Attached Figure Description

[0009] The features, advantages, and other aspects of various implementations of this disclosure will become more apparent from the accompanying drawings and the following detailed description, which illustrate, by way of example and not limitation, several implementations of this disclosure. In the drawings:

[0010] Figure 1 This schematically illustrates a block diagram of a generator in which a technical solution according to an exemplary implementation of the present disclosure can be used;

[0011] Figure 2 A block diagram of a generation unit according to an exemplary implementation of the present disclosure is shown schematically;

[0012] Figure 3 A flowchart illustrating an exemplary implementation of this disclosure for generating audio data is shown.

[0013] Figure 4 A block diagram of an upsampling network according to an exemplary implementation of this disclosure is illustrated schematically;

[0014] Figure 5 A block diagram illustrating the structure of a shuffling network model according to an exemplary implementation of this disclosure is shown schematically.

[0015] Figure 6 A block diagram of an adversarial network according to an exemplary implementation of this disclosure is illustrated schematically;

[0016] Figure 7A A block diagram of a frequency conversion unit according to an exemplary implementation of the present disclosure is shown schematically;

[0017] Figure 7B A block diagram of a time-frequency (TF) unit according to an exemplary implementation of the present disclosure is illustrated.

[0018] Figure 8 A block diagram illustrating a navigation application scenario according to an exemplary implementation of this disclosure is shown schematically; and

[0019] Figure 9 A block diagram of a computing device / server for generating audio data, according to an exemplary implementation of the present disclosure, is shown schematically. Detailed Implementation

[0020] Preferred implementations of this disclosure will now be described in more detail with reference to the accompanying drawings. While preferred implementations of this disclosure are shown in the drawings, it should be understood that this disclosure may be implemented in various forms and should not be limited to the implementations set forth herein. Rather, these implementations are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0021] The term “comprising” and its variations, as used herein, signify open inclusion, i.e., “including but not limited to.” Unless otherwise stated, the term “or” means “and / or.” The term “based on” means “at least partially based on.” The terms “one example implementation” and “one implementation” mean “at least one example implementation.” The term “another implementation” means “at least one additional implementation.” The terms “first,” “second,” etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0022] Several speech generators have been proposed so far; see first for more details. Figure 1 This describes an application environment based on an exemplary implementation of the present disclosure. Figure 1 A block diagram 100 schematically illustrates a generator in which a technical solution according to an exemplary implementation of this disclosure can be used. (See diagram 100 for more details.) Figure 1 As shown, text features 110 (e.g., the embedding of the text to be generated into speech) can be input into generator 140. Generator 140 can then generate higher-dimensional audio data 112 based on the lower-dimensional text features 110. Specifically, the generator can perform deconvolution and upsampling operations, and use orthogonal filters to divide the signal into multiple frequency bands through downsampling at different factors. Generator 140 can then learn and process each frequency band separately, and finally synthesize the multiple frequency bands into a full-band signal through convolution operations.

[0023] Generator 140 may include one or more generation units, and each generation unit may include two main parts. For example, generation unit 120 may include an upsampling network 122 and a residual network 124. According to an exemplary implementation of this disclosure, four or other numbers of residual networks 124 may be included. Generator 140 may include one or more other generation units 130, and generation units 130 may have a similar structure. For example, generation unit 130 may include an upsampling network 132 and a residual network 134. Due to the large number of residual networks and the significant computational resources required, it is difficult to achieve high-quality speech generation on computing devices with limited computational resources (e.g., mobile terminals).

[0024] To at least partially address the shortcomings of the aforementioned technical solutions, a method for generating audio data is proposed according to an exemplary implementation of this disclosure. Specifically, a shuffling network model with lower computational resource requirements can be used to replace the residual network, thereby reducing the computational overhead of each generation unit. Furthermore, to compensate for the accuracy deficiency introduced by the shuffling network model, enhancement operations can be performed on the input data of the shuffling network model to improve the accuracy of the signal to be processed. Using the exemplary implementation of this disclosure, an effective balance can be achieved between reducing computational resource overhead and ensuring signal processing accuracy, thereby improving the efficiency of audio data generation.

[0025] See below. Figure 2 A summary of an exemplary implementation according to this disclosure is provided. Figure 2 A block diagram 200 schematically illustrates a generation unit 210 according to an exemplary implementation of the present disclosure, wherein... Figure 2 The left side shows the existing generation unit 130, and Figure 2 The right side shows a generation unit 210 according to an exemplary implementation of this disclosure. For example... Figure 2 As shown, the structure of the generation unit 130 can be modified to obtain the network structure shown in the generation unit 210. Specifically, the residual network 134 can be replaced with the shuffling network model 230, and an enhancement unit 222 can be added to the upsampling network 132 to obtain a new upsampling network 220.

[0026] It will be understood that the shuffling network model is an efficient CNN (Convolutional Neural Network) model, involving relatively low computational cost and particularly suitable for use on mobile devices with limited computing resources. The design goal of the shuffling network model is to achieve higher model accuracy with limited computing resources, striking a balance between computational speed and accuracy. The core of the shuffling network model employs two operations: pointwise grouped convolution and channel shuffling, to significantly reduce the model's computational cost while maintaining accuracy. The shuffling network model reduces model size and increases computational speed by designing a more efficient network structure. According to an exemplary implementation of this disclosure, replacing the residual network 134 in the generation unit 130 with a shuffling network model can greatly reduce the generator's computational resource requirements, thereby increasing the speed of audio generation on small terminal devices.

[0027] While reducing computational resource requirements, the shuffling network model may lead to a decrease in processing accuracy. In this case, an enhancement unit 222 can be added to the upsampling network to improve the accuracy of the input data to be processed by the shuffling network model 230. In this way, by improving the accuracy of the input data, the accuracy decrease caused by the shuffling network model 230 can be compensated for to some extent, thereby ensuring a balance between speed and accuracy in the audio generation process.

[0028] See below. Figure 3 Further details describe an exemplary implementation according to this disclosure. Figure 3 A flowchart illustrating a method 300 for generating audio data according to an exemplary implementation of the present disclosure is shown schematically. In the generator according to the exemplary implementation of the present disclosure, the following can be utilized: Figure 2 The generation unit 210 is shown, and method 300 can be executed by the generation unit 210. Specifically, at block 310, in response to receiving text feature data 110, a deconvolution operation is performed on the text feature data 110 to determine the deconvolution data of the text feature data. Here, the text feature data 110 can adopt a variety of predefined formats based on the underlying network architecture used by the generator.

[0029] According to an exemplary implementation of this disclosure, a generator based on a Mel Adversarial Network (MelGAN) can be used, where the text feature data 110 can be represented using Mel spectra. Mel frequencies are proposed based on the characteristics of human hearing and have a non-linear relationship with Hz frequencies. Mel frequency cepstral coefficients (MFCCs) utilize this relationship to calculate Hz spectral features, and MFCCs have been widely used in the field of speech recognition. Due to the non-linear relationship between Mel frequencies and Hz frequencies, the calculation accuracy of MFCCs decreases as the frequency increases. In applications, often only low-frequency MFCCs are used, while mid- and high-frequency MFCCs are discarded.

[0030] It will be understood that the process of generating high-dimensional audio data 112 from low-dimensional text feature data 110 is an upsampling process, which requires gradually expanding the dimension of the input text feature data 110 in multiple steps. According to an exemplary implementation of this disclosure, deconvolution operations can be performed based on deconvolution units in the upsampling network to achieve the upsampling objective. Figure 4 A block diagram 400 of an upsampling network 220 according to an exemplary implementation of this disclosure is illustrated. Figure 4 As shown, the upsampling network 220 may include two branches, where the right branch has the structure of a conventional upsampling network, that is, it uses the deconvolution unit 410 to perform deconvolution operations.

[0031] It will be understood that deconvolution is a special type of convolution. First, the dimensionality of the input data is increased by padding with zeros according to a certain ratio, and then the convolution kernel is rotated for forward convolution. At this point, the deconvolution unit 410 can convert the input text feature data 110 into deconvolution data, thereby increasing the dimensionality. Although the deconvolution unit 410 can increase the dimensionality of the input data, zero padding introduces inaccuracies into the output of the deconvolution unit 410, which in turn affects the accuracy of the input data of subsequent processing units. It will be understood that deconvolution is the inverse process of downsampling through convolution kernels, and implementing this inverse process through network learning is complex and time-consuming. At the beginning of model training, insufficient learning may lead to inaccuracies in some information, causing the residual module to introduce greater bias when converting from frequency domain signals to time domain signals. To address these shortcomings, an enhancement processing branch can be added to the left side of the upsampling network 220 (i.e., the enhancement unit 222 now includes the upsampling unit 420 and the convolution unit 430), and data augmentation can be achieved based on the augmented and deconvolutioned data.

[0032] return Figure 3Further details on generating the augmented data are described below. At box 320, the deconvolutional data can be augmented using upsampled data associated with the text feature data to generate augmented deconvolutional data. According to an exemplary implementation of this disclosure, upsampled data associated with the text feature data 110 can be generated using upsampling unit 420 and convolution unit 430.

[0033] Within the context of this disclosure, data augmentation can be performed in various ways. For example, interpolation can be used to implement the upsampling unit 420. Interpolation methods can include linear interpolation, assuming a linear functional relationship, where the dimensionality of the input data can be increased based on linear interpolation between two points in the input data. Another example is nearest-neighbor interpolation. Specifically, when determining the interpolation at the target location, the value at the closest known location to the target location can be used as the interpolation at the target location. Yet another example is the use of bilinear interpolation to increase dimensionality. Interpolation can be obtained along different directions to calculate the final interpolation. Utilizing the exemplary implementations of this disclosure, the dimensionality of input data can be increased in a flexible manner to provide more information for subsequent data processing.

[0034] According to one exemplary implementation of this disclosure Figure 4 The upsampling unit 420 can perform bilinear adjacent interpolation to determine the interpolated data of the text feature data 110. The convolution unit 430 can perform convolution operations on the interpolated data to determine the convolutional interpolation data. The outputs of the left and right branches can be transformed to the same dimension to generate augmented data based on the two outputs.

[0035] At this point, the convolutional interpolation data output from the left branch and the deconvolutional data output from the right branch have the same dimension, enabling data augmentation. Specifically, the augmented deconvolutional data can be determined based on the sum of the convolutional interpolation data and the deconvolutional data. In this case, the augmented deconvolutional data output by the upsampling network 220 will include information from both the left and right branches. That is, the augmented deconvolutional data can fully utilize information obtained through different methods to achieve dimensionality expansion, thereby improving the accuracy of subsequent data processing.

[0036] According to one exemplary implementation of this disclosure, deconvolution data can be determined based on a weighted sum of convolutional interpolation data and deconvolutional data. For example, different weights can be assigned to the two branches, with the branch that contributes more to improving audio generation quality being given a higher weight. It will be understood that, although... Figure 4Only one enhancement unit is shown in the left branch. According to one exemplary implementation of this disclosure, the upsampling network 220 may include more enhancement branches similar to those shown in the left branch. For example, one enhancement branch may obtain the first interpolated data based on linear interpolation, and another enhancement branch may obtain the second interpolated data based on nearest-neighbor interpolation. Furthermore, the final deconvolution data may be determined based on a weighted sum of the first interpolated data, the second interpolated data, and the deconvolution data.

[0037] According to an exemplary implementation of this disclosure, the weights of each branch can be learned based on machine learning techniques and utilizing historical training data. Using this exemplary implementation, interpolated data can supplement the deconvolution operation with richer information, enabling the enhanced deconvolution data to have higher accuracy, thereby contributing to further improvements in audio generation quality.

[0038] return Figure 3 Box 330 in the diagram can utilize a shuffling network model to process enhanced deconvolutional data to generate audio data 112 corresponding to text feature data 110. The core idea of ​​the shuffling network model is to perform a shuffling operation on different channels, thereby addressing the drawbacks of grouped convolution. Grouped convolution can group different input feature data and then use different convolution kernels to convolve each group, thus reducing the computational cost of convolution. See below for more details. Figure 5 For more details describing the shuffling network model, the Figure 5 A block diagram 500 schematically illustrates the structure of a shuffling network model according to an exemplary implementation of the present disclosure.

[0039] Figure 5 This illustrates a shuffling network model improved upon residual units. Specifically, the shuffling network model may include convolutional layers, depthwise convolution (DWConv) layers, and other convolutional layers. Figure 5 As shown, the shuffling network model can include a three-layer structure: at layer 510, a 1×1 convolution operation is performed (incorporating BatchNorm ReLU into the convolution process); at layer 520, a 3×3 depthwise convolution is performed (incorporating BatchNorm ReLU into the convolution process) to reduce computational cost; this 3×3 convolution is the bottleneck layer; and at layer 530, a 1×1 convolution is performed (incorporating BatchNorm into the convolution process). Furthermore, a short-circuit connection can be performed at layer 540 to add the input data to the output of layer 530. By using a shuffling network model instead of a residual network, the generator's computational resource requirements can be reduced, thus facilitating audio generation on computationally limited devices.

[0040] According to an exemplary implementation of this disclosure, when implementing a generator based on MelGAN, the generator may include multiple residual blocks. In this case, methods such as... Figure 5 The shuffling network model structure shown replaces at least a portion of the residual blocks. In this way, the quality of audio generation can be ensured while reducing computational resource overhead. According to an exemplary implementation of this disclosure, the residual blocks and the shuffling network model can be alternated. For example, in a generator comprising four residual blocks, the first and third residual blocks can be retained, and the second and fourth residual blocks can be replaced using the shuffling network model. In this way, the residual blocks and the shuffling network model alternate, thereby ensuring a balance between reducing computational resource overhead and ensuring audio quality. That is, the audio generation speed can be improved without significant loss of sound quality.

[0041] The details of the generator implemented based on MelGAN have been described above; in the following text, please refer to... Figure 6 Describe the operation of the discriminator in MelGAN. Figure 6 A block diagram 600 of an adversarial network according to an exemplary implementation of this disclosure is illustrated schematically. Figure 6 As shown, MelGAN can include two parts: a generator 610 and a discriminator 620. The generator 610 can execute the method 300 described above, and the discriminator 620 can determine the authenticity of the audio data 614 generated by the generator 610. Specifically, the discriminator 620 can determine the label 622 of the audio data 614. For example, the label "true" indicates that the discriminator 620 considers the audio data 614 to be real audio data, while the label "false" indicates that the discriminator 620 considers the audio data 614 to be not real audio data. The label 622 can be used to optimize the generator 610. Specifically, the parameters of the generator 610 can be continuously updated to make the label distribution output by the discriminator 620 tend to be more balanced. A conventional discriminator outputs the label 622 based on the frequency information in the audio data 614.

[0042] According to an exemplary implementation of this disclosure, a Frequency Transform Block (FTB) unit can be added to the discriminator 620 to further consider the time-domain information of the audio data 614, as well as the correlation between the time-domain information and the frequency-domain information, in addition to the frequency information. The FTB unit can consider the time-domain information and capture the correlation between the time and frequency domains. Figure 7A A block diagram 700A of a frequency conversion unit according to an exemplary implementation of the present disclosure is illustrated schematically. Figure 7AAs shown, the TF unit 710 can be used to process the received audio data. This TF unit 710 can utilize a frequency attention mechanism to ensure that each frequency point has relevant information from other frequency bands.

[0043] Figure 7B A block diagram 700B of a TF unit according to an exemplary implementation of this disclosure is illustrated schematically. Figure 7B As shown, at convolutional layer 712, a 2D convolution operation (with a 1×1 kernel) can be performed on the frequency domain features of the input audio data. At convolutional layer 714, a 1D convolution operation can be performed on the output of convolutional layer 712. Subsequently, at multiplication layer 716, the output of convolutional layer 714 can be multiplied with the frequency domain features. Furthermore, a fully connected layer 718 can be used to process the product of the output of multiplication layer 716, so as to map the product to an embedding that includes the time-domain-frequency domain correlation. In this way, multiple time points in the audio data and the correlation of audio features at multiple time points can be obtained.

[0044] According to an exemplary implementation of this disclosure, frequency information of multiple audio frames located at multiple time points can be extracted from audio data. Specifically, the audio data can be divided into multiple audio segments according to a predetermined step size. For example, each audio segment can be specified to include 512 sampling points, in which case the audio data can be divided into multiple frequency bands with a length of 512 sampling points. According to an exemplary implementation of this disclosure, two consecutive audio segments among the multiple audio segments may include an overlapping portion (e.g., defined by the parameter hop_size). The length of the overlapping portion can be pre-specified (e.g., hop_size = 256 sampling points). Assuming the first audio segment includes sampling points 1-512, the second audio segment includes sampling points 257-763, and so on. Each audio segment can be processed separately and the corresponding audio information can be extracted. In this case, each audio feature can include more information, thereby improving the accuracy of identification.

[0045] According to an exemplary implementation of this disclosure, it is possible to utilize Figure 7B The TF unit 710 shown is used to process each audio segment, thereby obtaining the TF correlation features associated with each audio segment. Return Figure 7AAt box 720, the output of TF unit 710 (i.e., TF correlation features) can be concatenated with the original frequency domain features. At this point, the audio features can include both the original frequency domain features and the TF correlation features. Further, at box 730, a convolution operation can be performed on the concatenated audio features to obtain the final audio features. In this way, the final audio features can include information from the time domain, frequency domain, and time-frequency correlation. Therefore, more content of the audio data to be identified can be described in a more accurate way. Using the exemplary implementation of this disclosure, the discriminator 620 can identify the authenticity of the input audio data based on richer information, thereby improving the accuracy of the discriminator 620.

[0046] Furthermore, improving the accuracy of the discriminator 620 helps optimize the parameters of the generator 610, leading to more realistic audio data generation by the generator 610. According to one exemplary implementation of this disclosure, labeled training data can be used to train the generators 610 and 620 according to one exemplary implementation of this disclosure. According to one exemplary implementation of this disclosure, the generator 610 and discriminator 620 according to one exemplary implementation of this disclosure can be implemented using an MB_MelGAN (Multi-Band MelGAN) network structure.

[0047] Generator 610 can be configured based on the following parameters: Use a three-layer upsampling network with an upsampling factor of 64 (i.e., after three upsampling operations, the output data is 64 times the input data), with upsampling factors of 2, 4, and 8 for the three layers. You can set `hop_size` to 256 and use a four-layer network. The shuffling network model can use two channels, in which case the output channel is half the size of the original residual block. Refer to the standard configuration for MB_MelGAN to set other parameters for generator 610.

[0048] The discriminator 620 can be configured with the following parameters. The discriminator 620 can be configured to include both time-domain and frequency-domain discrimination. The time-domain discrimination part can be implemented based on the standard MB_MelGAN configuration, i.e., downsampling the input data twice. The frequency-domain discriminator can be implemented based on the method described above. Specifically, in the discriminator 620, the output data of the Short Time Fourier Transform (STFT) unit can be fed to the FTB unit described above. The fft_size (Fast Fourier Transform size) can be set to 512, shift_size (shift size) to 120, and win_length (window length) to 600, and frequency_dim (frequency dimension) to 257.

[0049] The specific configurations of the generator 610 and discriminator 620 according to an exemplary implementation of this disclosure have been described above. In different application environments, the training process can be performed based on different training datasets. For example, in a Chinese language environment, the publicly available training dataset CSMSC (Chinese Standard Mandarin Speech Corpus) can be used to perform the training process. This training dataset includes 12 hours of single-speaker Chinese speech in 48kHz, 16-bit WAV audio format. The generator 610 and discriminator 620 described above can be trained based on the training dataset CSMSC. Specifically, the Adam optimizer can be used, and training can be performed with the learning rate decreasing as the training steps increase. For example, the learning rate can be gradually reduced from 0.001 to 0.000001 (alternatively, the learning rate can be in other time ranges).

[0050] According to an exemplary implementation of this disclosure, the generator 610 can be trained independently first. For example, the number of training iterations can be specified as 20k (or other values). After the generator 610 has been trained independently, the generator 610 and the discriminator 620 can be trained together. A training stopping condition can be specified, for example, a predetermined number of training iterations (e.g., 80k or other values) can be specified, or a predetermined convergence state or training time can be specified. For example, training can be performed in batch mode, with a batch size of 32 (or other values), in which case 32 frames can be randomly selected from each corpus for training. The training process is complete when the above-mentioned training stopping condition is met. In subsequent use, only the trained generator 610 needs to be called, while the discriminator 620 is only effective during the training phase and is not directly used to generate audio data.

[0051] Audio data can be generated using a trained generator 610. For example, text features can be input into the generator 610, which will then output corresponding audio data. Employing a generator 610 according to an exemplary implementation of this disclosure can reduce the computational resource requirements of the processing and achieve better audio quality. Audio quality can be evaluated through manual testing. For example, 20 sentences can be randomly selected from a test set, and 10 listeners can rate them. Table 1 below shows a comparison between the generator of this disclosure and generators implemented based on several existing network architectures.

[0052] Table 1 Comparison of audio generation parameters

[0053]

[0054] Table 1 compares the implementation of this disclosure with existing HiFi-GAN V2 and MB-MelGAN technologies. Specifically, each column lists several comparison parameters: number of parameters, GFLOPS (floating-point operations), RTF (real-time factor), MOS score, and TTS score. As can be seen from Table 1, the implementation of this disclosure outperforms existing technologies in all parameters and provides data closer to the true value (i.e., real speech). Therefore, the implementation of this disclosure can improve the quality and speed of audio generation while reducing computational load.

[0055] According to one exemplary implementation of this disclosure, generator 610 can be applied to different application environments. Figure 8 A block diagram 800 schematically illustrates a navigation application scenario according to an exemplary implementation of the present disclosure. In this case, the generator 610 can be deployed at a terminal device and invoked by a navigation application on the terminal device. The positioning unit 810 at the terminal device can acquire the terminal device's position 812 in real time and provide the position 812 to the prediction unit 820. Here, the prediction unit 820 can generate navigation text 814 based on the position 812. The feature extraction unit 830 can extract text features 816 and provide them to the generator 610. Further, the generator 610 can operate as described above and output audio data 818 to a speaker 840 for playback. Using the exemplary implementation of the present disclosure, the generator 610 can run on a terminal device with limited computing resources, ensuring that the generator 610 can be applied in a navigation application environment. In this way, the corresponding navigation voice can be automatically played on the terminal device based on the acquired position.

[0056] See above Figures 2 to 8 A method for generating audio data is described. According to an exemplary implementation of this disclosure, an apparatus for generating audio data is provided, including a generator for converting text feature data into audio data. The generator includes: a deconvolution module configured to perform a deconvolution operation on the text feature data in response to receiving the text feature data to determine deconvolution data of the text feature data; an enhancement module configured to enhance the deconvolution data using upsampled data associated with the text feature data to generate enhanced deconvolution data; and a generation module configured to process the enhanced deconvolution data using a shuffling network model to generate audio data corresponding to the text feature data.

[0057] According to an exemplary implementation of this disclosure, the enhancement module includes: an interpolation module configured to perform interpolation operations on text feature data to use the interpolated data of the text feature data as upsampled data; a convolution module configured to perform convolution operations on the interpolated data to determine convolutional interpolated data of the interpolated data; and a data enhancement module configured to enhance deconvolutional data using the convolutional interpolated data to generate enhanced deconvolutional data.

[0058] According to an exemplary implementation of this disclosure, the data augmentation module includes a weighted summation module configured to generate augmented deconvolution data based on a weighted summation of convolutional interpolation data and deconvolutional data.

[0059] According to one exemplary implementation of this disclosure, the shuffled network model includes: convolutional layers, depthwise convolutional layers, and convolutional layers.

[0060] According to one exemplary implementation of this disclosure, the generator is a generator based on a Mel adversarial network, and the text feature data is a Mel graph.

[0061] According to an exemplary implementation of this disclosure, the generator is a generator based on a multi-band Mel adversarial network, the generator comprising multiple residual networks, and at least one of the residual networks is replaced by a shuffled network model.

[0062] According to an exemplary implementation of this disclosure, the Mel adversarial network further includes a discriminator, and the apparatus further includes: a discrimination module configured to use the discriminator to discriminate the authenticity of audio data generated by the generator; and an optimization module configured to optimize the generator based on the authenticity of the audio data.

[0063] According to an exemplary implementation of this disclosure, the authentication module includes: an extraction module configured to extract frequency information of multiple audio frames located at multiple time points from the audio data; an association determination module configured to determine the association relationship between the frequency information and the multiple time points; and an authenticity authentication module configured to determine the authenticity of the audio data based on the association relationship.

[0064] According to an exemplary implementation of this disclosure, the extraction module includes: a segmentation module configured to divide the audio data into multiple audio segments according to a predetermined step size, wherein two consecutive audio segments among the multiple audio segments include an overlapping portion; and an audio information determination module configured to determine audio information for multiple time points within the multiple audio segments. The association determination module is further configured to determine association relationships based on a frequency attention mechanism.

[0065] According to an exemplary implementation of this disclosure, the generator is invoked by a navigation application running on a terminal device, and the module further includes: a text generation module configured to generate navigation text based on the location of the terminal device; an input module configured to input text feature data associated with the navigation text to the generator; and a playback module configured to play the generated audio data.

[0066] Figure 9 A block diagram schematically illustrates a computing device / server 900 for generating audio data according to an exemplary implementation of this disclosure. It should be understood that... Figure 9 The computing device / server 900 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein.

[0067] like Figure 9 As shown, the computing device / server 900 is in the form of a general-purpose computing device. Components of the computing device / server 900 may include, but are not limited to, one or more processors or processing units 910, memory 920, storage devices 930, one or more communication units 940, one or more input devices 950, and one or more output devices 960. The processing unit 910 may be a physical or virtual processor and is capable of performing various processes according to programs stored in the memory 920. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of the computing device / server 900.

[0068] The computing device / server 900 typically includes multiple computer storage media. Such media can be any available media accessible to the computing device / server 900, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 920 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 930 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data (e.g., training data for training) and accessible within the computing device / server 900.

[0069] The computing device / server 900 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 9As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 920 may include computer program product 925 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0070] The communication unit 940 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of the computing device / server 900 can be implemented as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the computing device / server 900 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0071] Input device 950 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 960 can be one or more output devices, such as a monitor, speaker, printer, etc. The computing device / server 900 can also communicate with one or more external devices (not shown) via communication unit 940 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with the computing device / server 900, or with any device (e.g., network card, modem, etc.) that enables the computing device / server 900 to communicate with one or more other computing devices. Such communication can be performed via input / output (I / O) interfaces (not shown).

[0072] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores one or more computer instructions, wherein one or more computer instructions are executed by a processor to implement the methods described above.

[0073] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0074] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0075] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0076] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0077] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the implementations disclosed herein.

Claims

1. A method for generating audio data, comprising: In the generator used to convert text feature data into audio data In response to receiving text feature data, a deconvolution operation is performed on the text feature data to determine the deconvolution data of the text feature data; Using upsampled data associated with the text feature data, the deconvolution data is enhanced to generate enhanced deconvolution data, including: An interpolation operation is performed on the text feature data to use the interpolated data of the text feature data as the upsampled data; Perform a convolution operation on the interpolated data to determine the convolutional interpolation data of the interpolated data; and The deconvolutional data is enhanced using the convolutional interpolation data to generate the enhanced deconvolutional data; and The enhanced deconvolutional data is processed using a shuffle network model to generate audio data corresponding to the text feature data.

2. The method of claim 1, wherein enhancing the deconvolution data using the convolution interpolation data to generate the enhanced deconvolution data comprises: The enhanced deconvolution data is generated by weighted summation of the convolutional interpolation data and the deconvolutional data.

3. The method according to claim 1, wherein the shuffling network model comprises: Convolutional layers, depthwise convolutional layers, and convolutional layers.

4. The method according to claim 3, wherein the generator is a generator implemented based on a Mel adversarial network, and the text feature data is a Mel graph.

5. The method according to claim 4, wherein the generator is a generator implemented based on a multi-band Mel adversarial network, the generator comprising multiple residual networks, and at least one of the multiple residual networks is replaced by the shuffling network model.

6. The method of claim 4, wherein the Mel adversarial network further comprises a discriminator, and the method further comprises: The discriminator is used to verify the authenticity of the audio data generated by the generator; as well as The generator is optimized based on the authenticity of the audio data.

7. The method of claim 6, wherein using the discriminator to determine the authenticity of the audio data generated by the generator comprises: Extract frequency information of multiple audio frames located at multiple time points from the audio data; Determine the correlation between the frequency information and the plurality of time points; as well as The authenticity of the audio data is determined based on the aforementioned correlation.

8. The method of claim 7, wherein extracting the frequency information of the plurality of audio frames located at the plurality of time points from the audio data comprises: The audio data is divided into multiple audio segments according to a predetermined step size, and two consecutive audio segments among the multiple audio segments include an overlapping portion; as well as The frequency information is determined for the multiple time points within the multiple audio segments.

9. The method of claim 8, wherein determining the correlation between the frequency information and the plurality of time points comprises: The correlation is determined based on a frequency attention mechanism.

10. The method of claim 1, wherein the generator is invoked by a navigation application running on a terminal device, and the method further comprises: Navigation text is generated based on the location of the terminal device; Input text feature data associated with the navigation text into the generator; as well as Play the generated audio data.

11. An apparatus for generating audio data, comprising a generator for converting text feature data into audio data, the generator comprising: A deconvolution module is configured to perform a deconvolution operation on the received text feature data in response to determine the deconvolution data of the text feature data; An enhancement module, configured to enhance the deconvolution data using upsampled data associated with the text feature data to generate enhanced deconvolution data, includes: An interpolation module is configured to perform interpolation operations on the text feature data to use the interpolated data of the text feature data as the upsampled data; A convolution module is configured to perform convolution operations on the interpolated data to determine convolutional interpolation data of the interpolated data; and A data augmentation module is configured to augment the deconvolution data using the convolution interpolation data to generate the augmented deconvolution data; and The generation module is configured to process the enhanced deconvolutional data using a shuffled network model to generate audio data corresponding to the text feature data.

12. An electronic device, comprising: Memory and processor; The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having stored thereon one or more computer instructions, wherein the one or more computer instructions are executed by a processor to implement the method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program / instructions, wherein the computer program / instructions, when executed by a processor, implement the method according to any one of claims 1 to 10.

Images