Audio data generation device, adversarial learning method for audio data generation device, learning method for audio data generation device, and speech synthesis processing system

Abstract

To provide an audio data generation device for achieving high-speed audio generation processing with high quality without using GPU which can perform high-speed processing.SOLUTION: An audio data generation device 100 has a structure for acquiring a plurality of pieces of stream data by a multi-stream generation part 1, and can perform hostile learning with a high-precision audio data identification device Dev_D by introducing a learnable convolution processing part 3. The audio data generation device 100 acquired by the hostile learning can perform high-speed and high-precision audio data generation processing. Since the audio data generation device 100 includes a simple structure, high-quality audio data generation processing (voice synthesis processing, for example) can be achieved at high speed without using GPU which can perform high-speed processing.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This invention relates to audio data synthesis technology (e.g., speech synthesis technology). [Background technology]

[0002] In recent years, speech synthesis technology using neural networks has advanced significantly, making it possible to synthesize high-quality speech that is almost indistinguishable from natural speech. Many neural network-based speech synthesis technologies require high-speed GPU (Graphics Processing Unit) computing to perform speech synthesis processing in real time. However, for widespread adoption as a practical service, it is crucial to develop a technology that can synthesize speech quickly and with high quality using only the CPU (Central Processing Unit), without the need for a GPU.

[0003] Two technologies that use a CPU to realize a high-speed and high-quality neural vocoder are Multi-band MelGAN (see Non-Patent Document 1) and HiFi-GAN (see Non-Patent Document 2). Both are based on generative adversarial networks and are methods that train the generator and discriminator simultaneously (adversarial learning type neural vocoder). The generator is trained to deceive the discriminator, and the discriminator is trained to determine that the audio waveform used for training is real and the audio waveform generated by the generator is fake. In other words, the discriminator is trained to distinguish between real data and fake data with high accuracy.

[0004] MelGAN (see Non-Patent Literature 3), the predecessor of Multi-band MelGAN, uses a generator that converts input acoustic features into an audio waveform using several upsampling and convolutional layers. In Multi-band MelGAN, to speed up conventional MelGAN, a full-band audio signal is divided into multiple subband signals (= multiband signals) using subband processing based on multirate signal processing. The generator simultaneously generates audio waveforms (subband signals) for the divided multiple bands. After applying zero-insertion upsampling to the generated subband signals, a pre-calculated composite filter (FIR filter) is used to generate a full-band audio signal from the signal after zero-insertion upsampling. In this case, the discriminator is learned from (1) the short-time Fourier transform (STFT) amplitude loss of the multiband signal, (2) the STFT amplitude loss of the full-band signal, and (3) the discrimination loss, which is the discrimination result of the discriminator. As a result, in Multi-band MelGAN, the final upsampling process (for example, upsampling to quadruple the number of data points when using four subband signals) is simplified to a simple zero insertion and FIR filtering process. Consequently, Multi-band MelGAN enables faster speech synthesis while maintaining the accuracy of MelGAN.

[0005] On the other hand, HiFi-GAN, like MelGAN, consists of a generator made up of several upsampling and convolutional layers, and two types of discriminators. The generator with 512 channels in the first layer is called the V1 generator, and the generator with 128 channels in the first layer is called the V2 generator.

[0006] The V1 generator is capable of high-quality speech synthesis processing and can generate speech in real time by using multiple CPU cores. The V2 generator cannot generate speech with the same high accuracy as the V1 generator, but it can perform high-speed speech synthesis with a real-time factor (time required to generate one second of speech) of about 0.1 even with a single CPU core.

[0007] HiFi-GAN introduces two discriminators, a multi-period discriminator and a multi-scale discriminator, enabling highly accurate modeling of the periodic pattern and continuity of speech waveforms, as well as their long-term dependence. As a result, HiFi-GAN utilizes a sophisticated network (a neural network that considers diverse features of speech waveforms, both global and local) to achieve high-speed processing and higher-quality speech synthesis than Multi-band MelGAN. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] G. Yang, S. Yang, K. Liu, P. Fang, W. Chen, and L. Xie, "Multi-band MelGAN: Faster waveform generation for high-quality text-to-speech," in Proc. SLT, Jan. 2021, pp. 492-498. [Non-Patent Document 2] J. Kong, J. Kim, and J. Bae, "HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis," in Proc. NeurIPS, Dec. 2020, pp. 17022-17033. [Non-Patent Document 3] K. Kumar, R. Kumar, T. de Boissiere, L. Gestin, WZ Teoh, J. Sotelo, A. de Bre bisson, Y. Bengio, and A. C Courville, "MelGAN: Generative adversarial networks for conditional waveform synthesis," in Proc. NeurIPS, Dec. 2019, pp. 14910- 14921. [Overview of the project] [Problems that the invention aims to solve]

[0009] HiFi-GAN can generate (synthesize) high-quality speech at high speed, but there is a trade-off between sound quality and generation speed (speech synthesis processing speed). In other words, with the HiFi-GAN V1 generator (V1 model), the generated speech is of high quality, but the generation speed (speech synthesis processing speed) is not very fast. On the other hand, with the V2 generator (V2 model), the generation speed (speech synthesis processing speed) is as fast as Multi-band MelGAN, but the quality (sound quality) of the generated speech is not as high, and is about the same as Multi-band MelGAN.

[0010] As a simple solution to achieve high-quality speech synthesis processing at high speed, a method of introducing a multiband generation algorithm into HiFi-GAN was considered, and this method was investigated in preliminary experiments. However, it was found that the method of introducing a multiband generation algorithm into HiFi-GAN has the problem that the generator loss cannot be reduced and learning does not go well. The reason why the generator loss cannot be reduced and learning does not go well when introducing a multiband generation algorithm into HiFi-GAN is that the two classifiers of HiFi-GAN have very high discriminatory ability, so they can distinguish it as fake as soon as the constraint of multiband is introduced. Even when using pre-training using only STFT amplitude loss, as used in Multi-band MelGAN, the results showed that learning still did not go well.

[0011] Therefore, in view of the above problems, the present invention aims to realize an audio data generation device that can perform high-quality audio generation processing (e.g., speech synthesis processing) at high speed without using a GPU capable of high-speed processing. [Means for solving the problem]

[0012] The first invention for solving the above problems is an audio data generation device comprising a multistream generation unit, an upsampling unit, and a convolution processing unit.

[0013] The multistream generation unit includes a learnable functional unit that acquires multiple stream data from Mel spectrogram data.

[0014] The upsampling unit obtains upsampled multi-stream data by performing upsampling on each of the multiple stream data.

[0015] The convolution processing unit can learn the parameters for determining the convolution process, and acquires audio waveform data by performing the convolution process on upsampled multistream data.

[0016] This audio data generation device has a configuration that acquires multiple stream data (for example, four data-driven decomposition data (audio waveform data)) using a multi-stream generation unit, and furthermore, by introducing a learnable convolutional processing unit 3, adversarial learning with a high-precision audio data discrimination device becomes possible. Then, the audio data generation device acquired through this adversarial learning can perform high-speed and high-precision audio data generation processing. Moreover, because this audio data generation device has a simple configuration, it can achieve high-quality audio data generation processing (for example, speech synthesis processing) at high speed without using a high-speed GPU.

[0017] The second invention is the same as the first invention, wherein the convolution processing unit performs a bias-free convolution process.

[0018] This allows the convolution processing unit of this audio data generation device to have a configuration similar to that of an FIR filter.

[0019] The third invention is the first or second invention, wherein the upsampling unit performs zero-insertion type upsampling.

[0020] This allows the audio data generation device to perform upsampling with a simple configuration, enabling high-speed processing.

[0021] The fourth invention is an audio data generation device which is one of the first to third inventions, A global feature discriminator that includes a learnable functional unit and identifies the truth value of audio data based on the global features of the audio data, A detailed feature discriminator that includes a learnable functional unit and identifies the truth or falsity of audio data based on the detailed features of the audio data, This is an adversarial learning method for an audio data generator that is performed using an audio data identification device equipped with [a specific component]. The adversarial learning method for an audio data generator comprises an identification step, a loss evaluation step, a generator parameter update step, and a discriminator parameter update step.

[0022] The identification step involves inputting audio data generated by an audio data generation device, or the correct data for said audio data, into an audio data identification device, which then identifies the truthfulness of the input data.

[0023] The loss evaluation step obtains loss evaluation data using a loss function based on the result data from the identification step.

[0024] The generator parameter update step updates the parameters of the convolution processing unit of the audio data generator and the parameters of the learnable function unit of the multistream generation unit based on the loss evaluation data obtained in the loss evaluation step.

[0025] The discriminator parameter update step updates the parameters of the learnable functional parts of the global feature discriminator of the audio data discriminator, as well as the parameters of the learnable functional parts of the detailed feature discriminator of the audio data discriminator, based on the loss evaluation data obtained in the loss evaluation step.

[0026] In this adversarial learning method for audio data generation devices, adversarial learning is performed using an audio data discrimination device equipped with a global feature discrimination device and a detailed feature discrimination device, giving it strong discrimination capabilities. As a result, the audio data generation device after the learning process can generate high-precision audio data. Furthermore, in this adversarial learning method for audio data generation devices, the audio data generation device is equipped with a multi-stream generation unit that generates multiple streams and a convolutional processing unit that can learn from upsampled data. Therefore, even when performing adversarial learning using an audio data discrimination device with strong discrimination capabilities, learning can proceed efficiently and converge reliably.

[0027] The fifth invention is a learning method for an audio data generation device which is any of the first to third inventions, comprising an STFT loss evaluation step and a generator parameter update step.

[0028] The STFT loss evaluation step evaluates the loss between the audio data corresponding to the Mel spectrogram input to the audio data generation device and the generated audio data generated from the input Mel spectrogram by the audio data generation device, using the Short-Time Fourier Transform loss function.

[0029] The generator parameter update step updates the parameters of the convolution processing unit of the audio data generator and the parameters of the learnable function unit of the multistream generation unit based on the evaluation results in the STFT loss evaluation step.

[0030] This allows the audio data generator to perform its learning process using an evaluation value (loss value) obtained from a short-term Fourier transform loss function. Furthermore, for example, the learning process using this audio data generator learning method may be used as pre-training for adversarial learning using the audio data discrimination device of the audio data generator.

[0031] The sixth invention is a speech synthesis processing system comprising an audio processing device that outputs Mel spectrum data from text data, and an audio data generation device which is any of the first to third inventions.

[0032] This speech synthesis processing system uses an audio data generation device that can generate speech waveform data from a Mel spectrogram using a CPU without the need for a high-speed GPU. Therefore, it can perform high-speed, high-precision speech synthesis processing using a CPU without the need for a high-speed GPU. [Effects of the Invention]

[0033] According to the present invention, an audio data generation device can be realized that can perform high-quality audio generation processing (e.g., speech synthesis processing) at high speed without using a GPU capable of high-speed processing. [Brief explanation of the drawing]

[0034] [Figure 1] A schematic diagram of the audio data processing system 1000 according to the first embodiment. [Figure 2] A schematic diagram of the multi-stream generation unit 1 of the audio data generation device 100 of the audio data processing system 1000 according to the first embodiment. [Figure 3] A schematic diagram of the first MRF processing unit 122 of the multistream generation unit 1 of the audio data generation device 100 according to the first embodiment. [Figure 4] A schematic diagram of the components (ResBlock[n]) of the residual block group 1221 of the first MRF processing unit 122 according to the first embodiment. [Figure 5] A schematic diagram of the global feature identification unit DD1 of the audio data identification device Dev_D according to the first embodiment. [Figure 6] A schematic diagram of the detailed feature identification unit DD2 of the audio data identification device Dev_D according to the first embodiment. [Figure 7] A flowchart of the learning process performed by the audio data processing system 1000. [Figure 8] A diagram showing the CPU bus configuration. [Modes for carrying out the invention]

[0035] [First Embodiment] The first embodiment will be described below with reference to the drawings.

[0036] <1.1: Configuration of the Audio Data Processing System> Figure 1 is a schematic diagram of the audio data processing system 1000 according to the first embodiment.

[0037] Figure 2 is a schematic diagram of the multi-stream generation unit 1 of the audio data generation device 100 of the audio data processing system 1000 according to the first embodiment.

[0038] Figure 3 is a schematic diagram of the first MRF processing unit 122 of the multistream generation unit 1 of the audio data generation device 100 according to the first embodiment.

[0039] Figure 4 is a schematic diagram of the components (ResBlock[n]) of the residual block group 1221 of the first MRF processing unit 122 according to the first embodiment.

[0040] Figure 5 is a schematic diagram of the global feature identification unit DD1 of the audio data identification device Dev_D according to the first embodiment.

[0041] Figure 6 is a schematic diagram of the detailed feature identification unit DD2 of the audio data identification device Dev_D according to the first embodiment.

[0042] As shown in Figure 1, the audio data processing system 1000 comprises an audio data generation device 100, a generated data evaluation unit G_Ev, a selector SEL1, an audio data identification device Dev_D, an identified data evaluation unit D_Ev, and an updated data selection processing unit G_upd.

[0043] (1.1.1: Audio data generation device) As shown in Figure 1, the audio data generation device 100 comprises a multistream generation unit 1, an upsampling unit 2, and a convolution processing unit 3. The audio data generation device 100 takes data Din, which is data from a Mel spectrogram, as input, performs audio data generation processing on data Din, and acquires (outputs) audio waveform data Dout.

[0044] As shown in Figure 2, the multistream generation unit 1 comprises a first convolution processing unit 11, an MRF unit 12, a first activation processing unit 13, a second convolution processing unit 14, and a second activation processing unit 15.

[0045] The first convolution processing unit 11 receives data Din, which is data from a Mel spectrum, and performs a one-dimensional convolution (Conv1D processing) on ​​data Din (treating data Din as two-dimensional data and performing one-dimensional convolution processing). The first convolution processing unit 11 then outputs the data after the one-dimensional convolution (Conv1D processing) as data D11 to the MRF unit. The one-dimensional convolution (Conv1D processing) performed by the first convolution processing unit 11 is performed, for example, with a kernel size of "7" (corresponding to 7 samples) and a number of channels of "512".

[0046] As shown in Figure 2, the MRF unit 12 comprises a first upsampling unit 121, a first MRF processing unit 122, a second upsampling unit 123, and a second MRF processing unit 124.

[0047] The first upsampling unit 121 receives data D11 output from the first convolution processing unit 11 and performs upsampling on data D11. The first upsampling unit 121 outputs the data after upsampling as data D12 to the first MRF processing unit 122. The upsampling process performed by the first upsampling unit 121 is performed, for example, by increasing the number of samples in the input data to eight times (×8) and setting the number of channels to "256". The following methods can be used for the upsampling process. (1) Upsampling process using subpixel convolution For example, upsampling can be achieved by setting the kernel size to "3", performing a one-dimensional convolution (Conv1D) operation, and then performing a reshape operation. To achieve this, the number of input data samples can be increased by eight (x8), and the number of channels can be adjusted to 256 by adjusting the number of channels in the one-dimensional convolution (Conv1D) operation and the length and number of channels in the reshape operation. (2) Upsampling process using transposed convolution For example, upsampling can be achieved by performing a transposed convolution using an n×1 kernel with a stride of n / 2. The kernel size n and the number of channels for the transposed convolution can be adjusted to increase the number of input data samples by eight (×8) and result in 256 channels. (3) Sampling process using interpolation and one-dimensional convolution (Conv1D processing) For example, upsampling can be achieved by performing an interpolation process (e.g., interpolating adjacent samples) and then executing a one-dimensional convolution (Conv1D) process. To achieve this, the number of samples interpolated in the interpolation process, as well as the kernel size and number of channels in the one-dimensional convolution (Conv1D) process, should be adjusted to eight times (x8) the number of input data samples and to achieve 256 channels.

[0048] The first MRF processing unit 122 includes, for example, a residual block group 1221 and an adder 1222, as shown in Figure 3.

[0049] As shown in Figure 3, the residual block group 1221 consists of multiple residual blocks ResBlock[1]~ResBlock[|k] that each input the data D12 output from the first upsampling unit 121. r It is equipped with |].

[0050] Residual block ResBlock[n](1≦n≦|k r |)(|k r | is array k r The number of elements (array size) of the block BL1 has a configuration in which multiple blocks (|Dr[n]|) are connected in succession, as shown in Figure 4.

[0051] Block BL1 comprises multiple (|Dr[n,m]|) consecutively connected Block BL2 units, and an adder Add1 that adds data D12 to the output of the final Block BL2 unit.

[0052] As shown in Figure 4, block BL2 comprises an activation processing unit BL21 and a convolution processing unit BL22.

[0053] The activation processing unit BL21 is a functional unit that performs activation processing using the Leaky ReLU function (indicated as "Leaky ReLU" in Figure 4).

[0054] The convolution processing unit BL2 is k rA functional unit that performs convolution processing using a [n]×1 kernel (the functional unit indicated as "k r [n]×1 Conv" in FIG. 4). Note that the convolution processing unit BL2 uses dilation as Dr[n,m,L], and k r performs convolution processing on the output data from the previous activation processing unit BL21 using a [n]×1 kernel.

[0055] For example, the case where k r and Dr are set as follows will be described. k r =[k1,k2,k3] Dr=[[[a1,a2],[b1,b2],[c1,c2]], [[d1,d2],[e1,e2],[f1,f2]]] In the above case, |k r | = 3 and |Dr[n,m]| = 3.

[0056] Also, in the above case, for Dr[n,m,L], Dr[1,1,1] = a1, Dr[1,1,2] = a2, Dr[1,1,2] = b1, Dr[1,2,2] = b2, ···, Dr[2,3,1] = f1, Dr[2,3,2] = f2.

[0057] The residual block ResBlock[n] outputs the processed result data as data D12_out[n] to the addition unit 1222 according to the above configuration.

[0058] The addition unit 1222 adds the output data D12_out[1]~D12_out[|k r |] from each block of the residual block group 1221, and outputs the addition result data as data D13 to the second upsampling unit 123.

[0059] The second upsampling unit 123 receives data D13 output from the first MRF processing unit 122 and performs upsampling on data D13. The second upsampling unit 123 outputs the data after upsampling as data D14 to the second MRF processing unit 124. The upsampling process performed by the second upsampling unit 123 is performed, for example, by increasing the number of samples in the input data to eight times (×8) and setting the number of channels to "128". The method of upsampling is the same as that of the first upsampling unit. (1) Upsampling process using subpixel convolution (2) Upsampling process using transposed convolution (3) Sampling process using interpolation and one-dimensional convolution (Conv1D processing) You can choose one of the following.

[0060] The second MRF processing unit 124 has the same configuration as the first MRF processing unit, and performs the same processing on the data D14 output from the second upsampling unit 123 as the first MRF processing unit (note that k r (The setting value of Dr may be different from the setting value of the first MRF processing unit.) Then, the second MRF processing unit 124 outputs the data after processing in the second MRF processing unit 124 as data D15 to the first activation processing unit 13.

[0061] The first activation processing unit 13 receives data D15 output from the second MRF processing unit 124 of the MRF unit 12 and performs an activation process on the data D15 using the Leaky ReLU function. The first activation processing unit 13 then outputs the data after the activation process as data D16 to the second convolution processing unit 14.

[0062] The second convolution processing unit 14 receives data D16 output from the first activation processing unit 13 and performs a one-dimensional convolution (Conv1D) operation on data D16. The second convolution processing unit 14 then outputs the data after the one-dimensional convolution (Conv1D) operation as data D17 to the second activation processing unit 15. The one-dimensional convolution (Conv1D) operation performed by the second convolution processing unit 14 is performed, for example, with a kernel size of "7" (corresponding to 7 samples) and a number of channels of "4".

[0063] The second activation processing unit 15 receives the data D17 output from the second convolution processing unit 14 and performs an activation process on the data D17 using the tanh function. The second activation processing unit 15 then outputs the activated data as data D1 to the upsampling unit 2. If the number of channels in the second convolution processing unit 14 is "4", data D1 will be audio waveform data (four audio waveform data) obtained by applying the activation process by the second activation processing unit 15 to each of the four audio waveform data output from the second convolution processing unit 14, i.e., multistream data (multiple audio waveform data).

[0064] Furthermore, during training, the multistream generation unit 1 outputs parameter update data update(θ) from the convolution processing unit 3. g_ms )(Parameter θ of the multistream generation unit 1 (second convolution unit 14, MRF unit 12, first convolution processing unit 11) g_ms Input the parameter update data (θ), and update the data (θ) g_ms Based on this, the parameter θ of the multistream generation unit 1 (second convolution unit 14, MRF unit 12, first convolution processing unit 11) g_cnv The update process (parameter update process to reduce loss) is performed.

[0065] Furthermore, the configuration of the MRF section of the multistream generation unit 1 (for example, the configuration of the first MRF processing unit 122, the second MRF processing unit 124, etc.) may be implemented, for example, by the technology disclosed in Non-Patent Document 2.

[0066] The upsampling unit 2 receives data D1 (multistream data (multiple audio waveform data)) output from the second activation processing unit 15 of the multistream generation unit 1, and performs, for example, zero-insertion type upsampling on the data D1. The upsampling process then outputs the data after the upsampling process as data D2 (multistream data after upsampling (multiple audio waveform data)) to the convolution processing unit 3.

[0067] The convolution processing unit 3 receives data D2 output from the upsampling unit 2 and performs a one-dimensional convolution process (Conv1D process (unbiased)) on the data D2. The convolution processing unit 3 then outputs the data after the one-dimensional convolution process (Conv1D process) as data Dout to the generated data evaluation unit and selector SEL1.

[0068] The one-dimensional convolution process (Conv1D process (unbiased)) performed by the convolution processing unit 3 is executed with, for example, a kernel size of "63" (equivalent to 63 samples) and a channel count of "1". In other words, the data D2 (multistream data (for example, four audio waveform data)) input to the convolution processing unit 3 is synthesized by the one-dimensional convolution process (Conv1D process (unbiased)) performed by the convolution processing unit 3 and acquired (generated) as a single audio waveform data.

[0069] Furthermore, during training, the convolution processing unit 3 processes the data update(θ) output from the update data selection processing unit G_upd. g_cnv )(Parameter θ of the convolutional layer of the convolutional processing unit 3) g_cnv Input the parameter update data (θ), and update the data (θ) g_cnvBased on this, the parameter θ of the convolutional layer of the convolutional processing unit 3 g_cnv The update process (parameter update process to reduce loss) is performed.

[0070] Furthermore, after performing the above update process, the convolution processing unit 3 generates parameter update data update(θ) to update the parameters of the multistream generation unit 1 (second convolution unit 14, MRF unit 12, first convolution processing unit 11). g_ms ) generates the parameter update data update(θ g_ms The multistream generation unit 1 outputs the output.

[0071] (1.1.2: Generated Data Evaluation Unit G_Ev) The generated data evaluation unit G_Ev receives data Dout output from the audio data generation device 100 and audio waveform data D_correct (ground truth data) corresponding to the input data Din (mel spectrogram data) of the audio data generation device 100 used to generate data Dout. The generated data evaluation unit G_Ev evaluates the error (loss) between data Dout and data D_correct (ground truth data) using an evaluation function (loss function) that evaluates the STFT loss (STFT: short-time Fourier transform). Then, based on the output (result) of the STFT loss evaluation function, the generated data evaluation unit G_Ev evaluates the parameter θ of the learnable function unit (learnable convolutional layer, etc.) of the audio data generation device 100. g The parameter update data is the data used to update the parameter, pre_update(θ g ) generates the parameter update data pre_update(θ g The updated data selection processing unit G_upd outputs this data.

[0072] (1.1.3: Selector SEL1) Selector SEL1 is a 2-input, 1-output switch that takes as input data Dout (data Dout generated by the audio data generator 100 (synthesized data (fake data))) output from the audio data generator 100 and the real data D_correct (for example, audio waveform data D_correct (correct data)) corresponding to the input data Din (mel spectrogram data) of the audio data generator 100 used to generate data Dout).

[0073] The selector SEL1, for example, selects either data Dout or data D_correct according to the selection signal sel1 output from the control unit (not shown), and inputs the selected data as data Dd1 to the audio data identification device Dev_D.

[0074] (1.1.4: Audio data identification device Dev_D) The audio data identification device Dev_D is a discriminator used in adversarial learning, and is the discriminator used when the audio data generator 100 is used as the generator in adversarial learning. As shown in Figure 1, the audio data identification device Dev_D comprises a global feature identification unit DD1 and a detailed feature identification unit DD2.

[0075] As shown in Figure 5, the global feature recognition unit DD1 is equipped with multiple classifiers MSD[k] (in Figure 5, classifiers MSD[1] to MSD[3]) (MSD: Multi-scale Descriminator). For the sake of explanation, the following description will assume that there are three classifiers MSD[k], but the number of classifiers MSD[k] is not limited to three and may be any other number.

[0076] The first classifier MSD[1] inputs the data Dd1 directly to the classifier, as shown in Figure 5. The classifier includes, for example, a convolutional layer MS1, a downsampling layer MS2 (for example, a configuration in which four downsampling layers are connected in succession), a convolutional layer MS3, and a convolutional layer MS4, as shown in Figure 5. The final convolutional layer MS4 outputs result data DD1_MSD_out[1] indicating the truth value of the input data Dd1 (whether the data Dd1 is real data or fake data generated by the audio generation process).

[0077] The second classifier MSD[2] comprises an average pooling layer and a classifier, as shown in Figure 5.

[0078] The mean pooling layer performs mean pooling, outputting the average of two adjacent (time-series adjacent) samples of Dd1.

[0079] The output of the average pooling layer is then input to the discrimination unit of the second discriminator MSD[2].

[0080] The identification unit of the second classifier MSD[2] has the same configuration as the identification unit of the first classifier MSD[1]. The identification unit of the second classifier MSD[2] outputs result data DD1_MSD_out[2] indicating the truth value of the input data Dd1 (whether data Dd1 is real data or fake data generated by the audio generation process).

[0081] The third classifier MSD[3] comprises an average pooling layer and a classifier, as shown in Figure 5.

[0082] The mean pooling layer performs mean pooling, outputting the average of the values ​​of four adjacent (time-series adjacent) samples of Dd1.

[0083] The output of the average pooling layer is then input to the discrimination unit of the third discriminator MSD[3].

[0084] The identification unit of the third classifier MSD[3] has the same configuration as the identification unit of the first classifier MSD[1]. The identification unit of the third classifier MSD[3] outputs result data DD1_MSD_out[3] indicating the truth value of the input data Dd1 (whether data Dd1 is real data or fake data generated by the audio generation process).

[0085] Then, the output data (data DD1_MSD_out[1]~DD1_MSD_out[3]) from the multiple classifiers MSD[k] of the global feature identification unit DD1 is output to the identification data evaluation unit D_Ev.

[0086] Furthermore, the data that combines the output data of multiple classifiers MSD[k] of the global feature identification unit DD1 (data DD1_MSD_out[1] to DD1_MSD_out[3]) is denoted as data Dd1_out.

[0087] The detailed feature identification unit DD2 comprises multiple classifiers MPD[k] (in Figure 6, classifiers MPD[1] to MPD[M]) (MPD: Multi-period Descriminator) (M: natural number), as shown in Figure 6.

[0088] The k-th classifier MPD[k] (k: natural number, 1 ≤ k ≤ M) comprises a reshaping unit and a classifier unit, as shown in Figure 5.

[0089] The reshaping unit divides the data Dd1 (one-dimensional data) into periods of p[k] (each of p[k] samples) and converts it into two-dimensional data (if the number of samples in data Dd1 is T, then two-dimensional data of p[k] × ceil(T / p[k]) (ceil() is the ceiling function)). The reshaping unit then outputs the processed two-dimensional data to the identification unit.

[0090] The identification unit performs a convolution operation on the 2D data output from the reshaping unit and obtains result data DD2_MPD_out[k] indicating the truth value of the input data Dd1 (whether data Dd1 is real data or fake data generated by the audio generation process).

[0091] As shown in Figure 6, the identification unit has a configuration in which four blocks, each containing a 5×1 convolutional layer (stride: (3,1), number of channels: 2^(5+L)) and an activation processing unit (a functional unit that performs activation processing using the Leaky ReLU function), are connected in succession. Further down the block, there is a configuration comprising a 5×1 convolutional layer (number of channels: 1024), an activation processing unit (a functional unit that performs activation processing using the Leaky ReLU function), and a 3×1 convolutional layer (number of channels: 1).

[0092] The identification unit performs convolution and activation processing according to the above configuration to obtain result data DD2_MPD_out[k] indicating the truth value of the input data Dd1 (whether data Dd1 is real data or fake data generated by the audio generation process).

[0093] Then, the output data (data DD2_MPD_out[1]~DD2_MPD_out[M]) from the multiple classifiers MPD[k] of the detailed feature identification unit DD2 is output to the identification data evaluation unit D_Ev.

[0094] Furthermore, the data that combines the output data of multiple classifiers MPD[k] of the detailed feature identification unit DD2 (data DD2_MPD_out[1] to DD2_MPD_out[M]) is denoted as data Dd2_out.

[0095] Furthermore, the global feature identification unit DD1 (classifier MSD[k]) and the detailed feature identification unit DD2 (classifier MPD[k]) may be implemented, for example, by the technologies disclosed in Non-Patent Document 2 and Non-Patent Document 3.

[0096] Furthermore, during training, the audio data discrimination device Dev_D outputs data GAN_update(θ) from the discrimination data evaluation unit D_Ev. d )(Parameter θ of the learnable part (convolutional layer, etc.) of the audio data identification device Dev_D) d Input the parameter update data (θ), and update the data (θ) d Based on this, the parameters θ of the learnable part (convolutional layer, etc.) of the audio data discrimination device Dev_D g_d The update process (parameter update process to reduce loss) is performed.

[0097] (1.1.5: Identification Data Evaluation Unit D_Ev) The identification data evaluation unit D_Ev receives the data Dd1_out and Dd2_out output from the audio data identification device Dev_D, and performs loss evaluation in adversarial learning using the loss function for the generator (corresponding to the audio data generation device 100) and the loss function for the discriminator (corresponding to the audio data identification device Dev_D).

[0098] Then, the identification data evaluation unit D_Ev determines the parameter θ of the learnable functional unit (learnable convolutional layer, etc.) of the audio data generation device 100 based on the results of the loss evaluation described above. g The parameter update data GAN_update(θ) is the data used to update the parameters. g ) generates the parameter update data GAN_update(θ g The updated data selection processing unit G_upd outputs this data.

[0099] Furthermore, the identification data evaluation unit D_Ev determines the parameter θ of the learnable functional unit (learnable convolutional layer, etc.) of the audio data identification device Dev_D based on the results of the loss evaluation described above. d The parameter update data GAN_update(θ) is the data used to update the parameters. d ) generates the parameter update data GAN_update(θ d ) is output to the audio data identification device Dev_D.

[0100] (1.1.6: Update data selection processing unit G_upd) The update data selection processing unit G_upd outputs parameter update data pre_update(θ) from the generated data evaluation unit G_Ev. g ) and parameter update data GAN_update(θ) output from the identification data evaluation unit D_Ev. g Enter )

[0101] Then, the update data selection processing unit G_upd, during pre-training, selects the parameter update data pre_update(θ g ) Select the parameter update data pre_update(θ g ) to parameter update data update(θ g_cnv The output is sent to the convolution processing unit 3 of the audio data generation device 100.

[0102] Furthermore, the update data selection processing unit G_upd, during training (adversarial training), outputs parameter update data GAN_update(θ g Select the parameter update data GAN_update(θ g ) to parameter update data update(θ g_cnv The output is sent to the convolution processing unit 3 of the audio data generation device 100.

[0103] <1.2: Operation of the Audio Data Processing System> The operation of the audio data processing system 1000, configured as described above, will be explained below. In the following, the operation of the audio data processing system 1000 will be explained in two parts: (1) learning process and (2) inference process (prediction process).

[0104] (1.2.1: Learning Process) Figure 7 is a flowchart of the learning process performed by the audio data processing system 1000.

[0105] The learning process performed by the audio data processing system 1000 will be explained below with reference to a flowchart.

[0106] (Step S1): In step S1, the audio data generation device 100 undergoes a pre-training process. Specifically, the following processes are performed.

[0107] The data Din, which is the data of the Mel spectrogram, is input to the first convolution processing unit 11 of the multistream generation unit 1.

[0108] The first convolution processing unit 11 then performs a one-dimensional convolution process (Conv1D process) on the data Din (treating the Mel spectrum data as two-dimensional data and performing a one-dimensional convolution process on it), and outputs the data after the one-dimensional convolution process (Conv1D process) to the MRF unit as data D11. The one-dimensional convolution process (Conv1D process) performed by the first convolution processing unit 11 is performed, for example, with a kernel size of "7" (corresponding to 7 samples) and a number of channels of "512".

[0109] The first upsampling unit 121 receives data D11 output from the first convolution processing unit 11 and performs upsampling on data D11. The first upsampling unit 121 outputs the data after upsampling as data D12 to the first MRF processing unit 122. The upsampling process performed by the first upsampling unit 121 is performed, for example, by increasing the number of samples in the input data to eight times (×8) and setting the number of channels to "256". As for the upsampling method, for example, upsampling by subpixel convolution is employed. That is, the first upsampling unit 121 performs upsampling by setting the kernel size to "3", executing a one-dimensional convolution (Conv1D processing), and then performing a reshape processing. To achieve this, you can increase the number of input data samples by eight (x8) and adjust the number of channels in the one-dimensional convolution (Conv1D) process, as well as the length and number of channels in the reshape process, so that the number of samples becomes "256".

[0110] The data D12 obtained through processing in the first upsampling unit 121 is then output to the first MRF processing unit 122.

[0111] In the first MRF processing unit 122, MRF processing is performed on data D12. Specifically, the following processing is performed.

[0112] Data D12 is the residual block ResBlock[n](1≦n≦|k) of residual block group 1221. r |)(|k r | is array k r The input represents the number of elements (array size) (a block configured as shown in Figure 4), and in block BL2, activation processing is performed by the Leaky ReLU function by the activation processing unit BL21, and k by block BL22 rA convolution operation using a [n]×1 kernel is performed multiple times (|Dr[n,m]| times), and the data after this operation is added to data D12 by adder Add1 (processing of block BL1).

[0113] Then, the above block BL processing is executed multiple times (|Dr[n]| times). The data after this processing is then output to the adder 1222 as data D12_out[n].

[0114] The summing unit 1222 processes the output data D12_out[1]~D12_out[|k] from each block of the residual block group 1221. r The sum of the values ​​is added, and the resulting data is output to the second upsampling unit 123 as data D13.

[0115] Thus, in the first MRF processing unit 122, convolution is performed using residual blocks with kernels corresponding to various receptive fields, and the data resulting from this processing is integrated in the adder 1222. As a result, the data D13 output from the adder 1222 is obtained as data containing features extracted using kernels corresponding to various receptive fields.

[0116] The data D13 obtained through processing in the first MRF processing unit 122 is then output to the second upsampling unit 123.

[0117] The second upsampling unit 123 performs upsampling on the data D13 output from the first MRF processing unit 122. The upsampling process performed by the second upsampling unit 123 is performed, for example, by increasing the number of samples in the input data to eight times (×8) and setting the number of channels to "128". The upsampling method is the same as that of the first upsampling unit, for example, upsampling by subpixel convolution.

[0118] The data D14 obtained through processing in the second upsampling unit 123 is then output to the second MRF processing unit 124.

[0119] The second MRF processing unit 124 has the same configuration as the first MRF processing unit, and performs the same processing on the data D14 output from the second upsampling unit 123 as the first MRF processing unit (note that k r (The setting value of Dr may be different from the setting value of the first MRF processing unit.) Then, the second MRF processing unit 124 outputs the data after processing in the second MRF processing unit 124 as data D15 to the first activation processing unit 13.

[0120] The first activation processing unit 13 performs an activation process using the Leaky ReLU function on the data D15 output from the second MRF processing unit 124 of the MRF unit 12. The first activation processing unit 13 then outputs the data after the activation process as data D16 to the second convolution processing unit 14.

[0121] The second convolution processing unit 14 performs a one-dimensional convolution (Conv1D) on the data D16 output from the first activation processing unit 13. The second convolution processing unit 14 then outputs the data after the one-dimensional convolution (Conv1D) as data D17 to the second activation processing unit 15. The one-dimensional convolution (Conv1D) performed by the second convolution processing unit 14 is performed, for example, with a kernel size of "7" (corresponding to 7 samples) and a number of channels of "4".

[0122] The second activation processing unit 15 performs an activation process using the tanh function on the data D17 output from the second convolution processing unit 14. The second activation processing unit 15 then outputs the activated data as data D1 to the upsampling unit 2. Note that if the number of channels of the second convolution processing unit 14 is "4", data D1 will be audio waveform data (four audio waveform data) obtained by applying the activation process by the second activation processing unit 15 to each of the four audio waveform data output from the second convolution processing unit 14, i.e., multistream data (multiple audio waveform data).

[0123] Then, the data D1 obtained through processing in the second activation processing unit 15 is output from the multistream generation unit 1 to the upsampling unit 2.

[0124] The upsampling unit 2 performs, for example, zero-insertion type upsampling on the data D1 (multistream data (multiple audio waveform data)) output from the second activation processing unit 15 of the multistream generation unit 1. The upsampling process then outputs the data after the upsampling process as data D2 (multistream data after upsampling (multiple audio waveform data)) to the convolution processing unit 3.

[0125] The convolution processing unit 3 performs a one-dimensional convolution (Conv1D processing (unbiased)) on the data D2 output from the upsampling unit 2. The convolution processing unit 3 then outputs the data after the one-dimensional convolution (Conv1D processing) as data Dout to the generated data evaluation unit and selector SEL1.

[0126] The one-dimensional convolution process (Conv1D process (unbiased)) performed by the convolution processing unit 3 is executed with, for example, a kernel size of "63" (equivalent to 63 samples) and a channel count of "1". In other words, the data D2 (multistream data (for example, four audio waveform data)) input to the convolution processing unit 3 is synthesized by the one-dimensional convolution process (Conv1D process (unbiased)) performed by the convolution processing unit 3 and acquired (generated) as a single audio waveform data.

[0127] Then, the data Dout obtained by executing the above process by the audio data generation device 100 is output to the generated data evaluation unit G_Ev.

[0128] The generated data evaluation unit G_Ev receives data Dout output from the audio data generation device 100 and audio waveform data D_correct (correct data) corresponding to the input data Din (mel spectrogram data) of the audio data generation device 100 used to generate the data Dout. The generated data evaluation unit G_Ev evaluates the error (loss) between data Dout and data D_correct (correct data) using an evaluation function (loss function) that evaluates the STFT loss (STFT: short-time Fourier transform).

[0129] Specifically, the generated data evaluation unit G_Ev evaluates the loss using the following STFT loss function.

[0130] Loss function L for the acquisition period (the period over which the FFT is applied) of one STFT sc , L mg We define it as follows:

number

number

number

[0131] The update data selection processing unit G_upd, during the pre-training of the audio data generation device 100, selects the parameter update data pre_update(θ g ) Select the parameter update data pre_update(θ g ) to parameter update data update(θ g_cnv The output is sent to the convolution processing unit 3 of the audio data generation device 100.

[0132] During pre-training, the convolution processing unit 3 processes the data update(θ) output from the update data selection processing unit G_upd. g_cnv )(Parameter θ of the convolutional layer of the convolutional processing unit 3) g_cnvInput the parameter update data (θ), and update the data (θ) g_cnv Based on this, the parameter θ of the convolutional layer of the convolutional processing unit 3 g_cnv The update process (parameter update process to reduce loss) is performed.

[0133] Furthermore, after performing the above update process, the convolution processing unit 3 generates parameter update data update(θ) to update the parameters of the multistream generation unit 1 (second convolution unit 14, MRF unit 12, first convolution processing unit 11). g_ms ) generates the parameter update data update(θ g_ms The multistream generation unit 1 outputs the output.

[0134] During pre-training, the multistream generation unit 1 outputs parameter update data update(θ) from the convolution processing unit 3. g_ms )(Parameter θ of the multistream generation unit 1 (second convolution unit 14, MRF unit 12, first convolution processing unit 11) g_ms Input the parameter update data (θ), and update the data (θ) g_ms Based on this, the parameter θ of the multistream generation unit 1 (second convolution unit 14, MRF unit 12, first convolution processing unit 11) g_cnv The update process (parameter update process to reduce loss) is performed.

[0135] Then, in the audio data processing system 1000, the above process is repeatedly executed (the above process is repeatedly executed while changing the input data Din), and when the STFT evaluation value (STFT loss value) obtained by the generated data evaluation unit G_Ev falls within (1) a predetermined range, or (2) no longer fluctuates above a predetermined value, it is determined that the pre-training process has converged.

[0136] Then, the parameters at which the above pre-training process is determined to have converged are set in the convolution processing unit 3 and the multi-stream generation unit 1 of the audio data generation device 100.

[0137] (Step S2): In step S2, loop processing (loop 1) (adversarial learning processing by audio data generation device 100 and audio data identification device Dev_D) is started.

[0138] (Step S3): In step S3, the parameter update process for the audio data identification device Dev_D is executed. Specifically, the following processes are performed.

[0139] Selector SEL1, for example, selects data Dout (data generated by the audio data generation device 100 (fake data)) according to the selection signal sel1 output from the control unit (not shown), and inputs the selected data as data Dd1 to the audio data identification device Dev_D.

[0140] As shown in Figure 5, the first classifier MSD[1] of the global feature recognition unit DD1 inputs the data Dd1 directly to the recognition unit. The recognition unit of the first classifier MSD[1] performs processing on the data Dd1 using a convolutional layer MS1, a downsampling layer MS2 (for example, a configuration in which four downsampling layers are connected in succession), a convolutional layer MS3, and a convolutional layer MS4, and outputs result data DD1_MSD_out[1] indicating the truth value of the input data Dd1 (whether the data Dd1 is real data or fake data generated by the audio generation process) to the recognition data evaluation unit D_Ev.

[0141] The same processing as described above is performed for classifiers other than the first classifier MSD[1] of the global feature identification unit DD1, and result data DD1_MSD_out[k] indicating the truth value of the input data Dd1 (whether data Dd1 is real data or fake data generated by the audio generation process) is output to the identification data evaluation unit D_Ev.

[0142] Furthermore, the kth classifier MPD[k] (k: natural number, 1≦k≦M) of the detailed feature identification unit DD2 inputs the data Dd1 into the reshaping section of the kth classifier MPD[k], as shown in Figure 5.

[0143] The reshaping section of the k-th classifier MPD[k] divides the data Dd1 (1-dimensional data) into periods of p[k] (every p[k] samples) and converts it into 2-dimensional data (where T is the number of samples in data Dd1, then 2-dimensional data of p[k] × ceil(T / p[k]) (ceil() is the ceiling function)). For example, if p[k]=3 and T=300, the reshaping section of the k-th classifier MPD[k] converts the data Dd1 (1-dimensional data) into 3 × 100 2-dimensional data.

[0144] Then, the reshaping unit of the k-th classifier MPD[k] outputs the processed 2D data to the classifier unit of the k-th classifier MPD[k].

[0145] The discrimination unit of the k-th classifier MPD[k] performs a convolution operation on the 2D data output from the reshaping unit and obtains result data DD2_MPD_out[k] indicating the truth value of the input data Dd1 (whether data Dd1 is real data or fake data generated by the audio generation process).

[0146] The result data DD2_MPD_out[1]~DD2_MPD_out[M] obtained by the multiple classifiers MPD[1]~MPD[M] of the detailed feature identification unit DD2 is output to the identification data evaluation unit D_Ev.

[0147] The identification data evaluation unit D_Ev receives the data Dd1_out (DD1_MSD_out[1]~DD1_MSD_out[3]) and Dd2_out (DD2_MPD_out[1]~DD2_MPD_out[M]) output from the audio data identification device Dev_D, and stores the result data from each identification device, including information on whether or not the determination was made correctly.

[0148] Next, the selector SEL1, for example, according to the selection signal sel1 output from the control unit (not shown), selects the real data D_correct (for example, the audio waveform data D_correct (correct data) (real data) corresponding to the input data Din (mel spectrogram data) of the audio data generation device 100 used to generate data Dout), and inputs the selected data as data Dd1 to the audio data identification device Dev_D. Then, the audio data identification device Dev_D performs the same processing as described above.

[0149] Then, similarly to the above, the identification data evaluation unit D_Ev receives the data Dd1_out (DD1_MSD_out[1]~DD1_MSD_out[3]) and Dd2_out (DD2_MPD_out[1]~DD2_MPD_out[M]) output from the audio data identification device Dev_D, and stores the result data from each identification device.

[0150] Furthermore, the processing performed by the audio data identification device Dev_D is repeated while changing the data input to the audio data identification device Dev_D between real data and fake data.

[0151] The identification data evaluation unit D_Ev obtains the probability that the result data from each classifier is correctly identified (the probability of determining genuine data as genuine and false data as false).

[0152] Then, using the acquired probabilities, the loss of the generator (corresponding to the audio data generation device 100) and the loss of the discriminator (corresponding to the audio data discriminator Dev_D) are evaluated using the adversarial learning loss function for the generator and the loss of the discriminator. Based on these evaluation values, the parameters of the learnable functional parts of the audio data discriminator Dev_D (discriminator) are updated, and the parameters of the learnable functional parts of the audio data generation device 100 are also updated.

[0153] Here, we will describe the adversarial learning loss function for the generator (corresponding to the audio data generator 100) and the adversarial learning loss function for the discriminator (corresponding to the audio data discriminator Dev_D).

[0154] Loss function L of discriminator D, which performs adversarial learning together with generator G. Adv (D;G), and the loss function L of the generator G that performs adversarial learning together with the discriminator D. Adv (G;D) is defined as follows:

number

number

number

Equation

Equation

[0155] The discrimination data evaluation unit \(D\_Ev\) acquires the probability that the result data of each discriminator correctly discriminates (the probability of determining genuine data as genuine and fake data as fake), performs processing corresponding to the above evaluation function, and the loss function \(L\) of adversarial learning of the audio data generation device 100 (generator) G and the loss function \(L\) of adversarial learning of the audio data discrimination device Dev_D (discriminator) D and acquires them.

[0156] And the discrimination data evaluation unit \(D\_Ev\) generates parameter update data \(GAN\_update(\theta\) D ) which is data for updating the parameters \(\theta\) of the learnable functional unit (learnable convolutional layer, etc.) of the audio data discrimination device Dev_D based on the above loss evaluation result (loss function \(L\) d and generates the parameter update data \(GAN\_update(\theta\) d ) for updating the parameters \(\theta\) of the learnable functional unit (learnable convolutional layer, etc.) of the audio data discrimination device Dev_D based on the above loss evaluation result (loss function \(L\) d) is output to the audio data identification device Dev_D.

[0157] The audio data discrimination device Dev_D, during training (adversarial learning), outputs data GAN_update(θ) from the discrimination data evaluation unit D_Ev. d )(Parameter θ of the learnable part (convolutional layer, etc.) of the audio data identification device Dev_D) d Input the parameter update data (θ), and update the data (θ) d Based on this, the parameters θ of the learnable part (convolutional layer, etc.) of the audio data discrimination device Dev_D g_d The update process (parameter update process to reduce loss) is performed.

[0158] (Step S4): In step S3, the parameter update process for the audio data generation device 100 is executed. Specifically, the following processes are performed.

[0159] Similar to step S3, the process performed by the audio data identification device Dev_D in step S3 (the same process as in step S3) is repeated, while the data input to the audio data identification device Dev_D is changed by selector SEL1 to either genuine data (data D_correct) or fake data (data Dout generated by the audio data generation device 100).

[0160] Then, similar to step S3, the identification data evaluation unit D_Ev obtains the probability that each classifier correctly identifies the result data (the probability of determining genuine data as genuine and false data as false).

[0161] Then, using the acquired probabilities, the loss of the generator (corresponding to the audio data generator 100) and the loss of the discriminator (corresponding to the audio data discriminator Dev_D) are evaluated using adversarial learning loss functions, and the parameters of the learnable functional parts of the audio data generator 100 are updated based on these evaluation values.

[0162] Specifically, the identification data evaluation unit D_Ev reduces the loss obtained by the loss function L of the adversarial learning of the generator (corresponding to the audio data generation device 100) when having K discriminators, and uses the parameter θ of the learnable functional unit (learnable convolutional layer, etc.) of the audio data generation device 100 G to generate parameter update data GAN_update(θ g ) which is data for updating, and outputs the parameter update data GAN_update(θ g ) to the update data selection processing unit G_upd. g

[0163] During the learning (adversarial learning) of the audio data generation device 100, the update data selection processing unit G_upd selects the parameter update data GAN_update(θ g ) and outputs the parameter update data GAN_update(θ g ) as parameter update data update(θ g_cnv ) to the convolutional processing unit 3 of the audio data generation device 100.

[0164] During the learning (adversarial learning), the convolutional processing unit 3 inputs the data update(θ g_cnv )(parameter update data of the parameter θ of the convolutional layer of the convolutional processing unit 3) output from the update data selection processing unit G_upd, and based on the data update(θ g_cnv ), performs update processing (parameter update processing to reduce the loss) on the parameter θ of the convolutional layer of the convolutional processing unit 3. g_cnv g_cnv

[0165] Also, after performing the above update processing, the convolutional processing unit 3 generates parameter update data update(θ g_ms ) for updating the parameters of the multi-stream generation unit 1 (the second convolutional unit 14, the MRF unit 12, the first convolutional processing unit 11), and the parameter update data update(θ g_msThe multistream generation unit 1 outputs the output.

[0166] The multistream generation unit 1 outputs parameter update data update(θ) from the convolution processing unit 3 during training (adversarial training). g_ms )(Parameter θ of the multistream generation unit 1 (second convolution unit 14, MRF unit 12, first convolution processing unit 11) g_ms Input the parameter update data (θ), and update the data (θ) g_ms Based on this, the parameter θ of the multistream generation unit 1 (second convolution unit 14, MRF unit 12, first convolution processing unit 11) g_cnv The update process (parameter update process to reduce loss) is performed.

[0167] (Step S5): In step S5, it is determined whether the termination condition of the loop process (loop 1) is met. If it is determined that the termination condition is not met, the process in steps S2 to S4 is repeated.

[0168] On the other hand, if it is determined that the termination condition of the loop process (loop 1) has been met, the learning process is terminated. The termination condition of the loop process (loop 1) is met when it can be determined that adversarial learning has converged, for example, in the following cases. (1) The discriminant data evaluation unit D_Ev determines the loss function L of the adversarial learning of the generator (corresponding to the audio data generation device 100). G The value of converges within a predetermined range, and the loss function L of the adversarial learning of the classifier (corresponding to the audio data classifier Dev_D) D When the value converges within a predetermined range. (2) The discrimination data evaluation unit D_Ev determines the loss function L of the adversarial learning of the generator (corresponding to the audio data generation device 100). G The change in the value of is within a predetermined range, and the loss function L of the adversarial learning of the classifier (corresponding to the audio data classifier Dev_D) is within a predetermined range. D When the amount of change in the value is within a predetermined range.

[0169] If the above termination conditions are met, the audio data processing system 1000 terminates the learning process and acquires the audio data generation device 100 with the parameters set in the audio data generation device 100 at the time the learning process is completed (parameters set in the learnable functional unit) as the optimal parameters, and the audio data generation device 100 with these optimal parameters set as the learned audio data generation device 100.

[0170] Then, the data of a predetermined Mel spectrogram is input as data Din to the trained audio data generator 100 (an audio data generator 100 with optimal parameters set), and by processing it with the trained audio data generator 100, audio waveform data Dout corresponding to the input Mel spectrogram is obtained.

[0171] The trained audio data generation device 100 is capable of very high-speed audio data generation processing and can generate very high-precision audio data (audio waveform data).

[0172] In the trained audio data generation device 100, the multistream generation unit 1 acquires multiple stream data (multistream data, for example, four data-driven decomposition data (audio waveform data)) as data D1. The acquired data D1 is subjected to zero-insertion upsampling processing by the upsampling unit 2. Furthermore, the convolution processing unit 3 performs convolution processing (Conv1D processing, unbiased) on the upsampled data (for example, four upsampled data-driven decomposition data (audio waveform data)). In other words, since the trained audio data generation device 100 acquires multiple stream data (multistream data), its configuration can be simplified. Moreover, audio waveform data can be generated by simply performing (1) upsampling processing (after zero-insertion upsampling processing) and (2) convolution processing by the convolution processing unit 3 (processing equivalent to applying an FIR filter to multiple stream data and synthesizing them) on the multiple stream data (multistream data).

[0173] In other words, since the pre-trained audio data generation device 100 performs processing with a simple configuration, it is possible to process it using a CPU without using a GPU capable of high-speed processing.

[0174] Furthermore, the trained audio data generator 100 includes a global feature identification unit DD1 that identifies global features and a detailed feature identification unit DD2 that identifies detailed features. By obtaining optimal parameters through adversarial learning using the audio data identification device Dev_D, which has extremely powerful identification capabilities, it is possible to generate highly accurate audio data (audio waveform data).

[0175] Furthermore, the audio data generation device 100 performs adversarial learning with the audio data identification device Dev_D using audio waveform data (data Dout) obtained by upsampling multiple stream data acquired by the multi-stream generation unit 1 and performing convolution processing. Therefore, unlike conventional technology (Multi-band MelGAN), there is no constraint that requires the use of subband signals, making it possible to efficiently advance adversarial learning.

[0176] As described above, the audio data generation device 100 has a configuration in which the multi-stream generation unit 1 acquires multiple stream data (for example, four data-driven decomposition data (audio waveform data)), and furthermore, by introducing a learnable convolutional processing unit 3, adversarial learning with the high-precision audio data discrimination device Dev_D becomes possible. Then, the audio data generation device 100 acquired through this adversarial learning can perform high-speed and high-precision audio data generation processing. Furthermore, because the audio data generation device 100 has a simple configuration, it can achieve high-quality audio data generation processing (for example, speech synthesis processing) at high speed without using a high-speed GPU.

[0177] [Other embodiments] In the above embodiment, the case in which the audio data generation device 100 is configured with the multi-stream generation unit 1 based on the HiFi-GAN configuration has been described, but it is not limited to this, and for example, the multi-stream generation unit 1 may be configured with the Multi-band MelGAN configuration (upsampling processing block, residual block configuration).

[0178] Furthermore, the loss function shown in the above embodiment as the loss function used for adversarial learning is just one example, and other loss functions may be used to perform adversarial learning in the audio data processing system 1000.

[0179] Furthermore, the audio data generation device 100 (a trained audio data generation device 100) of the above embodiment may be connected to, for example, an audio data processing system that generates Mel spectrograms from text data, thereby realizing a speech synthesis system (TTS system, TTS: Text-to-Speech).

[0180] Furthermore, in the audio data processing system 1000, audio data generation device 100, and audio data identification device Dev_D described in the above embodiment, each block may be individually integrated into a single chip using semiconductor devices such as LSIs, or it may be integrated into a single chip including some or all of the blocks.

[0181] Although we have used the term LSI here, depending on the degree of integration, they may also be called IC, system LSI, super LSI, or ultra LSI.

[0182] Furthermore, the method of integrated circuit implementation is not limited to LSIs; it may also be implemented using dedicated circuits or general-purpose processors. FPGAs (Field Programmable Gate Arrays), which can be programmed after LSI manufacturing, or reconfigurable processors, which allow for the reconfiguration of the connections and settings of the circuit cells within the LSI, may also be used.

[0183] Furthermore, some or all of the processing of each functional block in each of the above embodiments may be implemented by a program. And some or all of the processing of each functional block in each of the above embodiments is performed by the central processing unit (CPU) in a computer. The programs for each of these processes are stored in a storage device such as a hard disk or ROM, and are read from the ROM or RAM and executed.

[0184] Furthermore, each of the processes in the above embodiment may be implemented by hardware, or by software (including cases where it is implemented together with an OS (operating system), middleware, or a predetermined library). It may also be implemented by a combination of software and hardware processing.

[0185] For example, when each functional part of the above embodiment is implemented by software, the hardware configuration shown in Figure 8 (for example, a hardware configuration in which a CPU, GPU, ROM, RAM, input unit, output unit, communication unit, storage unit (for example, a storage unit implemented by an HDD, SSD, etc.), and an external media drive are connected by a bus) may be used to implement each functional part by software processing.

[0186] Furthermore, when each of the functional units of the above embodiment is implemented by software, the software may be implemented using a single computer having the hardware configuration shown in Figure 8, or it may be implemented using distributed processing with multiple computers.

[0187] Furthermore, the execution order of the processing method in the above embodiments (including modified examples) is not necessarily limited to the description of the above embodiments, and the execution order can be changed without departing from the spirit of the invention.

[0188] A computer program that causes a computer to execute the method described above, and a computer-readable recording medium on which such program is recorded, are included in the scope of the present invention. Examples of computer-readable recording media include flexible disks, hard disks, CD-ROMs, MOs, DVDs, DVD-ROMs, DVD-RAMs, high-capacity DVDs, next-generation DVDs, and semiconductor memory.

[0189] The above-mentioned computer program is not limited to being recorded on the above-mentioned recording medium, but may also be transmitted via telecommunications lines, wireless or wired communication lines, networks such as the Internet, etc.

[0190] Furthermore, in this specification and in the claims, "optimization" means bringing something to its best state, and the parameters for "optimizing" a system (model) are the parameters at which the value of the system's objective function becomes the optimal value. The "optimal value" is the maximum value when the system performs better as the value of the system's objective function increases, and the minimum value when the system performs better as the value of the system's objective function decreases. The "optimal value" may also be an extremum. The "optimal value" may also allow for a predetermined error (measurement error, quantization error, etc.) and may be a value that falls within a predetermined range (a range that can be considered to have converged sufficiently).

[0191] It should be noted that the specific configuration of the present invention is not limited to the embodiments (including modified examples) described above, and various changes and modifications are possible without departing from the spirit of the invention. [Explanation of Symbols]

[0192] 1000 Audio Data Processing System 100 Audio data generation device 1. Multistream generation unit 2. Upsampling section 3. Convolution Processing Unit Dev_D Audio Data Identifier D_Ev Identification Data Evaluation Unit G_Ev Data Generation Evaluation Unit

Claims

1. A multi-stream generation unit that includes a learnable functional unit and acquires multiple stream data from Mel spectrogram data, An upsampling unit that performs upsampling on each of the aforementioned multiple stream data to acquire upsampling multistream data, A convolution processing unit capable of learning parameters for determining the convolution process, the convolution processing unit that acquires audio waveform data by performing a convolution process on the upsampled multistream data, An audio data generation device equipped with, A global feature discriminator that includes a learnable functional unit and identifies the truth value of audio data based on the global features of the audio data, A detailed feature discriminator that includes a learnable functional unit and identifies the truth or falsity of audio data based on the detailed features of the audio data, An adversarial learning method for an audio data generation device, which is performed using an audio data identification device comprising: An identification step in which the audio data generated by the audio data generation device, or the correct data of said audio data, is input to the audio data identification device, and the audio data identification device is used to identify the truth or falsity of the input data; A loss evaluation step is performed to obtain loss evaluation data using a loss function based on the result data of the identification step, A generator parameter update step, in which the parameters of the convolution processing unit of the audio data generation device and the parameters of the learnable function unit of the multistream generation unit are updated based on the loss evaluation data obtained in the loss evaluation step, A discriminator parameter update step is performed to update the parameters of the learnable functional unit of the global feature discriminator of the audio data discriminator, based on the loss evaluation data obtained in the loss evaluation step, and to update the parameters of the learnable functional unit of the detailed feature discriminator of the audio data discriminator, An adversarial learning method for an audio data generation device equipped with [a specific feature / function].

2. An audio data generation device that can be used together with the audio data identification device described in Claim 1 to perform the adversarial learning method described in Claim 1, and is capable of performing inference processing, A multi-stream generation unit for audio data generation, which includes a function unit with configurable parameters and acquires multiple stream data from Mel spectrogram data, An upsampling unit that performs upsampling on each of the aforementioned multiple stream data to acquire upsampling multistream data, A convolution processing unit capable of setting parameters for determining the convolution process, the convolution processing unit that acquires audio waveform data by performing a convolution process on the upsampled multistream data, Equipped with, Each of the plurality of stream data acquired by the multi-stream generation unit for audio data generation is stream data corresponding to a signal with a lower rate than the audio waveform data. The aforementioned multi-stream generation unit for generating audio data is: It includes a multistream generation convolution processing unit that performs convolution processing using a predetermined kernel, an MRF processing unit that performs MRF processing to acquire diverse data by adding the result data of dilation convolution processing using multiple kernel sizes and multiple types of dilation rates, a multistream generation upsampling processing unit that performs upsampling processing, and a multistream generation activation processing unit that performs processing using a predetermined activation function. When the audio data generation device is used to perform the adversarial learning method for the audio data generation device described in claim 1, the parameters of the multi-stream generation unit for audio data generation are updated based on the loss evaluation data obtained by the loss function. When the inference process is performed, the adversarial learning method for the audio data generation device described in claim 1 is executed, and when the learning process by the adversarial learning method is completed, the optimal parameters for multistream generation, which are parameters set in the multistream generation unit, are set as parameters for the audio data generation multistream generation unit, and with these parameters set, the process of acquiring the plurality of stream data from the Mel spectrogram data is executed. The aforementioned convolution processing unit is This performs a convolution operation that results in an output channel of "1". When the audio data generation device is used to perform the adversarial learning method for the audio data generation device described in claim 1, the parameters of the convolution processing unit are updated based on the loss evaluation data obtained by the loss function. When the inference processing is performed, the adversarial learning method for the audio data generation device described in claim 1 is executed, and when the learning process by the adversarial learning method is completed, the optimal parameters for the convolution processing, which are parameters set in the convolution processing unit, are set, and the convolution processing is executed with the optimal parameters for the convolution processing set. Audio data generation device.

3. The aforementioned convolution processing unit performs a convolution process without bias. The audio data generation apparatus according to claim 2.

4. The upsampling unit performs zero-insertion type upsampling. The audio data generation apparatus according to claim 2 or 3.

5. A learning method for an audio data generation device according to any one of claims 2 to 4, An STFT loss evaluation step in which the loss between the audio data corresponding to the Mel spectrogram input to the audio data generation device and the generated audio data generated from the input Mel spectrogram in the audio data generation device is evaluated using a Short-Time Fourier Transform loss function, A generator parameter update step, in which the parameters of the convolution processing unit of the audio data generation device and the parameters of the learnable function unit of the multistream generation unit are updated based on the evaluation results in the STFT loss evaluation step, A learning method for an audio data generation device equipped with the following features.

6. An audio processing unit that outputs Mel spectrum data from text data, An audio data generation device according to any one of claims 2 to 4, A speech synthesis processing system equipped with the following features.

Images