Audio processing device, audio processing method, and audio processing program

Abstract

An audio processing device (1) includes a merging unit (121), a prediction model (15), and an optimization unit (162). The merging unit (121) converts a multi-stage acoustic token A that represents each frame of an audio waveform as a plurality of values into a merged acoustic token z that represents each frame of the audio waveform as a single value. The prediction model (15) uses a model to predict a merged acoustic token ^z that represents each frame of the audio waveform as a single value on the basis of the audio waveform. The optimization unit (162) optimizes the model to reduce the difference between the merged acoustic token z and the merged acoustic token ^z.

Description

Speech processing device, speech processing method, and speech processing program

[0001] The present invention relates to a speech processing device, a speech processing method, and a speech processing program.

[0002] The Neural Audio Codec (NAC) proposed in Non-Patent Documents 1 and 2 is a waveform generator based on residual vector quantization. It uses multi-stage vector quantization to represent the general shape and details of sound using discrete acoustic tokens in each stage. Because NAC is a powerful model capable of representing any sound, not just speech, it is used not only for speech generation (e.g., speech synthesis, voice conversion) but also as a waveform generator for music and ambient sound generation (see, for example, Non-Patent Documents 3-7).

[0003] Models that utilize NAC as a waveform generator consist of two models: one that first predicts the first stage of acoustic tokens representing the general shape of the sound (coarse prediction), and another model that uses this to predict the remaining multi-stage acoustic tokens from another model (fine prediction). To achieve high-precision predictions, the fine prediction model employs a non-autoregressive model that can represent detailed sound information by considering all past and future time-series information. Non-patent documents 8-10 propose a method for constructing NAC with fewer stages while maintaining quality in order to reduce the computational cost during inference of the fine prediction model.

[0004] One advantage of using NAC is that it can convert audio waveforms into short discrete tokens. If the output is discrete tokens, then a neural transducer (RNNT: Recurrent Neural Network Transducer), which is useful for tasks that convert input sequences to output sequences left-to-right, can be used for acoustic token prediction (see, for example, Non-Patent Document 1).

[0005] In the field of speech recognition, RNNTs are known to be robust to omissions and repetitions of predicted tokens, unlike encoder-decoder models based on attention mechanisms (see, for example, Non-Patent Document 12). However, when RNNTs are used for speech generation tasks such as speech synthesis and voice conversion, neural transducers do not support the prediction of multi-stage discrete tokens. For this reason, the technique described in Non-Patent Document 13 employs a two-model configuration of coarse and fine prediction for acoustic token prediction, similar to other cited documents that use NACs for sound generation. In the technique described in Non-Patent Document 13, the first stage acoustic token is predicted using RNNTs, and the remaining multi-stage acoustic tokens are obtained using a non-autoregressive model.

[0006] Zeghidour, Neil, et al. "SoundStream: An end-to-end neural audio codec." IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 30 pp. 495-507, 2021Defossez, Alexandre, et al. "High fidelity neural audio compression." Transactions on Machine Learning Research, isnn. 2835-8856, 2023.Borsos, Zalan, et al. "AudioLM: a language modeling approach to audio generation." IEEE / ACM transactions on audio, speech, and language processing, vol. 31, pp. 2523-2533, 2023Wang, Chengyi, et al. "Neural codec language models are zero-shot text to speech synthesizers." arXiv preprint arXiv:2301.02111, 2023.Shen, Kai, et al. "Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers." Proc. ICLR, 2024.Wang, Zhichao, et al. "LM-VC: Zero-shot voice conversion via speech generation based on language models." IEEE Signal Processing Letters, vol.30, pp.1157-1161, 2023.Agostinelli, Andrea, et al. "MusicLM: Generating music from text." arXiv preprint arXiv:2301.11325, 2023.Yang, Dongchao, et al. "Hifi-codec: Group-residual vector quantization for high fidelity audio codec." arXiv preprint arXiv:2305.02765, 2023.Wu, Yi-Chiao, et al. "Audiodec: An open-source streaming high-fidelity neural audio codec." Proc. ICASSP, pp. 1-5, 2023.Ai, Yang, et al. "APCodec: A Neural Audio Codec with Parallel Amplitude and Phase Spectrum Encoding and Decoding." IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp.3256-3269, 2024.Graves, Alex. "Sequence transduction with recurrent neural networks.” Proc. ICML Representation Learning Workshop, 2012.Gulati, A., et al. “Conformer: Convolution-augmented Transformer for Speech Recognition.” Proc. Interspeech, pp. 5036-5040, 2020Du, Chenpeng, et al. "VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speech." arXiv preprint arXiv:2401.14321, 2024.Hiroki Kanagawa, et al.“VC-T: Streaming Voice Conversion Based on Neural Transducer,” Proc. Interspeech, pp. 2088-2092, 2023.

[0007] However, conventional technologies may not be able to efficiently predict acoustic tokens.

[0008] For example, when using two models—coarse and fine prediction—to predict acoustic tokens in NAC, two decoding steps are required. In particular, the entire time series must be considered once to drive the fine prediction. Therefore, it is difficult to operate in tasks that require sequential inference from the beginning as the input is given, without waiting for the end (e.g., incremental speech synthesis, streaming voice conversion).

[0009] Furthermore, even if sounds can be represented by discrete tokens thanks to NAC, RNNT cannot be used because it does not support multi-stage token prediction.

[0010] To solve the above-mentioned problems and achieve the objective, the audio processing device according to the present invention is characterized by comprising: a merging unit that converts a first token, in which each frame of the audio waveform is represented by multiple values, into a second token, in which each frame of the audio waveform is represented by a single value; a prediction unit that predicts a third token, in which each frame of the audio waveform is represented by a single value, using a model based on the audio waveform; and an optimization unit that optimizes the model so that the difference between the second tokens becomes small.

[0011] According to the present invention, it becomes possible to efficiently predict acoustic tokens.

[0012] Figure 1 is a diagram showing an example configuration of the audio processing device according to the first embodiment. Figure 2 is a diagram illustrating a method for merging acoustic tokens. Figure 3 is a diagram showing an example of a correspondence table between tokens. Figure 4 is a flowchart showing the processing flow of the audio processing device according to the first embodiment. Figure 5 is a diagram showing an example configuration of the audio processing device according to the second embodiment. Figure 6 is a flowchart showing the processing flow of the audio processing device according to the second embodiment. Figure 7 is a diagram showing an example configuration of the audio processing device according to the third embodiment. Figure 8 is a flowchart showing the processing flow of the audio processing device according to the third embodiment. Figure 9 is a diagram showing an example configuration of the audio processing device according to the fourth embodiment. Figure 10 is a flowchart showing the processing flow of the audio processing device according to the fourth embodiment. Figure 11 is a diagram showing an example of a computer that executes an audio processing program.

[0013] Hereinafter, one embodiment of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to this embodiment. Furthermore, in the drawings, the same parts are denoted by the same reference numerals.

[0014] [First Embodiment] Figure 1 is a diagram showing an example configuration of a speech processing device according to the first embodiment. In the first embodiment, the model learning process performed by the speech processing device is described. The model outputs acoustic tokens. The speech processing also includes processing related to various acoustic tasks such as speech synthesis, voice conversion, music generation, and ambient sound generation.

[0015] Here, the speech processing device 1 performs the task of speech synthesis. The speech processing device 1 trains a model to obtain acoustic tokens for controlling the text to an intended speaker (outputting it in the voice of the intended speaker). For example, according to the acoustic tokens, it is possible to obtain a voice that mimics the speaker shown in the speech waveform y, and reads the text aloud. The speech waveform may be information that includes the frequency components of the voice at each time point. The speech waveform may also be referred to as speech, sound, speech signal, or acoustic signal, etc.

[0016] As shown in Figure 1, the speech processing device 1 comprises a multi-stage acoustic token extraction unit 11, an acoustic token merging unit 12, a text analysis unit 13, a speaker vector generation unit 14, a prediction model 15, and a learning unit 16.

[0017] The multi-stage acoustic token extraction unit 11 includes a conversion unit 111. The conversion unit 111 converts the input audio waveform y into multi-stage acoustic tokens A. First, the audio waveform y corresponding to the text is input to the conversion means from audio waveform to multi-stage acoustic tokens to obtain multi-stage acoustic tokens A. For example, the conversion unit 111 is the encoder part of NAC. The multi-stage acoustic tokens A are a multi-stage discrete token sequence based on residual vector quantization, with Q stages (number of stages of discrete tokens) and U frames, each containing K-class symbols. Here, K, Q, and U are all positive integers. For example, each frame corresponds to a time point in the audio waveform.

[0018] The acoustic token merging unit 12 includes a merging unit 121. The merging unit 121 merges the multi-stage acoustic tokens A received from the multi-stage acoustic token extraction unit 11, thereby creating a merged acoustic token z = [z u=1 ,…,z u=U Obtain ].

[0019] The merging method by the merging unit 121 will be explained using Figure 2. Figure 2 is a diagram illustrating the method of merging acoustic tokens. In the example in Figure 2, K=64 and Q=2. The correspondence between the multi-stage acoustic token A and the merged acoustic token z in this case is shown in the table in Figure 3. Figure 3 is a diagram showing an example of a correspondence table between tokens. The table in Figure 3 may be stored in the audio processing device 1.

[0020] The merging unit 121 may calculate the merged acoustic token without using a table. For example, let A_q be the value of the Qth stage of the multi-stage acoustic token. In this case, A_q is the integer part of the quotient obtained by dividing the merged acoustic token z by K. Furthermore, A_q-1 is the integer part of the quotient obtained by dividing the remainder by K. By repeating this process of dividing the remainder by K and taking the integer part of the quotient, the value of the multi-stage acoustic token can be obtained. The merging unit 121 can perform merging using this relationship between the multi-stage acoustic token and the merged acoustic token. For example, if K=64 and Q=2, the merging unit 121 calculates z=(A_2×64)+A_1.

[0021] The merge unit 121 combines the multi-stage acoustic token A into a single-stage (1 stage) K Q Merge into the class token. Furthermore, the merge unit 121 is a single-stage K Q The class token is given a whitespace symbol and an End Of String (EOS) symbol for learning and inference with RNNT. As a result, the merge unit 121 is a single-stage K Q The total number of tokens for the class is K Q Convert to a +2 class, U+1 frame token.

[0022] For example, if the multi-stage acoustic token A of a frame containing an audio waveform y is "(63,2)", the merge unit 121 refers to a record in the table where the first row is "63" and the second row is "2", and converts the acoustic token of that frame to "255". Note that the merged acoustic tokens z in the table may be set so that they do not overlap between records.

[0023] Furthermore, if the number of merged acoustic token classes is at most around 5,000, it can withstand streaming processing from a computational complexity standpoint. In the example in Figure 2, the number of merged acoustic token classes is approximately 4,096.

[0024] The speaker vector generation unit 14 obtains a speaker vector s from the speech waveform y. Here, the speaker vector s is a fixed-length vector. If the speaker to be synthesized (the speaker shown in the speech waveform y) is one of several specific people, the speaker vector s may be a 1-hot vector. If the speaker to be synthesized is unspecified, the speaker vector s may be an i-vector or x-vector derived from speaker recognition or speaker matching.

[0025] The text analysis unit 13 converts the text into text tokens c=[c_(t=1),…,c_(t=T)]. Here, the text analysis unit 13 converts the text into time-series data of length T. For example, the text analysis unit 13 performs processes to divide a sentence into morphemes or words, and also processes to convert the text into characters or phonemes.

[0026] The speaker vector generation unit 14 inputs the speaker vector s to the prediction model 15. The text analysis unit 13 also inputs the text token c to the prediction model 15.

[0027] The prediction model 15 predicts the merged acoustic token ^z (with ^ directly above z) using a model based on a vector representing speaker features generated from the speech waveform and content tokens, which are data containing sentence elements based on the speech waveform. The encoder and decoder in the prediction model 15 are implemented by a neural network.

[0028] The prediction model 15 predicts the merged acoustic token ^z using a text token encoder 151, a merged acoustic token encoder 152, and a merged acoustic token generation decoder 153, which are learnable modules of an RNNT.

[0029] The text token encoder 151 encodes text token c. The merged acoustic token encoder 152 encodes merged acoustic token z or merged acoustic token ^z. The merged acoustic token generation decoder 153 decodes the data encoded by the text token encoder 151 and the data encoded by the merged acoustic token encoder 152.

[0030] The merged acoustic token generation decoder 153 is input with speaker vector s for speaker conditioning. During training, following teacher-forcing, the merged acoustic token encoder 152 is input with the correct merged acoustic token z.

[0031] In Teacher-forcing, the encoder 152 for merged acoustic tokens receives the correct merged acoustic token z as input during training, and the previously predicted merged acoustic token ^z as input during inference. This is known to cause a mismatch between training and inference, leading to a degradation in the performance of the encoder 152. In contrast, this embodiment reduces this mismatch by using a scheduled sampling algorithm.

[0032] The learning unit 16 includes a prediction error calculation unit 161 and an optimization unit 162. The prediction error calculation unit 161 calculates the difference between the merged acoustic token z and the predicted merged acoustic token ^z as the prediction error. The optimization unit 162 updates (optimizes) the parameters of each module of the prediction model 15 (text token encoder 151, merged acoustic token encoder 152, merged acoustic token generation decoder 153) so that the prediction error is reduced. The optimization unit 162 can update each parameter by backpropagation of the neural network.

[0033] The processing flow of the audio processing device 1 according to the first embodiment will be explained using Figure 4. Figure 4 is a flowchart showing the processing flow of the audio processing device according to the first embodiment.

[0034] As shown in FIG. 4, first, the audio processing device 1 extracts multi-stage acoustic tokens from the audio waveform (step S101). Next, the audio processing device 1 merges the multi-stage acoustic tokens to obtain the correct merged acoustic tokens (step S102).

[0035] Here, the audio processing device 1 extracts text tokens from the text (step S103). Also, the audio processing device 1 generates a speaker vector from the audio waveform (step S104).

[0036] Then, the audio processing device 1 predicts the merged acoustic tokens using the model based on the text tokens and the speaker vector (step S105). Further, the audio processing device 1 calculates the error between the predicted acoustic tokens and the correct merged acoustic tokens (step S106), and optimizes the model so that the error becomes small (step S107).

[0037] As described so far, in the first embodiment, the merge unit 121 converts the multi-stage acoustic token A representing each frame of the audio waveform with a plurality of values into the merged acoustic token z representing each frame of the audio waveform with one value. Also, the prediction model 15 predicts the merged acoustic token ^z representing each frame of the audio waveform with one value using the model based on the audio waveform. The optimization unit 162 optimizes the model so that the difference between the merged acoustic token z and the merged acoustic token z becomes small.

[0038] Note that the prediction model 15 is an example of a prediction unit. The multi-stage acoustic token A is an example of the first token. The merged acoustic token z is an example of the second token. The predicted merged acoustic token ^z is an example of the third token.

[0039] Also, the merge unit 121 refers to a table associating a combination of a plurality of values representing each frame of the audio waveform with one value representing each frame of the audio waveform, and converts the multi-stage acoustic token A into the merged acoustic token z.

[0040] In the conventional technology that uses a NAC as an audio waveform generator (see, for example, Non-Patent Documents 3, 5, 6, 7, and 13), two types of acoustic token models were used to predict multi-stage acoustic tokens.

[0041] The first embodiment enables the prediction of acoustic tokens based on an input using one model. Also, by expressing acoustic tokens in a single stage, the RNNT can be used for the prediction model 15, enabling robust prediction of acoustic tokens against repetition and dropout. Thus, according to the first embodiment, efficient prediction of acoustic tokens becomes possible. Also, for example, when the prediction model 15 is combined with a huge generative language model, the generated text result can be seamlessly converted into speech.

[0042] [Second Embodiment] The speech processing apparatus of the second embodiment performs an inference process using the prediction model learned according to the first embodiment. In the embodiments after the second, explanations of matters common to the first embodiment will be omitted as appropriate.

[0043] FIG. 5 is a diagram showing a configuration example of the speech processing apparatus according to the second embodiment. As shown in FIG. 5, the speech processing apparatus 2 includes a text analysis unit 23, a speaker vector generation unit 24, a prediction model 25, an acoustic token expansion unit 27, and an audio waveform generation unit 28.

[0044] The text analysis unit 23 obtains a text token c through the same process as the text analysis unit 13. Also, the speaker vector generation unit 24 obtains a speaker vector s from the audio waveform y of the target speaker through the same process as the speaker vector generation unit 14. Tgt from the audio waveform y of the target speaker Tgt to obtain a speaker vector s.

[0045] The prediction model 25 predicts a merged acoustic token ^z that represents each frame of the audio waveform with one value, using a model based on the audio waveform. The predicted merged acoustic token ^z is an example of the fourth token.

[0046] The text token encoder 251, the merged acoustic token encoder 252, and the merged acoustic token generation decoder 253 of the prediction model 25 each perform the same processing as the text token encoder 151, the merged acoustic token encoder 152, and the merged acoustic token generation decoder 153.

[0047] The acoustic token expansion unit 27 includes an expansion unit 271. The expansion unit 271 expands the merged acoustic token ^z predicted by the model into a predicted multi-stage acoustic token ^A that represents each frame of the speech waveform with a plurality of values. The predicted multi-stage acoustic token ^A is an example of the fifth token. That is, the expansion unit 271 refers to the table in FIG. 3 and performs the inverse transformation (reverse transformation) of the transformation in the first embodiment.

[0048] The speech waveform generation unit 28 has a conversion unit 281. The conversion unit 281 converts the predicted multi-stage acoustic token ^A into a predicted speech waveform ^y Tgt For example, the conversion unit 281 performs the conversion using the decoder part of NAC.

[0049] Using FIG. 6, the processing flow of the speech processing device 2 of the second embodiment will be described. FIG. 6 is a flowchart showing the processing flow of the speech processing device according to the second embodiment.

[0050] As shown in FIG. 6, first, the speech processing device 2 extracts text tokens from the text (step S201). Also, the speech processing device 2 generates a speaker vector from the speech waveform (step S202).

[0051] Subsequently, the speech processing device 2 predicts merged acoustic tokens using the model based on the text tokens and the speaker vector (step S203).

[0052] Here, the speech processing device 2 expands the predicted merged acoustic tokens into multi-stage acoustic tokens (step S204). Then, the speech processing device 2 generates a speech waveform from the multi-stage acoustic tokens (step S205). After that, the speech processing device 2 outputs the generated speech waveform (step S206).

[0053] [Third Embodiment] In the third embodiment and the fourth embodiment described later, content tokens are used instead of the text tokens described above. That is, the prediction model predicts margined acoustic talk based on speaker vectors and content tokens.

[0054] Text tokens are time-series data converted from text, consisting of elements such as morphemes, words, characters, and phonemes from the text. Content tokens, on the other hand, are data obtained from audio waveforms through speech recognition, and consist of elements such as words, characters, and phonemes. Content tokens do not need to be time-series data. Furthermore, the elements of content tokens may be obtained through speech recognition of audio waveforms.

[0055] Figure 7 shows an example of the configuration of a speech processing device according to the third embodiment. As shown in Figure 7, the speech processing device 3 includes a multi-stage acoustic token extraction unit 31, an acoustic token merging unit 32, a content token generation unit 33a, a speaker vector generation unit 34, a prediction model 35, and a learning unit 36.

[0056] The multi-stage acoustic token extraction unit 31 includes a conversion unit 311. The conversion unit 311 obtains multi-stage acoustic tokens A from the audio waveform y by the same process as the conversion unit 111.

[0057] The acoustic token merging unit 32 includes a merging unit 321. The merging unit 321 obtains merged acoustic tokens z from multi-stage acoustic tokens A by the same process as the merging unit 121.

[0058] The content token generation unit 33a generates content tokens c from the audio waveform y. For example, the content token generation unit 33a may perform discrete tokenization of the intermediate representation of a self-supervised learning model that takes the audio waveform y as input by performing vector quantization on the intermediate representation. The content token generation unit 33a outputs the discrete tokens obtained here as content tokens c.

[0059] The speaker vector generation unit 34 generates a speaker vector s from the audio waveform y by the same process as the speaker vector generation unit 14. Tgt To obtain.

[0060] The prediction model 35 predicts the merged acoustic token ^z using the model, based on a vector representing speaker features generated from the speech waveform and text tokens, which are time-series data containing sentence elements based on text.

[0061] The encoder 352 for merged acoustic tokens and the decoder 353 for generating merged acoustic tokens of the prediction model 35 perform the same processing as the encoder 152 for merged acoustic tokens and the decoder 153 for generating merged acoustic tokens, respectively. The encoder 351a for content tokens encodes content token c.

[0062] Here, the merged acoustic token generation decoder 153 decodes the data encoded by the text token encoder 151 and the data encoded by the merged acoustic token encoder 152. Meanwhile, the merged acoustic token generation decoder 353 decodes the data encoded by the content token encoder 351a and the data encoded by the merged acoustic token encoder 352.

[0063] The learning unit 36 ​​includes a prediction error calculation unit 361 and an optimization unit 362. The prediction error calculation unit 361 and the optimization unit 362 perform the same processing as the prediction error calculation unit 161 and the optimization unit 162, respectively.

[0064] The processing flow of the audio processing device 3 according to the third embodiment will be explained using Figure 8. Figure 8 is a flowchart showing the processing flow of the audio processing device according to the third embodiment.

[0065] As shown in Figure 8, first, the audio processing device 3 extracts multi-stage acoustic tokens from the audio waveform (step S301). Next, the audio processing device 3 merges the multi-stage acoustic tokens to obtain the correct merged acoustic token (step S302).

[0066] Here, the audio processing device 3 extracts content tokens from the audio waveform (step S303). The audio processing device 3 also generates a speaker vector from the audio waveform (step S304).

[0067] Then, the speech processing device 3 predicts merged acoustic tokens using a model based on the content tokens and speaker vectors (step S305). Furthermore, the speech processing device 3 calculates the error between the predicted acoustic tokens and the correct merged acoustic tokens (step S306) and optimizes the model to minimize the error (step S307).

[0068] [Fourth Embodiment] The speech processing device of the fourth embodiment performs inference processing using the predictive model trained according to the third embodiment.

[0069] Figure 9 shows an example of the configuration of the speech processing device according to the fourth embodiment. As shown in Figure 9, the speech processing device 4 includes a content token generation unit 43a, a speaker vector generation unit 44, a prediction model 45, an acoustic token expansion unit 47, and a speech waveform generation unit 48.

[0070] The content token generation unit 43a generates the audio waveform y of the source speaker by processing in the same manner as the content token generation unit 33a. Src The content token c is obtained from this. The speaker vector generation unit 44 also processes the target speaker's voice waveform y in the same way as the speaker vector generation unit 34. Tgt Speaker vectors Tgt To obtain.

[0071] The content token encoder 451a, the merged acoustic token encoder 452, and the merged acoustic token generation decoder 453 of the prediction model 45 perform the same processing as the content token encoder 351a, the merged acoustic token encoder 352, and the merged acoustic token generation decoder 353, respectively.

[0072] The acoustic token expansion unit 47 includes an expansion unit 471. The expansion unit 471 expands the predicted merged acoustic token ^z into the predicted multi-stage acoustic token ^A by the same process as the expansion unit 271.

[0073] The audio waveform generation unit 48 includes a conversion unit 481. The conversion unit 481 processes the predicted multi-stage acoustic token ^A and the predicted audio waveform ^y in the same manner as the conversion unit 281. Tgt Convert to.

[0074] The processing flow of the audio processing device 4 according to the fourth embodiment will be explained using Figure 10. Figure 10 is a flowchart showing the processing flow of the audio processing device according to the fourth embodiment.

[0075] As shown in Figure 10, first, the audio processing device 4 extracts content tokens from the source speaker's audio waveform (step S401). The audio processing device 4 also generates a speaker vector from the target speaker's audio waveform (step S402).

[0076] Next, the speech processing device 4 predicts merged acoustic tokens using a model based on the content tokens and speaker vectors (step S403).

[0077] Here, the audio processing device 4 expands the predicted merged acoustic tokens into multi-stage acoustic tokens (step S404). Then, the audio processing device 4 generates an audio waveform from the multi-stage acoustic tokens (step S405). After that, the audio processing device 4 outputs the generated audio waveform (step S406).

[0078] The third and fourth embodiments are effective for voice conversion tasks. For example, according to the third embodiment, a voice conversion model is used in which NAC is the speech waveform generator, and a streaming voice conversion model can be constructed from non-parallel data.

[0079] RNNT requires that the content of the input and output data correspond left-to-right. Therefore, as in voice conversion using RNNT (see, for example, Non-Patent Document 4), parallel data in which the utterances of the source speaker and the target speaker match was necessary.

[0080] Parallel data is expensive to collect, making it difficult to train multi-speaker voice conversion models. In response, voice conversion methods based on non-parallel data, which do not require parallel data, have been proposed in the market.

[0081] In voice conversion based on non-parallel data, the model learns to reconstruct the speech by decomposing it into content (speech content) and speaker information during training. However, because the speech sequence lengths must match during error calculation for model training, models that involve speech rate conversion are difficult to train.

[0082] The third and fourth embodiments are characterized by acquiring content from audio, predicting acoustic tokens from the content, separating audio waveform generation from the acoustic tokens, and using RNNT for predicting acoustic tokens from content.

[0083] According to the third and fourth embodiments, the strengths of RNNT can be enjoyed in tasks such as voice quality conversion based on non-parallel data. The strengths of RNNT are that it operates over streaming and is robust in maintaining the left-to-right correspondence between input and output. These strengths enable robust sound quality conversion for streaming. Furthermore, since RNNT can handle different input and output sequence lengths, voice quality conversion with speech velocity is possible even from non-parallel data.

[0084] [Program] As one embodiment, the voice processing devices described above (voice processing device 1, voice processing device 2, voice processing device 3, voice processing device 4) can be implemented by installing a voice processing program that performs the above processing as packaged software or online software on a desired computer. For example, by having an information processing device execute the above voice processing program, the information processing device can be made to function as a voice processing device. The information processing device referred to here includes desktop or notebook personal computers. In addition, the information processing device also includes smartphones, tablet terminals, etc.

[0085] Figure 11 shows an example of a computer that executes an audio processing program. Computer 1000 has, for example, memory 1010 and a CPU (Central Processing Unit) 1020. Computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0086] Memory 1010 includes ROM (Read Only Memory) 1011 and RAM (Random Access Memory) 1012. ROM 1011 stores, for example, a boot program such as BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. For example, a removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0087] The hard disk drive 1090 stores, for example, the OS 1091, application programs 1092, program modules 1093, and program data 1094. That is, the programs that define each process of the audio processing device are implemented as program modules 1093 in which executable code for the computer is written. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, a program module 1093 for performing processes similar to the functional configuration of the audio processing device is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD.

[0088] Furthermore, the configuration data used in the processing of the above-described embodiment is stored as program data 1094 in, for example, memory 1010 or hard disk drive 1090. The CPU 1020 then reads the program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as needed and executes the processing of the above-described embodiment.

[0089] Furthermore, the program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090; for example, they may be stored in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via a network interface 1070.

[0090] Although embodiments applying the invention made by the present inventors have been described above, the present invention is not limited by the descriptions and drawings that constitute part of the disclosure of the present invention in this embodiment. That is, all other embodiments, examples, and operational techniques made by those skilled in the art based on this embodiment are included in the scope of the present invention.

[0091] 1, 2, 3, 4 Speech Processing Unit 11, 31 Multi-stage Acoustic Token Extraction Unit 12, 32 Acoustic Token Merging Unit 13 Text Analysis Unit 14, 24 Speaker Vector Generation Unit 15, 25, 35, 45 Prediction Model 16, 36 Learning Unit 23 Text Analysis Unit 27, 47 Acoustic Token Expansion Unit 28, 48 Speech Waveform Generation Unit 33a, 43a Content Token Generation Unit 34, 44 Speaker Vector Generation Unit 111, 311 Conversion Unit 121, 321 Merging Unit 151, 251 Text Token Encoder 152, 252, 352, 452 Merged Acoustic Token Encoder 153, 253, 353, 453 Merged Acoustic Token Generation Decoder 161, 361 Prediction Error Calculation Unit 162, 362 Optimization Unit 351a, 451a Encoder for content token 271, 471 Decompression unit 281, 481 Conversion unit

Claims

1. A speech processing device comprising: a merging unit that converts a first token, in which each frame of the speech waveform is represented by multiple values, into a second token, in which each frame of the speech waveform is represented by a single value; a prediction unit that predicts a third token, in which each frame of the speech waveform is represented by a single value, using a model based on the speech waveform; and an optimization unit that optimizes the model so that the difference between the second tokens becomes small.

2. The audio processing device according to claim 1, characterized in that the merging unit refers to a table that associates a combination of multiple values ​​representing each frame of the audio waveform with a single value representing each frame of the audio waveform, and converts the first token into the second token.

3. The speech processing apparatus according to claim 1 or 2, characterized in that the prediction unit predicts the third token using the model based on a vector representing speaker characteristics generated from the speech waveform and text tokens which are time-series data including elements of a text-based sentence.

4. The speech processing apparatus according to claim 1 or 2, wherein the prediction unit predicts the third token using the model based on a vector representing speaker characteristics generated from the speech waveform and a content token which is data containing elements of a sentence based on the speech waveform.

5. A sound processing device comprising: a prediction unit that predicts a fourth token, in which each frame of the sound waveform is represented by a single value, using a model based on the sound waveform; and an expansion unit that expands the fourth token predicted by the model into a fifth token, in which each frame of the sound waveform is represented by multiple values.

6. A speech processing method performed by a speech processing device, comprising: a merging unit that converts a first token, in which each frame of a speech waveform is represented by multiple values, into a second token, in which each frame of the speech waveform is represented by a single value; a prediction unit that predicts a third token, in which each frame of the speech waveform is represented by a single value, using a model based on the speech waveform; and an optimization unit that optimizes the model so that the difference between the second tokens becomes small.

7. A speech processing program characterized by causing a computer to perform the following steps: a merge step of converting a first token, in which each frame of the speech waveform is represented by multiple values, into a second token, in which each frame of the speech waveform is represented by a single value; a prediction step of predicting a third token, in which each frame of the speech waveform is represented by a single value, using a model based on the speech waveform; and an optimization step of optimizing the model so that the difference between the second tokens becomes small.

Images