Speech processing device, speech processing system, speech processing method, and program

Abstract

To improve the accuracy of audio processing. [Solution] The audio processing device includes a feature extraction unit that extracts acoustic feature vectors from audio data, a first encoding unit that converts the acoustic feature vectors into first feature vectors to be used in a first audio processing task, a second encoding unit that converts the acoustic feature vectors into second feature vectors to be used in a second audio processing task, and an output unit that outputs a processing result based on at least one of the first feature vector or the second feature vector.

Description

[Technical Field]

[0001] The present invention relates to a speech processing device, a speech processing system, a speech processing method, and a program. [Background technology]

[0002] Techniques are known for performing speech processing tasks based on features extracted from speech data. For example, Patent Document 1 discloses an information processing device that generates a common speech unit for normal speech and whispering, which is a latent representation that absorbs the difference between normal speech and whispering, after pre-training on normal speech and whispering. [Overview of the project] [Problems that the invention aims to solve]

[0003] However, conventional techniques have room for improvement in the accuracy of speech processing. For example, speech data contains various types of information, including linguistic information, paralinguistic information, and non-linguistic information, but conventional feature extraction methods do not adequately separate this information and are unable to extract features suitable for speech processing tasks.

[0004] One embodiment of the present invention aims to improve the accuracy of speech processing. [Means for solving the problem]

[0005] An audio processing device according to one embodiment of the present invention includes: a feature extraction unit that extracts acoustic feature vectors from audio data; a first encoding unit that converts the acoustic feature vectors into first feature vectors to be used in a first audio processing task; a second encoding unit that converts the acoustic feature vectors into second feature vectors to be used in a second audio processing task; and an output unit that outputs a processing result based on at least one of the first feature vector or the second feature vector. [Effects of the Invention]

[0006] According to one embodiment of the present invention, the accuracy of speech processing can be improved.

Brief Description of the Drawings

[0007] [Figure 1] It is a block diagram showing an example of the overall configuration of an audio processing system according to an embodiment of the present invention. [Figure 2] It is a block diagram showing an example of the hardware configuration of a computer according to an embodiment of the present invention. [Figure 3] It is a block diagram showing an example of a machine learning model according to an embodiment of the present invention. [Figure 4] It is a block diagram showing an example of the functional configuration of an audio processing system according to an embodiment of the present invention. [Figure 5] It is a flowchart showing an example of model generation processing according to an embodiment of the present invention. [Figure 6] It is a flowchart showing an example of task execution processing according to an embodiment of the present invention. [Figure 7] It is a diagram showing an example of an audio conversion task according to an embodiment of the present invention. [Figure 8] It is a diagram showing an example of prediction accuracy according to an embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0008] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, components having the same function are denoted by the same reference numerals, and redundant descriptions are omitted.

[0009] [Embodiment] An embodiment of the present invention is an information processing system that executes tasks related to audio processing. Hereinafter, the information processing system according to the present embodiment will be referred to as an "audio processing system".

[0010] In this embodiment, the voice processing system may execute a task related to any voice processing (hereinafter simply referred to as "voice processing task"). As an example, the voice processing task may include feature extraction, voice recognition, speaker recognition, emotion recognition, voice conversion, and the like. The voice conversion task may include, as an example, voice synthesis, voice quality conversion, and the like. The voice quality conversion task may include, as an example, speaker conversion, emotion conversion, and the like.

[0011] The voice data includes various types of information including language information, para-language information or non-language information, etc. The language information is information that can be described in a natural language. The language information is used, for example, in a voice recognition task or the like. The para-language information is information that cannot be described in a natural language and is information intentionally given by the speaker. The para-language information is used, for example, in an emotion recognition task or the like. The non-language information is information that cannot be described in a natural language and is information given regardless of the speaker's intention. The non-language information is used, for example, in speaker recognition or the like.

[0012] In recent years, various recognizers that enable high-precision recognition have been proposed based on machine learning technologies such as deep learning. These recognizers obtain a recognition result by inputting an acoustic feature vector extracted from voice data into a deep learning model. The acoustic feature vector is, for example, a feature quantity designed based on the human perception mechanism such as a logarithmic mel filter bank feature quantity or a mel frequency cepstrum coefficient (MFCC), or a feature quantity extracted by a machine learning model using self-supervised learning. The machine learning model using self-supervised learning includes, for example, HuBERT described in Reference 1.

[0013] [Reference 1] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, Abdelrahman Mohamed, "HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units," IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451-3460, 2021.

[0014] In particular, by extracting features using machine learning models employing self-supervised learning, it is possible to construct machine learning models specialized for various speech processing tasks. In self-supervised learning, first, a large amount of unlabeled speech data is used to mask a portion of the input acoustic feature vector, and the model learns to estimate the masked portion from information of other related parts. This allows for the construction of a feature extractor that extracts unique feature representations from speech data. Subsequently, a classifier is added after the feature extractor, and fine-tuning is performed with speech data that has been labeled according to the speech processing task. This allows for the construction of a recognizer tailored to the speech processing task.

[0015] However, conventional technologies still have room for improvement in the accuracy of speech processing tasks. For example, the acoustic feature vector used for a particular speech processing task may retain features unnecessary for that task, which can reduce accuracy. For instance, if paralinguistic or nonlinguistic information remains in the acoustic feature vector used for a speech recognition task, it can affect the accuracy of the speech processing result. The same applies to emotion recognition tasks or speaker recognition tasks; if the acoustic feature vector contains information unnecessary for the speech processing task, accuracy may not be sufficiently improved.

[0016] One embodiment of the present invention aims to improve the accuracy of speech processing. To this end, in this embodiment, an acoustic feature vector is extracted from speech data, the acoustic feature vector is converted into a first feature vector to be used in a first speech processing task and a second feature vector to be used in a second speech processing task, and a processing result based on at least one of the first feature vector or the second feature vector is output.

[0017] According to this embodiment, the accuracy of speech processing can be improved by converting acoustic feature vectors into feature vectors suitable for speech processing tasks. Furthermore, according to this embodiment, various speech processing tasks can be efficiently executed by simultaneously extracting feature vectors suitable for multiple different speech processing tasks.

[0018] <Overall configuration of the voice processing system> The overall configuration of the voice processing system according to one embodiment of the present invention will be described with reference to Figure 1. Figure 1 is a block diagram showing an example of the overall configuration of the voice processing system according to one embodiment of the present invention.

[0019] As shown in Figure 1, the voice processing system 1000 includes a learning device 10 and a voice processing device 20. The learning device 10 and the voice processing device 20 are connected to a communication network N. The communication network N is configured so that each connected device can communicate with one another.

[0020] Communication network N is constructed using wired communication networks such as the Internet, LAN (Local Area Network), or WAN (Wide Area Network). Communication network N may also include wireless communication networks such as Wi-Fi or short-range wireless communication, or mobile communication networks such as WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution), or 5G (5th Generation).

[0021] The learning device 10 is an example of an information processing device that generates a machine learning model for performing speech processing tasks. The learning device 10 may be a computer such as a personal computer, workstation, or server.

[0022] The learning device 10 pre-stores audio data to be learned. Based on the audio data to be learned, the learning device 10 generates a trained machine learning model (hereinafter also simply referred to as the "trained model").

[0023] The speech processing device 20 is an example of an information processing device that performs a speech processing task based on a trained model. The speech processing device 20 may be a computer such as a personal computer, workstation, or server.

[0024] The speech processing device 20 stores a trained model. The trained model may be a machine learning model generated by the learning device 10. The speech processing device 20 accepts speech data to be processed as input. The speech processing device 20 executes the speech processing task by inputting the input speech data into the trained model. The speech processing device 20 outputs the result of processing the speech data. The result of processing the speech data may include the result of executing the speech processing task.

[0025] Furthermore, the learning device 10 or the voice processing device 20 is not limited to a computer, as long as it is a device equipped with communication capabilities. The learning device 10 or the voice processing device 20 may be, for example, an image forming apparatus (printer, fax machine, MFP (Multifunction Peripheral / Product / Printer: digital multifunction device), scanner), a PJ (Projector), an IWB (Interactive White Board: a whiteboard with electronic blackboard functionality that enables mutual communication), an output device such as a digital signage, a HUD (Head Up Display) device, industrial machinery, imaging devices, sound collection devices, medical equipment, networked home appliances, automobiles (Connected Car), notebook PCs (Personal Computers), mobile phones, smartphones, tablet devices, game consoles, PDAs (Personal Digital Assistants), digital cameras, wearable PCs, or desktop PCs.

[0026] The configuration of the voice processing system 1000 shown in Figure 1 is an example, and other configurations are possible. For example, the voice processing system 1000 may include a plurality of learning devices 10 or a plurality of voice processing devices 20. For example, the learning device 10 or voice processing device 20 may be implemented by a single information processing device, or it may be a system implemented by a plurality of information processing devices. The voice processing system 1000 includes various devices that perform at least one of the input and output of electronic data, and these devices may utilize various services provided by the voice processing system 1000.

[0027] <Hardware configuration of the voice processing system> The hardware configuration of each device included in the voice processing system 1000 will be described with reference to Figure 2. The learning device 10 or voice processing device 20 included in the voice processing system 1000 may, for example, be implemented by a computer. Figure 2 is a block diagram showing an example of the hardware configuration of a computer according to one embodiment of the present invention.

[0028] As shown in Figure 2, the computer 500 is equipped with a CPU 501, ROM 502, RAM 503, HD 504, HDD (Hard Disk Drive) controller 505, display 506, external device connection I / F (Interface) 508, network I / F 509, bus line 510, keyboard 511, pointing device 512, DVD-RW (Digital Versatile Disk Rewritable) drive 514, and media I / F 516.

[0029] Of these components, the CPU 501 controls the operation of the entire computer. The ROM 502 stores programs used to drive the CPU 501, such as the IPL. The RAM 503 is used as the work area for the CPU 501. The HD 504 stores various data, such as programs. The HDD controller 505 controls the reading or writing of various data to the HD 504 according to the control of the CPU 501.

[0030] The display 506 displays various information such as cursors, menus, windows, characters, or images. The external device connection interface 508 is an interface for connecting various external devices. In this case, external devices include, for example, USB (Universal Serial Bus) memory and printers. The network interface 509 is an interface for data communication using the communication network N. The bus line 510 is an address bus, data bus, etc., for electrically connecting each component such as the CPU 501 shown in Figure 2.

[0031] The keyboard 511 is a type of input means equipped with multiple keys for inputting characters, numbers, and various instructions. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting processing targets, and moving the cursor. The DVD-RW drive 514 controls the reading or writing of various data to the DVD-RW 513, which is an example of a removable recording medium. Note that it is not limited to DVD-RW, but may also be DVD-R, etc. The media interface 516 controls the reading or writing (storage) of data to the recording medium 515, such as flash memory.

[0032] <Machine learning model configuration> The configuration of the machine learning model according to this embodiment will be described with reference to Figure 3. Figure 3 is a block diagram showing an example of a machine learning model according to one embodiment of the present invention.

[0033] Machine learning model M is an example of a machine learning model for performing speech processing tasks. Machine learning model M may include, for example, a neural network. The neural network may be, for example, a deep neural network based on deep learning, a recurrent neural network, a sequence transformation model, a neural network with an attention mechanism, or an autoregressive model (e.g., Transformer).

[0034] As shown in Figure 3, the machine learning model M is input with audio data. The audio data is electronic data based on human speech. The audio data may be a time-domain audio signal recorded from human speech. The audio data may also be data obtained by converting a time-domain audio signal to the frequency domain. For example, the audio data may be a sequence obtained by converting each frame of the audio signal into a logarithmic mel spectrogram.

[0035] The machine learning model M includes a feature extractor E0, a speech recognition encoder E1, a speaker recognition encoder E2, an emotion recognition encoder E3, a speech recognition decoder D1, a speaker recognition decoder D2, an emotion recognition decoder D3, a reconstruction decoder D4, and a speech converter D5.

[0036] Note that the feature extractor E0 is an example of a feature extraction unit. The speech recognition encoder E1, speaker recognition encoder E2, and emotion recognition encoder E3 are examples of a first encoding unit or a second encoding unit. The speech recognition decoder D1 is an example of a speech recognition unit. The speaker recognition decoder D2 is an example of a speaker recognition unit. The emotion recognition decoder D3 is an example of an emotion recognition unit. The reconstruction decoder D4 is an example of a reconstruction unit. The speech converter D5 is an example of a speech conversion unit.

[0037] The feature extractor E0 extracts acoustic feature vectors from the audio data input to the machine learning model M. The feature extractor E0 may, for example, be a machine learning model using self-supervised learning. The acoustic feature vectors extracted by the feature extractor E0 are input to the speech recognition encoder E1, the speaker recognition encoder E2, and the emotion recognition encoder E3.

[0038] The speech recognition encoder E1 converts the acoustic feature vector output from the feature extractor E0 into a speech recognition feature vector. The speech recognition feature vector is the acoustic feature vector used for the speech recognition task. In this embodiment, the speech recognition encoder E1 is trained to convert the acoustic feature vector into a feature vector suitable for the speech recognition task. The speech recognition feature vector output by the speech recognition encoder E1 is input to the speech recognition decoder D1 and the reconstructed decoder D4.

[0039] The speaker recognition encoder E2 converts the acoustic feature vector output from the feature extractor E0 into a speaker recognition feature vector. The speaker recognition feature vector is the acoustic feature vector used for the speaker recognition task. In this embodiment, the speaker recognition encoder E2 is trained to convert the acoustic feature vector into a feature vector suitable for the speaker recognition task. The speaker recognition feature vector output by the speaker recognition encoder E2 is input to the speaker recognition decoder D2 and the reconstruction decoder D4.

[0040] The emotion recognition encoder E3 converts the acoustic feature vector output from the feature extractor E0 into an emotion recognition feature vector. The emotion recognition feature vector is the acoustic feature vector used for the emotion recognition task. In this embodiment, the emotion recognition encoder E3 is trained to convert the acoustic feature vector into a feature vector suitable for the emotion recognition task. The emotion recognition feature vector output by the emotion recognition encoder E3 is input to the emotion recognition decoder D3 and the reconstruction decoder D4.

[0041] The speech recognition decoder D1 generates text data based on the speech recognition feature vector output from the speech recognition encoder E1. The text data includes a transcript of the speech contained in the audio data.

[0042] The speaker recognition decoder D2 generates speaker labels based on the speaker recognition feature vector output from the speaker recognition encoder E2. The speaker labels contain identification information that identifies the speaker of the speech contained in the audio data.

[0043] The emotion recognition decoder D3 generates emotion labels based on the emotion recognition feature vector output from the emotion recognition encoder E3. The emotion labels contain identification information that identifies the emotion of the speech contained in the audio data.

[0044] The reconstructed decoder D4 reconstructs the acoustic feature vector based on the speech recognition feature vector output from the speech recognition encoder E1, the speaker recognition feature vector output from the speaker recognition encoder E2, and the emotion recognition feature vector output from the emotion recognition encoder E3. The reconstructed acoustic feature vector reconstructed by the reconstructed decoder D4 is input to the speech converter D5.

[0045] The speech converter D5 converts the reconstructed acoustic feature vector output from the reconstructed decoder D4 into speech data. Based on the acoustic feature vector, the speech converter D5 may generate synthesized speech data or voice-transformed speech data. Synthesized speech data is speech data in which the utterance content has been transformed while maintaining the speaker and emotion of the speech data input to the machine learning model M. Voice-transformed speech data is speech data in which the speaker or emotion has been transformed while maintaining the utterance content of the speech data input to the machine learning model M.

[0046] The machine learning model M outputs the results of a speech processing task. The results of the speech processing task may include at least one of the following: the results of a feature extraction task, the results of a speech recognition task, the results of a speaker recognition task, the results of an emotion recognition task, or the results of a speech conversion task.

[0047] The result of the feature extraction task may include at least one of the following: a speech recognition feature vector output by the speech recognition encoder E1, a speaker recognition feature vector output by the speaker recognition encoder E2, an emotion recognition feature vector output by the emotion recognition encoder E3, or an acoustic feature vector output by the reconstruction decoder D4.

[0048] The results of the speech recognition task may include text data output by the speech recognition decoder D1. The results of the speaker recognition task may include speaker labels output by the speaker recognition decoder D2. The results of the emotion recognition task may include emotion labels output by the emotion recognition decoder D3. The results of the speech conversion task may include speech data output by the speech converter D5.

[0049] <Functional Configuration of the Voice Processing System> The functional configuration of the voice processing system 1000 will be explained with reference to Figure 4. Figure 4 is a block diagram showing an example of the functional configuration of a voice processing system according to one embodiment of the present invention.

[0050] Learning device As shown in Figure 4, the learning device 10 comprises a data storage unit 101, a feature extraction unit 110, a model generation unit 120, and a model output unit 130.

[0051] The data storage unit 101 is implemented, for example, using the HD504 shown in Figure 2. Reading or writing data stored in the HD504 is performed, for example, via the HDD controller 505.

[0052] The feature extraction unit 110, the model generation unit 120, and the model output unit 130 are realized, for example, by a process in which a program deployed from the HD 504 shown in Figure 2 onto the RAM 503 is executed by the CPU 501.

[0053] The data storage unit 101 stores training data. Training data is electronic data used to train a machine learning model. The training data includes speech data, text data, speaker labels, and sentiment labels.

[0054] Text data is the ground truth data for the speech recognition task and is electronic data containing the text transcribed from the audio data. Speaker labels are the ground truth data for the speaker recognition task and are electronic data containing identification information to identify the speaker in the audio data. Emotion labels are the ground truth data for the emotion recognition task and are electronic data containing identification information to identify the emotion in the audio data.

[0055] The feature extraction unit 110 extracts acoustic feature vectors from the audio data. The feature extraction unit 110 may also extract acoustic feature vectors from the audio data contained in the training data read from the data storage unit 101. The feature extraction unit 110 may also extract acoustic feature vectors from the audio data using a pre-trained feature extractor E0. The feature extractor E0 may be pre-trained using self-supervised learning. The feature extractor E0 may be trained using audio data stored in the data storage unit 101.

[0056] The model generation unit 120 generates a learned model. The model generation unit 120 may generate a learned model based on the learning data read from the data storage unit 101.

[0057] The model generation unit 120 may generate an audio recognition encoder E1. The model generation unit 120 may generate an audio recognition encoder E1 by learning the relationship between the acoustic feature vectors extracted from the audio data included in the learning data and the text data included in the learning data.

[0058] The model generation unit 120 may generate an audio recognition encoder E1 by supervised learning with the text data as the correct data. For example, the model generation unit 120 may update the parameters of the audio recognition encoder E1 so as to minimize the loss L Text for the audio recognition task. The loss L Text may be defined by Equation (1).

[0059]

Equation

[0060] However, L T is the CTC (Connectionist Temporal Classification) loss between the output of the audio recognition encoder E and the text data. L T-S and L T-E are the cross-entropy errors between the output of the audio recognition encoder E and the random data. The random data may be, for example, a uniform distribution. λ is a scaling factor. That is, the loss L Text is the weighted sum of the CTC loss L T with the correct data for the audio recognition task and the cross-entropy errors L T-S and L T-E with the random data. Note that L [[ID=e39]] T-S and L T-E may be the cross-entropy errors with the data based on the speaker label or the emotion label.

[0061] The model generation unit 120 may generate a speaker recognition encoder E2. The model generation unit 120 may generate the speaker recognition encoder E2 by learning the relationship between acoustic feature vectors extracted from speech data included in the training data and speaker labels included in the training data.

[0062] The model generation unit 120 may generate a speaker recognition encoder E2 by supervised learning using speaker labels as ground truth data. For example, the model generation unit 120 generates a loss L for the speaker recognition task. Speaker The parameters of the speaker recognition encoder E2 may be updated to minimize the loss L. Speaker may be defined by equation (2).

[0063]

number

[0064] However, L S L is the cross-entropy error between the output of the speaker recognition encoder E2 and the speaker label. S-E This is the cross-entropy error between the output of the speaker recognition encoder E2 and the random data. That is, the loss L Speaker This is the cross-entropy error L with the ground truth data for the speaker recognition task. S And the cross-entropy error L with the random data. S-E This is the weighted loss obtained by adding the two factors. S-E This can also be expressed as the cross-entropy error with the emotion-labeled data.

[0065] The model generation unit 120 may generate an emotion recognition encoder E3. The model generation unit 120 may generate the emotion recognition encoder E3 by learning the relationship between acoustic feature vectors extracted from the audio data included in the training data and the emotion labels included in the training data.

[0066] The model generation unit 120 may generate an emotion recognition encoder E3 by supervised learning using emotion labels as ground truth data. For example, the model generation unit 120 generates an emotion recognition encoder E3 by supervised learning using emotion labels as ground truth data. Emotion The parameters of the emotion recognition encoder E3 may be updated to minimize the loss L. Emotion It may also be defined by equation (3).

[0067]

number

[0068] However, L E L is the cross-entropy error between the output of the emotion recognition encoder E3 and the emotion label. E-S This is the cross-entropy error between the output of the emotion recognition encoder E3 and the random data. That is, the loss L Emotion This is the cross-entropy error L with the ground truth data for the emotion recognition task. E And the cross-entropy error L with the random data. E-S This is the weighted loss obtained by adding the two factors. E-S This may be defined as the cross-entropy error with the data based on speaker labels.

[0069] The model generation unit 120 may generate a speech recognition decoder D1. The model generation unit 120 may generate the speech recognition decoder D1 by learning the relationship between the speech recognition feature vector output from the speech recognition encoder E1 and the text data included in the training data.

[0070] The model generation unit 120 may generate a speech recognition decoder D1 by supervised learning using text data as ground truth data. The model generation unit 120 may update the parameters of the speech recognition decoder D1 so as to minimize the error between the output of the speech recognition decoder D1 when a speech recognition feature vector is input and the text data.

[0071] The model generation unit 120 may generate a speaker recognition decoder D2. The model generation unit 120 may generate the speaker recognition decoder D2 by learning the relationship between the speaker recognition feature vector output from the speaker recognition encoder E2 and the speaker labels included in the training data.

[0072] The model generation unit 120 may generate a speaker recognition decoder D2 by supervised learning using speaker labels as ground truth data. The model generation unit 120 may update the parameters of the speaker recognition decoder D2 so as to minimize the error between the output of the speaker recognition decoder D2 and the speaker label when a speaker recognition feature vector is input.

[0073] The model generation unit 120 may generate an emotion recognition decoder D3. The model generation unit 120 may generate the emotion recognition decoder D3 by learning the relationship between the emotion recognition feature vector output from the emotion recognition encoder E3 and the emotion labels included in the training data.

[0074] The model generation unit 120 may generate an emotion recognition decoder D3 by supervised learning using emotion labels as ground truth data. The model generation unit 120 may update the parameters of the emotion recognition decoder D3 so as to minimize the error between the output of the emotion recognition decoder D3 and the emotion label when an emotion recognition feature vector is input.

[0075] The model generation unit 120 may generate a reconstructed decoder D4. The model generation unit 120 may combine speech recognition feature vectors, speaker recognition feature vectors, and emotion recognition feature vectors. The model generation unit 120 may generate the reconstructed decoder D4 by learning the relationship between the combined feature vectors and acoustic feature vectors extracted from speech data included in the training data.

[0076] The model generation unit 120 calculates the loss L between the output of the reconstructed decoder D4 when a combined feature vector is input and the acoustic feature vector extracted from the audio data included in the training data. ReconstThe parameters of the reconstructed decoder D4 may be updated to minimize the loss L. Reconst For example, this could also be called Reconstruction Loss.

[0077] The model generation unit 120 may simultaneously train the speech recognition encoder E1, the speaker recognition encoder E2, the emotion recognition encoder E3, and the reconstruction decoder D4. The model generation unit 120 has a loss L Text , loss L Speaker , loss L Emotion and L Reconst The parameters of the speech recognition encoder E1, speaker recognition encoder E2, emotion recognition encoder E3, and reconstruction decoder D4 may be updated to minimize the sum L. The sum L may be defined by equation (4).

[0078]

number

[0079] The model output unit 130 outputs the trained model. The model output unit 130 may output the trained model generated by the model generation unit 120. The model output unit 130 may also transmit the trained model to the speech processing unit 20.

[0080] The trained model may include a speech recognition encoder E1, a speaker recognition encoder E2, an emotion recognition encoder E3, a speech recognition decoder D1, a speaker recognition decoder D2, an emotion recognition decoder D3, and a reconstruction decoder D4 generated by the model generation unit 120. Furthermore, the trained machine learning model may also include a pre-trained feature extractor E0 and a speech converter D5.

[0081] ≪Sound Processing Device≫ As shown in Figure 4, the voice processing device 20 includes a model storage unit 201, a voice input unit 210, a task execution unit 220, and a result output unit 230.

[0082] The model storage unit 201 is implemented, for example, using the HD504 shown in Figure 2. Reading or writing data stored in the HD504 is performed, for example, via the HDD controller 505.

[0083] The audio input unit 210, the task execution unit 220, and the result output unit 230 are realized, for example, by a process in which a program loaded from the HD 504 shown in Figure 2 onto the RAM 503 is executed by the CPU 501.

[0084] The model storage unit 201 stores pre-trained models. The pre-trained models stored in the model storage unit 201 may be generated by the learning device 10.

[0085] The voice input unit 210 accepts input of voice data to be processed. The voice input unit 210 may accept input of voice data via the microphone provided by the voice processing device 20. The voice input unit 210 may receive voice data from a terminal device equipped with a microphone via the communication network N. The voice input unit 210 may read voice data stored in a storage device such as the HD504.

[0086] The audio input unit 210 may accept audio signals in the time domain. The audio input unit 210 may accept audio signals in the frequency domain. If the audio input unit 210 accepts audio signals in the time domain, it may convert the audio signals to the frequency domain.

[0087] The task execution unit 220 executes a speech processing task. The task execution unit 220 may execute the speech processing task based on a trained model read from the model storage unit 201. The task execution unit 220 may execute the speech processing task based on speech data input to the speech input unit 210. The task execution unit 220 may execute the speech processing task by inputting the speech data into the trained model.

[0088] The result output unit 230 outputs the processing results of the audio data. The processing results of the audio data include the execution results of the audio processing tasks output by the trained model. For example, the processing results of the audio data may include at least one of the following: the processing results of a feature extraction task, the processing results of a speech recognition task, the processing results of a speaker recognition task, the processing results of an emotion recognition task, or the processing results of a speech conversion task.

[0089] The result output unit 230 may output the audio data processing results from the display 506 or speaker provided by the audio processing device 20. The result output unit 230 may transmit the audio data processing results to a terminal device equipped with a display or speaker via the communication network N. The result output unit 230 may transmit the audio data processing results to another information processing device or information processing system via the communication network N. The other information processing device or information processing system may use the audio data processing results for other audio processing.

[0090] <Processing procedure for audio processing methods> The speech processing method performed by the speech processing system 1000 will be described with reference to Figures 5 and 6. The speech processing method may include model generation processing (see Figure 5) and task execution processing (see Figure 6).

[0091] ≪Model Generation Process≫ The model generation process performed by the learning device 10 will be explained with reference to Figure 5. Figure 5 is a flowchart showing an example of the model generation process according to one embodiment of the present invention.

[0092] In step S101, the feature extraction unit 110 of the learning device 10 reads the training data from the data storage unit 101. The feature extraction unit 110 may also read one or more training data entries that have not been trained from the training data stored in the data storage unit 101.

[0093] In step S102, the feature extraction unit 110 of the learning device 10 extracts acoustic feature vectors from the audio data included in the training data read in step S101. Specifically, the feature extraction unit 110 inputs the audio data into the trained feature extractor E0. The feature extractor E0 extracts acoustic feature vectors from the input audio data and outputs the acoustic feature vectors. The feature extraction unit 110 acquires the acoustic feature vectors output from the feature extractor E0. The feature extraction unit 110 sends the acoustic feature vectors to the model generation unit 120. The model generation unit 120 receives the acoustic feature vectors from the feature extraction unit 110.

[0094] In step S103, the model generation unit 120 of the learning device 10 acquires text data from the training data read in step S101. Based on the acoustic feature vector received in step S102 and the text data acquired from the training data, the model generation unit 120 generates a speech recognition encoder E1.

[0095] In step S104, the model generation unit 120 of the learning device 10 obtains speaker labels from the training data read in step S101. Based on the acoustic feature vector received in step S102 and the speaker labels obtained from the training data, the model generation unit 120 generates a speaker recognition encoder E2.

[0096] In step S105, the model generation unit 120 of the learning device 10 obtains emotion labels from the training data read in step S101. Based on the acoustic feature vector received in step S102 and the emotion labels obtained from the training data, the model generation unit 120 generates an emotion recognition encoder E3.

[0097] In step S106, the model generation unit 120 of the learning device 10 inputs the acoustic feature vector received in step S102 into the speech recognition encoder E1 generated in step S103. The speech recognition encoder E1 converts the input acoustic feature vector into a speech recognition feature vector and outputs the speech recognition feature vector. The model generation unit 120 generates a speech recognition decoder D1 based on the speech recognition feature vector output from the speech recognition encoder E1 and the text data acquired in step S103.

[0098] In step S107, the model generation unit 120 of the learning device 10 inputs the acoustic feature vector received in step S102 into the speaker recognition encoder E2 generated in step S104. The speaker recognition encoder E2 converts the input acoustic feature vector into a speaker recognition feature vector and outputs the speaker recognition feature vector. The model generation unit 120 generates a speaker recognition decoder D2 based on the speaker recognition feature vector output from the speaker recognition encoder E2 and the speaker label acquired in step S104.

[0099] In step S108, the model generation unit 120 of the learning device 10 inputs the acoustic feature vector received in step S102 to the emotion recognition encoder E3 generated in step S105. The emotion recognition encoder E3 converts the input acoustic feature vector into an emotion recognition feature vector and outputs the emotion recognition feature vector. The model generation unit 120 generates an emotion recognition decoder D3 based on the emotion recognition feature vector output from the emotion recognition encoder E3 and the emotion label acquired in step S105.

[0100] In step S109, the model generation unit 120 of the learning device 10 combines the speech recognition feature vector acquired in step S106, the speaker recognition feature vector acquired in step S107, and the emotion recognition feature vector acquired in step S108. Based on the combined feature vector and the acoustic feature vector received in step S102, the model generation unit 120 generates a reconstructed decoder D4.

[0101] The learning device 10 may repeatedly execute the processes from step S101 to step S109. The learning device 10 may also execute the processes from step S103 to step S109 simultaneously in parallel. For example, the learning device 10 may repeatedly update the parameters of each encoder and decoder included in the machine learning model until it satisfies the termination condition for terminating the learning of the machine learning model. The termination condition may be, for example, that the number of parameter updates exceeds a predetermined threshold. Alternatively, the termination condition may be, for example, that the amount of parameter updates converges.

[0102] In step S110, the model output unit 130 of the learning device 10 generates a trained model and outputs the trained model. The model output unit 130 may also transmit the trained model to the speech processing device 20. The speech processing device 20 may receive the trained model from the learning device 10 and store it in the model storage unit 201.

[0103] The model output unit 130 may store the trained model in a storage device such as the HD504 of the learning device 10. The trained model stored in the storage device of the voice processing device 20 may be read out by the voice processing device 20. The learning device 10 may also transmit the trained model stored in its storage device to the voice processing device 20 in response to a request from the voice processing device 20.

[0104] ≪Task Execution Process≫ The task execution process performed by the voice processing device 20 will be explained with reference to Figure 6. Figure 6 is a flowchart showing an example of the task execution process according to one embodiment of the present invention.

[0105] In step S201, the audio input unit 210 of the audio processing device 20 receives input of audio data to be processed. The audio input unit 210 sends the audio data to the task execution unit 220.

[0106] In step S202, the task execution unit 220 of the voice processing device 20 receives voice data from the voice input unit 210. The task execution unit 220 reads the trained model from the model storage unit 201.

[0107] The task execution unit 220 inputs the audio data into the trained model. The audio data input into the trained model is then input into the feature extractor E0. The feature extractor E0 extracts acoustic feature vectors from the input audio data. The feature extractor E0 sends the extracted acoustic feature vectors to the speech recognition encoder E1, the speaker recognition encoder E2, and the emotion recognition encoder E3.

[0108] In step S203, the speech recognition encoder E1 receives an acoustic feature vector from the feature extractor E0. The speech recognition encoder E1 converts the acoustic feature vector into a speech recognition feature vector. The speech recognition encoder E1 sends the speech recognition feature vector to the speech recognition decoder D1 and the reconstructed decoder D4.

[0109] In step S204, the speaker recognition encoder E2 receives an acoustic feature vector from the feature extractor E0. The speaker recognition encoder E2 converts the acoustic feature vector into a speaker recognition feature vector. The speaker recognition encoder E2 sends the speaker recognition feature vector to the speaker recognition decoder D2 and the reconstruction decoder D4.

[0110] In step S205, the emotion recognition encoder E3 receives an acoustic feature vector from the feature extractor E0. The emotion recognition encoder E3 converts the acoustic feature vector into an emotion recognition feature vector. The emotion recognition encoder E3 sends the emotion recognition feature vector to the emotion recognition decoder D3 and the reconstruction decoder D4.

[0111] In step S206, the speech recognition decoder D1 receives a speech recognition feature vector from the speech recognition encoder E1. Based on the speech recognition feature vector, the speech recognition decoder D1 performs a speech recognition task. Specifically, the speech recognition decoder D1 converts the speech recognition feature vector into text data that represents the content of the speech utterance.

[0112] In step S207, the speaker recognition decoder D2 receives a speaker recognition feature vector from the speaker recognition encoder E2. Based on the speaker recognition feature vector, the speaker recognition decoder D2 performs a speaker recognition task. Specifically, the speaker recognition decoder D2 converts the speaker recognition feature vector into speaker labels that indicate the speakers of the audio data.

[0113] In step S208, the emotion recognition decoder D3 receives an emotion recognition feature vector from the emotion recognition encoder E3. Based on the emotion recognition feature vector, the emotion recognition decoder D3 performs an emotion recognition task. Specifically, the emotion recognition decoder D3 converts the emotion recognition feature vector into emotion labels that indicate the emotions of the audio data.

[0114] In step S209, the reconstructed decoder D4 receives a speech recognition feature vector from the speech recognition encoder E1. The reconstructed decoder D4 also receives a speaker recognition feature vector from the speaker recognition encoder E2. Furthermore, the reconstructed decoder D4 receives an emotion recognition feature vector from the emotion recognition encoder E3.

[0115] The reconstructed decoder D4 combines the speech recognition feature vector, speaker recognition feature vector, and emotion recognition feature vector. Based on the combined feature vector, the reconstructed decoder D4 reconstructs the acoustic feature vector. The reconstructed decoder D4 sends the reconstructed acoustic feature vector to the speech converter D5.

[0116] In step S210, the speech converter D5 receives the reconstructed acoustic feature vector from the reconstructed decoder D4. Based on the reconstructed acoustic feature vector, the speech converter D5 performs a speech conversion task. Specifically, the speech converter D5 converts the reconstructed acoustic feature vector into speech data.

[0117] Furthermore, the speech processing device 20 only needs to execute the steps corresponding to the speech processing task it is to perform. Specifically, the speech processing device 20 does not need to execute one or more steps from steps S206 to S208 and step S210. For example, if only a speech recognition task is to be performed, the speech processing device 20 only needs to execute steps S203 and S206. Also, for example, if only a speech synthesis task is to be performed, the speech processing device 20 only needs to execute steps S203 to S205 and steps S209 to S210.

[0118] In step S211, the trained model outputs the execution result of the speech processing task. The trained model only needs to output the execution result of the speech processing task it has performed. The task execution unit 220 of the speech processing device 20 obtains the execution result of the speech processing task output from the trained model. The task execution unit 220 sends the execution result of the speech processing task to the result output unit 230.

[0119] The result output unit 230 receives the execution result of the voice processing task from the task execution unit 220. Based on the execution result of the voice processing task, the result output unit 230 generates the processing result of voice data and outputs the processing result. The result output unit 230 may output the processing result of voice data from the voice processing device 20. The result output unit 230 may transmit the processing result of voice data to a terminal device, another information processing device, or another information processing system via the communication network N.

[0120] <Application Examples> The machine learning model according to this embodiment can be configured to perform at least one of a speech synthesis task or a voice quality conversion task. The speech synthesis task is an example of a speech processing task that converts text data representing spoken content into speech data. The voice quality conversion task is an example of a speech processing task that converts the voice quality of speech data. Voice quality conversion may include speaker conversion or emotion conversion.

[0121] Figure 7 shows an example of a speech conversion task according to one embodiment of the present invention. Figure 7(A) shows a configuration for performing a speech synthesis task, which is an example of a speech conversion task. Figure 7(B) shows a configuration for performing a voice quality conversion task, which is another example of a speech conversion task.

[0122] As shown in Figure 7(A), when performing a speech synthesis task, the speech recognition feature vector input to the reconstruction decoder D4 is replaced with a speech recognition feature vector extracted from other speech data. In Figure 7(A), the reconstruction decoder D4 is input with a speaker recognition feature vector and an emotion recognition feature vector based on speech data A, and a speech recognition feature vector based on speech data B. In this case, the speech data output from the speech converter D5 is speech data that has been converted from the utterance content of speech data B while maintaining the speaker and emotion of speech data A.

[0123] As shown in Figure 7(B), when performing a voice conversion task, the speaker feature vector input to the reconstruction decoder D4 is replaced with a speaker recognition feature vector extracted from other speech data. In Figure 7(B), the reconstruction decoder D4 is input with a speech recognition feature vector and emotion recognition feature vector based on speech data A, and a speaker recognition feature vector based on speech data B. In this case, the speech data output from the speech converter D5 is speech data converted to the speaker of speech data B while maintaining the utterance content and emotion of speech data A.

[0124] Figure 7(B) shows an example of a voice conversion task that transforms the speaker, but a voice conversion task that transforms emotions can be configured similarly. In this case, the reconstruction decoder D4 only needs to be input with speech recognition feature vectors and speaker recognition feature vectors based on speech data A, and emotion recognition feature vectors based on speech data B. The speech data output from the speech converter D5 will be speech data that has been transformed from speech data A to speech data B's emotion while maintaining the utterance content and speaker of speech data A.

[0125] <Prediction accuracy> The prediction accuracy of the machine learning model according to this embodiment will be explained with reference to Figure 8. Figure 8 is a diagram showing an example of the prediction accuracy according to one embodiment of the present invention.

[0126] In Figure 8, "Comparative Example" shows the result of performing a speech processing task using acoustic feature vectors based on self-supervised learning. "Example 1" shows the result of performing a speech processing task using feature vectors extracted by the machine learning model M shown in Figure 3. "Example 2" shows the result of performing a speech processing task using feature vectors extracted by a machine learning model M shown in Figure 3, but with the reconstruction decoder D4 omitted.

[0127] Figure 8(A) shows the prediction accuracy for the speech recognition task. The evaluation metric for prediction accuracy was WER (Word Error Rate). As shown in Figure 8(A), Examples 1 and 2 show a significant improvement in WER compared to the comparative example. Furthermore, comparing Example 1 and Example 2, Example 1 shows a lower WER. This indicates that by training the speech recognition encoder E1 simultaneously with the reconstructed decoder D4, it is possible to extract feature vectors more accurately that are suitable for the speech recognition task.

[0128] Figure 8(B) shows the prediction accuracy for the speaker recognition task. The evaluation metric for prediction accuracy was EER (Equal Error Rate). As shown in Figure 8(B), Examples 1 and 2 show improved EER compared to the comparative example. Furthermore, comparing Example 1 and Example 2, Example 1 shows a lower EER. This indicates that by training the speaker recognition encoder E2 simultaneously with the reconstruction decoder D4, it is possible to extract feature vectors more accurately that are suitable for the speaker recognition task.

[0129] As shown in Figure 8, the machine learning model according to this embodiment shows improved prediction accuracy in all tasks, including speech recognition, speaker recognition, and emotion recognition, compared to conventional methods. Therefore, this embodiment demonstrates that the accuracy of speech processing can be improved.

[0130] <Effects of the Embodiment> The audio processing device 20 according to this embodiment extracts acoustic feature vectors from audio data, converts the acoustic feature vectors into a first feature vector to be used in a first audio processing task and a second feature vector to be used in a second audio processing task, and outputs a processing result based on at least one of the first feature vector or the second feature vector.

[0131] In one respect, this embodiment improves the accuracy of speech processing by converting acoustic feature vectors into feature vectors suitable for speech processing tasks. In another respect, this embodiment allows for the efficient execution of various speech processing tasks by simultaneously extracting feature vectors suitable for multiple different speech processing tasks.

[0132] The machine learning model according to this embodiment may, for example, be applied to a voice dialogue system using a virtual human or the like. This enables the generation of high-quality responses in the voice dialogue system's response generation. For example, by utilizing the processing results of an emotion recognition task, the voice dialogue system can generate responses that take the user's emotions into consideration. Also, for example, by utilizing the processing results of a speaker recognition task, the voice dialogue system can generate responses in a multi-person conversation while clearly identifying the person to whom it is responding.

[0133] Conventional voice dialogue systems often only use the processing results of the speech recognition task. However, by using the machine learning model according to this embodiment, the processing results of the emotion recognition task and the speaker recognition task can be obtained simultaneously, thereby improving the response quality of the voice dialogue system. Furthermore, conventional voice dialogue systems required the construction of separate models for each speech processing task. In contrast, the machine learning model according to this embodiment can execute multiple speech processing tasks with a single model, reducing construction costs.

[0134] The machine learning model according to this embodiment enables highly customizable speech synthesis or voice conversion in response generation for a speech dialogue system. Specifically, by replacing only the speaker recognition feature vector input to the reconstruction decoder D4 with a speaker recognition feature vector based on other speakers, the speaker can be accurately converted while maintaining the utterance content and emotion. Furthermore, by replacing only the speech recognition feature vector input to the reconstruction decoder D4 with a speech recognition feature vector based on other utterances, the utterance content can be accurately converted while maintaining the speaker and emotion.

[0135] Furthermore, the machine learning model according to this embodiment can be applied to the following systems. For example, in a speech recognition service, information other than the speech recognition result (e.g., emotion labels or speaker labels) can be simultaneously acquired and used to enhance functions such as conversation analysis or meeting minute generation. This makes it possible to visualize the changes in emotions during a meeting, which can be used as material for analyzing communication. In addition, since the content of each speaker's utterance can be recognized from the utterance and speaker label, it can be applied to meeting minute generation services.

[0136] [supplement] Each of the functions of the embodiments described above can be realized by one or more processing circuits. Hereinafter, "processing circuit" as used herein includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, as well as devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0137] The apparatus described in the examples represents only one of several computing environments for carrying out the embodiments disclosed herein. In one embodiment, the learning device 10 or the speech processing device 20 includes multiple computing devices, such as a server cluster. The multiple computing devices are configured to communicate with each other via any type of communication link, including a network or shared memory, and to perform the processing disclosed herein.

[0138] Examples of the present invention are as follows:

[0139] (Note 1) A feature extraction unit that extracts acoustic feature vectors from audio data, A first encoding unit that converts the aforementioned acoustic feature vector into a first feature vector to be used in a first speech processing task, A second encoding unit that converts the aforementioned acoustic feature vector into a second feature vector to be used in a second speech processing task, An output unit that outputs a processing result based on at least one of the first feature vector or the second feature vector, A voice processing device equipped with the following features.

[0140] (Note 2) The first speech processing task includes a speech recognition task, a speaker recognition task, or an emotion recognition task, The second speech processing task includes a speech recognition task, a speaker recognition task, or an emotion recognition task, which is different from the first speech processing task. The audio processing device described in Appendix 1.

[0141] (Note 3) The system further comprises a third encoding unit that converts the aforementioned acoustic feature vector into a third feature vector used for a third speech processing task, The output unit outputs the processing result based on at least one of the first feature vector, the second feature vector, or the third feature vector. The audio processing device described in Appendix 1 or 2.

[0142] (Note 4) The first speech processing task described above is a speech recognition task, The second speech processing task described above is a speaker recognition task, The third speech processing task is an emotion recognition task. The audio processing device described in Appendix 3.

[0143] (Note 5) The system further includes a speech recognition unit that converts the first feature vector into text data representing the spoken content of the speech data. The audio processing device described in Appendix 4.

[0144] (Note 6) The system further includes a speaker recognition unit that converts the second feature vector into a speaker label indicating the speaker of the audio data. The audio processing device described in Appendix 4.

[0145] (Note 7) The system further includes an emotion recognition unit that converts the third feature vector into an emotion label indicating the emotion of the audio data. The audio processing device described in Appendix 4.

[0146] (Note 8) The first encoding unit is trained to minimize the loss based on the ground truth data and the loss based on random data for the first speech processing task. The second coding unit is trained to minimize the loss based on the ground truth data and the loss based on random data for the second speech processing task. The audio processing device described in Appendix 1 or 2.

[0147] (Note 9) The system further comprises a reconstruction unit that reconstructs the acoustic feature vector based on the first feature vector and the second feature vector. The audio processing device described in Appendix 1 or 2.

[0148] (Note 10) The reconstruction unit reconstructs the acoustic feature vector based on the first feature vector extracted from the first audio data and the second feature vector extracted from the second audio data. The audio processing device described in Appendix 9.

[0149] (Note 11) The system further includes an audio conversion unit that converts the acoustic feature vector reconstructed by the reconstruction unit into third audio data. The audio processing device described in Appendix 10.

[0150] (Note 12) A speech processing system including a learning device and a speech processing device, The aforementioned audio processing device is A feature extraction unit that extracts a first acoustic feature vector from the first audio data, A first encoding unit that converts the first acoustic feature vector into a first feature vector to be used for a first speech processing task, A second encoding unit that converts the first acoustic feature vector into a second feature vector used for a second speech processing task, An output unit that outputs a processing result based on at least one of the first feature vector or the second feature vector, Equipped with, The learning device is The first coding unit is trained to minimize the loss based on the ground truth data and the loss based on random data for the first speech processing task. The system includes a model generation unit that learns the second coding unit to minimize the loss based on the ground truth data and the loss based on random data for the second speech processing task. A voice processing system equipped with the following features.

[0151] (Note 13) Computers Procedure for extracting acoustic feature vectors from audio data, A procedure for converting the aforementioned acoustic feature vector into a first feature vector to be used in a first speech processing task, A procedure for converting the aforementioned acoustic feature vector into a second feature vector to be used in a second speech processing task, A procedure for outputting a processing result based on at least one of the first feature vector or the second feature vector, A method for performing audio processing.

[0152] (Note 14) On the computer, Procedure for extracting acoustic feature vectors from audio data, A procedure for converting the aforementioned acoustic feature vector into a first feature vector to be used in a first speech processing task, A procedure for converting the aforementioned acoustic feature vector into a second feature vector to be used in a second speech processing task, A procedure for outputting a processing result based on at least one of the first feature vector or the second feature vector, A program to execute.

[0153] Although embodiments of the present invention have been described in detail above, the present invention is not limited to these embodiments, and various modifications or changes are possible within the scope of the gist of the present invention as described in the claims. [Explanation of symbols]

[0154] 10: Learning device 20: Voice Processing Device 101: Data storage unit 110: Feature extraction unit 120: Model generation unit 130: Model Output Section 201: Model Memory Unit 210: Voice input section 220: Task Execution Unit 230: Result output section 1000: Voice processing system [Prior art documents] [Patent Documents]

[0155] [Patent Document 1] Japanese Patent Publication No. 2024-85304

Claims

1. A feature extraction unit that extracts acoustic feature vectors from audio data, A first encoding unit that converts the aforementioned acoustic feature vector into a first feature vector to be used in a first speech processing task, A second encoding unit that converts the aforementioned acoustic feature vector into a second feature vector to be used in a second speech processing task, An output unit that outputs a processing result based on at least one of the first feature vector or the second feature vector, A voice processing device equipped with the following features.

2. The first speech processing task includes a speech recognition task, a speaker recognition task, or an emotion recognition task, The second speech processing task includes a speech processing task that is different from the first speech processing task, among the speech recognition task, the speaker recognition task, or the emotion recognition task. The voice processing device according to claim 1.

3. The system further comprises a third encoding unit that converts the aforementioned acoustic feature vector into a third feature vector used for a third speech processing task, The output unit outputs the processing result based on at least one of the first feature vector, the second feature vector, or the third feature vector. The voice processing device according to claim 1 or 2.

4. The first speech processing task is a speech recognition task, The second speech processing task described above is a speaker recognition task, The third speech processing task is an emotion recognition task. The audio processing device according to claim 3.

5. The system further includes a speech recognition unit that converts the first feature vector into text data representing the spoken content of the speech data. The audio processing device according to claim 4.

6. The system further includes a speaker recognition unit that converts the second feature vector into a speaker label indicating the speaker of the audio data. The audio processing device according to claim 4.

7. The system further includes an emotion recognition unit that converts the third feature vector into an emotion label indicating the emotion of the audio data. The audio processing device according to claim 4.

8. The first encoding unit is trained to minimize the loss based on the ground truth data and the loss based on random data for the first speech processing task. The second encoding unit is trained to minimize the loss based on the ground truth data and the loss based on random data for the second speech processing task. The voice processing device according to claim 1 or 2.

9. The system further comprises a reconstruction unit that reconstructs the acoustic feature vector based on the first feature vector and the second feature vector. The voice processing device according to claim 1 or 2.

10. The reconstruction unit reconstructs the acoustic feature vector based on the first feature vector extracted from the first audio data and the second feature vector extracted from the second audio data. The voice processing device according to claim 9.

11. The system further includes an audio conversion unit that converts the acoustic feature vector reconstructed by the reconstruction unit into third audio data. The audio processing device according to claim 10.

12. A speech processing system including a learning device and a speech processing device, The aforementioned audio processing device is A feature extraction unit that extracts a first acoustic feature vector from the first audio data, A first encoding unit that converts the first acoustic feature vector into a first feature vector to be used in a first speech processing task, A second encoding unit that converts the first acoustic feature vector into a second feature vector used for a second speech processing task, An output unit that outputs a processing result based on at least one of the first feature vector or the second feature vector, Equipped with, The learning device is The first encoding unit is trained to minimize the loss based on the ground truth data and the loss based on random data for the first speech processing task. The system includes a model generation unit that learns the second coding unit to minimize the loss based on the ground truth data and the loss based on random data for the second speech processing task. Voice processing system.

13. Computers Procedure for extracting acoustic feature vectors from audio data, A procedure for converting the aforementioned acoustic feature vector into a first feature vector to be used in a first speech processing task, A procedure for converting the aforementioned acoustic feature vector into a second feature vector to be used in a second speech processing task, A procedure for outputting a processing result based on at least one of the first feature vector or the second feature vector, A method for performing audio processing.

14. On the computer, Procedure for extracting acoustic feature vectors from audio data, A procedure for converting the aforementioned acoustic feature vector into a first feature vector to be used in a first speech processing task, A procedure for converting the aforementioned acoustic feature vector into a second feature vector to be used in a second speech processing task, A procedure for outputting a processing result based on at least one of the first feature vector or the second feature vector, A program to execute.

Images