Audio processing device, audio processing system, audio processing method and program

Abstract

Efficiently train models to perform tasks related to speech processing. [Solution] The voice processing device has a task execution unit that executes tasks related to voice processing based on a trained model, and the trained model is trained using voice data and labels obtained by converting feature vectors extracted from the voice data in accordance with predetermined rules.

Description

[Technical Field]

[0001] The present invention relates to an audio processing device, an audio processing system, an audio processing method, and a program. [Background technology]

[0002] Techniques for executing tasks related to speech processing based on machine learning techniques are known. For example, Patent Document 1 discloses an information processing device that acquires speech data, extracts speech features from the speech data, acquires speech expressions from the speech features, and inputs the speech expressions to a speech recognition unit to acquire text data. Summary of the Invention [Problem to be solved by the invention]

[0003] However, conventional techniques have room for more efficient execution of tasks related to speech processing. For example, conventional speech recognition techniques often train speech recognizers through supervised learning using speech data and transcripts, but preparing a large amount of transcripts of speech data is costly.

[0004] An embodiment of the present invention aims to efficiently perform tasks related to audio processing. [Means for solving the problem]

[0005] One embodiment of the present invention relates to a speech processing device, and the speech processing device includes a task execution unit that executes tasks related to speech processing based on a trained model. The trained model is trained using speech data and labels obtained by converting feature vectors extracted from the speech data in accordance with predetermined rules. [Effects of the Invention]

[0006] According to one embodiment of the present invention, tasks related to audio processing can be performed efficiently. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a block diagram showing an example of the overall configuration of a voice processing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing an example of a hardware configuration of a computer according to an embodiment of the present invention. [Figure 3] 1 is a block diagram showing an example of a functional configuration of a speech processing system according to an embodiment of the present invention; [Figure 4] FIG. 10 is a diagram illustrating an example of a label conversion process according to an embodiment of the present invention. [Figure 5] 10 is a flowchart illustrating an example of a model learning process according to an embodiment of the present invention. [Figure 6] 10 is a flowchart illustrating an example of a label conversion process according to an embodiment of the present invention. [Figure 7] 10 is a flowchart illustrating an example of a task execution process according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0008] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, components having the same functions are designated by the same reference numerals, and redundant description will be omitted.

[0009] [Embodiment] One embodiment of the present invention is an information processing system that executes tasks related to speech processing. Hereinafter, the information processing system according to this embodiment will be referred to as a "speech processing system." In this embodiment, the speech processing system may execute any task related to speech processing (hereinafter simply referred to as a "speech processing task"). The speech processing task may include, for example, speech recognition, speech synthesis, speech enhancement, speaker recognition, speaker authentication, emotion recognition, voice activity detection, etc.

[0010] Speech recognition, an example of a speech processing task, is a technology for converting speech data containing human words, voices, or conversations into text data. Speech recognition technology is increasingly being used in business settings, for example, to display subtitles during meetings, create minutes, or create reports, etc. Speech recognition technology makes it easier to convert speech into text and input data into a system than when a person converts speech into text using a keyboard. For this reason, speech recognition technology is expected to be an effective input method that will lead to improved business efficiency.

[0011] Conventionally, speech recognizer training has typically been performed using supervised learning, which involves using speech with transcription, a pair of speech data and its corresponding transcription text, to learn the correspondence between the two. In supervised learning, training a speech recognizer with high recognition accuracy requires the preparation of a large amount of speech with transcription, and the high cost of obtaining training data is an issue.

[0012] Reference 1 discloses a method for training a speech recognizer using pseudo-supervised learning, also known as semi-supervised learning. In the method described in Reference 1, a speech recognizer is trained using a small amount of speech with transcription, and the trained speech recognizer is used to generate text from a large amount of speech without transcription. A pair of input speech and output text with a high degree of confidence in the inference process is adopted as a new speech with transcription, and the trained speech recognizer is updated using semi-supervised learning. However, in the process of generating transcription text for semi-supervised learning, there is no guarantee that the text inferred by the trained speech recognizer and its confidence level will be reliable enough as training data. Performing semi-supervised learning using incorrect transcription text may actually hinder the speech recognizer's learning.

[0013] [Reference 1] Patent No. 7052866

[0014] Reference 2 proposes another method using a large amount of untranscribed speech. Reference 2 applies a masked language model, which is used in machine learning processes such as large-scale language models, to speech representation learning. In the method described in Reference 2, a large amount of untranscribed speech is converted into acoustic features in advance, and then an N-class classification machine learning model is generated using self-supervised learning. N-class classification involves masking a certain percentage of frames, and then predicting which of the representative values ​​of a predetermined N number of acoustic features the masked frame most closely resembles by referring to the context of the preceding and following unmasked frames. Next, transfer learning to a speech recognition task is performed using a small amount of transcribed speech, with the parameters of the pre-trained machine learning model as initial values. Reference 2 shows that performance can be improved without the transcription costs compared to speech recognizers using only a small amount of transcribed speech.

[0015] [Reference 2] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, Abdelrahman Mohamed, "HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units," IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3451-3460, 2021.

[0016] As in Reference 2, in order to perform pre-training by classification of masked frames, a finite number of correct labels is required. Meanwhile, unlike masked language models that take discrete vectors such as text information as input, speech recognizers take continuous vectors as input signals. Therefore, in Reference 2, in order to perform pre-training by classification of masked frames for input speech, acoustic features are quantized into N classes in advance, and a finite number of correct labels for self-supervised learning (hereinafter referred to as "self-supervised labels") are created.

[0017] Reference 2 uses the k-means algorithm to classify the acoustic features of all frames of untranscribed speech into N classes and then uses the resulting class numbers as self-supervised labels for each frame. Reference 2 also proposes a method for generating classification labels using product quantization. However, these quantization methods rely on statistical distributions based on the population of untranscribed speech, and the centers of gravity of each class are limited to the subspace formed by the population. Therefore, when new untranscribed speech with different statistical distributions is added to the pre-training process, there is no guarantee that the appropriate classes will exist within those subspaces. In such cases, continuing to train the pre-training model may lead to imbalance in the classification model, potentially resulting in insufficient performance improvement when applying transfer training to a speech recognition model.

[0018] An embodiment of the present invention aims to efficiently execute a speech processing task based on a trained model that has been trained using speech data and labels obtained by converting feature vectors extracted from the speech data in accordance with predetermined rules.

[0019] In this embodiment, the self-supervised learning method eliminates statistical inference elements from the self-supervised label generation process during the pre-training model learning process, eliminating the dependency of the self-supervised labels on the data distribution. Furthermore, the self-supervised labels are derived based solely on the linguistic information (phonemes) of the speech data, enabling self-supervised learning with high affinity to speech processing tasks.

[0020] According to this embodiment, self-supervised labels are derived using deterministic operations that follow predetermined rules, without using the statistical distribution of the dataset, which allows for efficient execution of speech processing tasks. Furthermore, according to this embodiment, a pre-trained model that can be additionally trained using a small amount of transcribed text is generated, allowing for efficient execution of various speech processing tasks.

[0021] <Overall configuration of the voice processing system> The overall configuration of the voice processing system will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the overall configuration of a voice processing system according to an embodiment of the present invention.

[0022] 1, the speech processing system 1000 includes a model learning device 10 and a speech processing device 20. The model learning device 10 and the speech processing device 20 are connected to a communication network N. The communication network N is configured so that the connected devices can communicate with each other.

[0023] The communication network N is constructed by a network using wired communication such as the Internet, a local area network (LAN), or a wide area network (WAN). The communication network N may include not only wired communication but also wireless communication such as a wireless LAN or short-range wireless communication, or a network using mobile communication such as WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution), or 5G (5th Generation).

[0024] The model learning device 10 is an example of an information processing device that generates a machine learning model for executing a speech processing task. The model learning device 10 may be, for example, a computer such as a personal computer, a workstation, or a server.

[0025] The machine learning model may be, for example, a neural network, which may be, for example, a deep neural network based on deep learning, a recurrent neural network, an attention mechanism model, an autoregressive model (e.g., a Transformer), or the like.

[0026] The model learning device 10 stores speech data to be learned in advance. The speech data to be learned includes untranscribed speech and speech with transcription. The speech with transcription requires less data than untranscribed speech. The model learning device 10 generates a pre-trained model based on the untranscribed speech. The model learning device 10 additionally trains the pre-trained model based on the transcription-included speech to generate a trained machine learning model (hereinafter simply referred to as a "trained model").

[0027] The speech processing device 20 is an example of an information processing device that executes a speech processing task based on a trained model. The speech processing device 20 may be, for example, a computer such as a personal computer, a workstation, or a server.

[0028] The voice processing device 20 stores a trained model. The trained model may be generated by the model training device 10. The voice processing device 20 receives input of voice data to be processed. The voice processing device 20 executes a voice processing task by inputting the input voice data into the trained model. The voice processing device 20 outputs the execution result of the voice processing task.

[0029] Note that model learning device 10 or speech processing device 20 is not limited to a computer as long as it is a device with a communication function. Model learning device 10 or speech processing device 20 may be, for example, an image forming device (printer, fax machine, MFP (Multifunction Peripheral / Product / Printer: digital multifunction device), scanner device), PJ (Projector: projector), IWB (Interactive White Board: white board with electronic blackboard function that allows mutual communication), output device such as digital signage, HUD (Head Up Display) device, industrial machinery, imaging device, sound collection device, medical equipment, network home appliance, automobile (Connected Car), notebook PC (Personal Computer), mobile phone, smartphone, tablet terminal, game console, PDA (Personal Digital Assistant), digital camera, wearable PC, desktop PC, etc.

[0030] 1 is an example, and other configurations may be used. For example, the model learning device 10 or the speech processing device 20 may be realized by one information processing device, or may be a system realized by multiple information processing devices. The speech processing system 1000 includes various devices that input and / or output electronic data, and these devices may use various services provided by the speech processing system 1000.

[0031] <Hardware configuration of the voice processing system> The hardware configuration of each device included in the speech processing system 1000 will be described with reference to Fig. 2. The model learning device 10 or the speech processing device 20 included in the speech processing system 1000 may be realized by a computer, for example. Fig. 2 is a block diagram showing an example of the hardware configuration of a computer according to an embodiment of the present invention.

[0032] As shown in FIG. 2, the computer 500 includes a CPU 501, a ROM 502, a RAM 503, a HD 504, a HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 516.

[0033] Of these, the CPU 501 controls the operation of the entire computer. The ROM 502 stores programs used to drive the CPU 501, such as the IPL. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading and writing of various data from and to the HD 504 under the control of the CPU 501.

[0034] The display 506 displays various types of information such as a cursor, menus, windows, characters, or images. The external device connection I / F 508 is an interface for connecting various types of external devices. In this case, the external devices are, for example, USB (Universal Serial Bus) memories or printers. The network I / F 509 is an interface for data communication using the communication network N1. The bus line 510 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 501 shown in FIG. 2.

[0035] The keyboard 511 is a type of input means having multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 514 controls reading and writing of various data from a DVD-RW 513, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R, etc. The media I / F 516 controls reading and writing (storing) of data from a recording medium 515, such as a flash memory.

[0036] <Functional configuration of the voice processing system> The functional configuration of the speech processing system 1000 will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the functional configuration of the speech processing system according to one embodiment of the present invention.

[0037] <Model learning device> As shown in FIG. 3, the model learning device 10 includes an unlabeled data storage unit 101, a labeled data storage unit 102, a feature extraction unit 110, a label conversion unit 120, a model generation unit 130, and an additional learning unit 140.

[0038] The unlabeled data storage unit 101 and the labeled data storage unit 102 are realized, for example, using the HD 504 shown in Fig. 2. Reading or writing of data stored in the HD 504 is performed via an HDD controller 505, for example.

[0039] The feature extraction unit 110, the label conversion unit 120, the model generation unit 130, and the additional learning unit 140 are realized, for example, by processing that is executed by the CPU 501 in accordance with a program loaded from the HD 504 onto the RAM 503 shown in FIG.

[0040] The unlabeled data storage unit 101 stores unlabeled data in advance. The unlabeled data is data to which no correct answer label has been assigned. In this embodiment, the unlabeled data may be untranscribed speech data (i.e., untranscribed speech). The unlabeled data storage unit 101 stores a sufficient amount of unlabeled data in advance.

[0041] The labeled data storage unit 102 stores labeled data in advance. The labeled data is data to which a correct answer label has been assigned. In this embodiment, the labeled data may be a pair of audio data and text data transcribed from the audio data (i.e., audio with transcription). The labeled data storage unit 102 only needs to store a very small amount of labeled data.

[0042] The audio data is electronic data based on human speech. The audio data may be a time-domain audio signal containing recorded human speech. The audio data may also be data obtained by converting a time-domain audio signal into a frequency domain. In this embodiment, the audio data is a sequence obtained by converting each frame of the audio signal into a logarithmic mel spectrogram. The number of dimensions of the logarithmic mel spectrogram may be any number, but in this embodiment, the number is set to 80, for example.

[0043] The text data included in the labeled data may be any text data that indicates the content of the utterance included in the audio data. The text data included in the labeled data does not have to be human-transcribed text data. The text data included in the labeled data may be, for example, the results of speech recognition of the audio data.

[0044] The feature extraction unit 110 extracts feature vectors from the unlabeled data. The feature extraction unit 110 may generate a feature vector sequence by extracting a feature vector for each frame of the unlabeled data. In this embodiment, the feature vectors are Mel-Frequency Cepstrum Coefficients (MFCCs). For example, the feature extraction unit 110 may convert an 80-dimensional logarithmic Mel spectrogram into 80-dimensional Mel-frequency cepstrum coefficients by applying a discrete cosine transform.

[0045] The label conversion unit 120 converts the feature vectors into self-supervised labels. The label conversion unit 120 may convert the feature vectors extracted by the feature extraction unit 110 into self-supervised labels. The label conversion unit 120 may convert the feature vectors extracted from unlabeled data into self-supervised labels. The label conversion unit 120 may generate a sequence of self-supervised labels by converting each feature vector corresponding to each frame of the unlabeled data into a self-supervised label.

[0046] The label conversion unit 120 converts the feature vector into a self-supervised label according to a predetermined conversion rule. The conversion rule is a rule that deterministically derives a unique self-supervised label from the feature vector itself, without relying on the statistical distribution of unlabeled data.

[0047] The label conversion unit 120 may quantize the feature vector to a predetermined number of integers. The label conversion unit 120 may quantize part of the feature vector to integers. The label conversion unit 120 may acquire an element of the feature vector that indicates linguistic information as part of the feature vector. The label conversion unit 120 may acquire an element of a predetermined d dimension from the feature vector as part of the feature vector. The label conversion unit 120 may acquire an element of d dimensions or less from the feature vector as part of the feature vector. Here, d is an integer less than the number of dimensions of the feature vector. In this embodiment, d is an integer greater than or equal to 4 and less than 80. d may be set to any integer, and may be, for example, 10.

[0048] Mel-frequency cepstrum coefficients are generated by performing a discrete cosine transform on a logarithmic Mel spectrogram. Therefore, the linguistic information (phonemes) of a speech signal is stored in low-dimensional elements, while paralinguistic and nonlinguistic information is stored in high-dimensional elements. Nonlinguistic information includes, for example, tone of voice, prosody, and noise. Therefore, by extracting low-dimensional elements from a feature vector and discarding high-dimensional elements, the linguistic information of a speech signal can be preserved while being quantized to a small number of integers.

[0049] The label conversion unit 120 may convert each dimension of the feature vector into a single-digit β-ary number, and then convert the β-ary integer obtained by concatenating the digits into a decimal number, thereby quantizing the vector into a decimal integer, where β is an integer equal to or greater than 2.

[0050] Specifically, the label conversion unit 120 first normalizes the feature vector to a mean of 0 and a variance of 1. Next, the label conversion unit 120 converts each dimension of the normalized feature vector into a single-digit β-adic number. The label conversion unit 120 may convert each dimension of the feature vector into a β-adic number by comparing it with β-1 thresholds.

[0051] For example, the label conversion unit 120 converts the n-th element x of the feature vector by using the formula (1). n where n is an integer between 1 and d, and λ1, λ2, . . . , λ β-1 is a predetermined threshold value.

[0052]

number

[0053] As an example, when converting to binary numbers (that is, β=2), the label conversion unit 120 may set the threshold value λ1=0 and calculate equation (2).

[0054]

number

[0055] As another example, when converting to a ternary number (that is, β=3), the label conversion unit 120 may calculate equation (3) by setting the thresholds λ1=−0.5 and λ2=0.5.

[0056]

number

[0057] The label conversion unit 120 concatenates the β-ary numbers corresponding to each dimension of the feature vector according to the number of dimensions. As a result, an integer expressed as a d-digit β-ary number is generated. The label conversion unit 120 converts the d-digit β-ary number into a decimal number. The label conversion unit 120 obtains the integer expressed as a decimal number as a self-supervised label.

[0058] The feature vector (Mel-frequency cepstrum coefficients), which was a continuous vector, is d For example, if elements of a feature vector with 10 or fewer dimensions are converted to binary numbers, β = 2 and d = 10, so β d =2 10 = 1024 classes. For example, if elements of a feature vector with 6 or less dimensions are converted to ternary numbers, β = 3 and d = 6, so β d =3 6 =Can be quantized into 729 classes.

[0059] The label conversion process executed by the label conversion unit 120 will be described with reference to Fig. 4. Fig. 4 is a diagram for explaining an example of the label conversion process according to one embodiment of the present invention.

[0060] x1 to x shown in Figure 4 N is a feature vector corresponding to frames 1 to N of the speech signal. n (where n is an integer between 1 and N) is an 80-dimensional real vector R 80 The label conversion unit 120 converts the feature vector x n A vector ^x containing elements of the d dimensions from the lowest dimension n Note that ^xn is a d-dimensional real vector R d is.

[0061] The label conversion unit 120 converts the vector ^x n Each normalized vector ~x n The label conversion unit 120 generates the vector ~x n In this way, the label conversion unit 120 converts each of the d dimensions of the feature vectors x1 to x2 corresponding to frames 1 to N into a β-ary number, and then converts the d-digit β-ary number into a decimal number. N Each is self-supervised label c1~c N In addition, the self-supervised label c n is 0 or more β d It will be an integer less than

[0062] The model generation unit 130 generates a pre-trained model based on the unlabeled data and the self-supervised labels. The model generation unit 130 may input the unlabeled data to the pre-trained model during training and update the parameters of the pre-trained model based on the error between the output of the pre-trained model and the self-supervised labels. As an example, the model generation unit 130 may update the weights of the intermediate layer of the neural network included in the pre-trained model based on the backpropagation algorithm.

[0063] The additional training unit 140 generates a trained model by additionally training the pre-trained model based on the labeled data. The additional training unit 140 may additionally train the pre-trained model by fine tuning. The additional training unit 140 may additionally train the pre-trained model by transfer learning.

[0064] The additional training unit 140 may additionally train the pre-trained model so that it can perform various voice processing tasks, such as speech recognition, speech synthesis, speech enhancement, speaker recognition, speaker authentication, emotion recognition, and voice activity detection.

[0065] <Sound processing device> As shown in FIG. 3, the voice processing device 20 includes a model storage unit 201, a voice input unit 210, a task execution unit 220, and a result output unit 230.

[0066] The model storage unit 201 is realized, for example, by using the HD 504 shown in Fig. 2. Reading or writing of data stored in the HD 504 is performed via an HDD controller 505, for example.

[0067] The voice input unit 210, task execution unit 220, and result output unit 230 are realized by, for example, processing that is executed by the CPU 501 in accordance with a program loaded onto the RAM 503 from the HD 504 shown in FIG.

[0068] Trained models are stored in the model storage unit 201. The trained models stored in the model storage unit 201 may be generated by the model learning device 10. The trained models stored in the model storage unit 201 may be pre-trained models trained using unlabeled data, which have been additionally trained using labeled data.

[0069] The audio input unit 210 accepts input of audio data to be processed. The audio input unit 210 may accept input of audio data via a microphone connected to an external device connection I / F provided in the audio processing device 20. The audio input unit 210 may receive audio data via a communication network N from a terminal device equipped with a microphone.

[0070] The audio input unit 210 may receive an input of an audio signal. The audio input unit 210 may receive an input of a logarithmic mel spectrogram obtained by converting the audio signal. When receiving an input of an audio signal, the audio input unit 210 may convert each frame of the audio signal into a logarithmic mel spectrogram. The number of dimensions of the logarithmic mel spectrogram may be arbitrary, but in this embodiment, as an example, the number of dimensions is set to 80.

[0071] The task execution unit 220 executes a voice processing task. The task execution unit 220 may execute the voice processing task based on a trained model read out from the model storage unit 201. The task execution unit 220 may execute the voice processing task based on voice data input to the voice input unit 210. The task execution unit 220 may execute the voice processing task by inputting the voice data to the trained model.

[0072] The task executing unit 220 may execute various voice processing tasks. The task executing unit 220 may execute a task of voice recognition of voice data input to the voice input unit 210. The tasks executed by the task executing unit 220 are not limited to voice recognition, and may include, for example, voice synthesis, voice enhancement, speaker recognition, speaker authentication, emotion recognition, voice activity detection, etc. Note that when the task executing unit 220 executes a task that does not require input voice (for example, voice synthesis, etc.), the voice processing device 20 does not need to include the voice input unit 210.

[0073] The result output unit 230 outputs the execution result of the voice processing task. For example, if the voice processing task is voice recognition, the execution result of the voice processing task includes the recognition result of the voice data. The recognition result of the voice data may include text data indicating the content of the utterance included in the voice data.

[0074] For example, if the speech processing task is speech synthesis, the execution result of the speech processing task includes a speech signal synthesized from text data. For example, if the speech processing task is speech enhancement, the execution result of the speech processing task includes a speech signal with enhanced speech. For example, if the speech processing task is speaker recognition, the execution result of the speech processing task includes identification information for identifying the speaker. For example, if the speech processing task is speaker authentication, the execution result of the speech processing task includes a speaker authentication result. For example, if the speech processing task is emotion recognition, the execution result of the speech processing task includes an emotion label. For example, if the speech processing task is voice activity detection, the execution result of the speech processing task includes information indicating the voice activity in the speech data.

[0075] The result output unit 230 may display the execution result of the voice processing task on a display 506 provided in the voice processing device 20. The result output unit 230 may transmit the execution result of the voice processing task to a terminal device equipped with a display via the communication network N.

[0076] <Processing procedure for audio processing method> The speech processing method executed by the speech processing system 1000 will be described with reference to Figures 5 to 7. The speech processing method may include a model learning process (see Figure 5) and a task execution process (see Figure 7).

[0077] <Model learning process> The model training process is a process for generating a trained model for executing a voice processing task. Fig. 5 is a flowchart showing an example of the training process according to an embodiment of the present invention.

[0078] In step S101, the feature extraction unit 110 of the model learning device 10 reads out unlabeled data from the unlabeled data storage unit 101. The feature extraction unit 110 may read out one or more pieces of unlabeled data that have not yet been learned from the unlabeled data stored in the unlabeled data storage unit 101.

[0079] In step S102, the feature extraction unit 110 of the model learning device 10 extracts a feature vector from the unlabeled data read in step S101. Specifically, the feature extraction unit 110 applies a discrete cosine transform to the unlabeled data, which is a logarithmic mel spectrogram, to convert it into mel-frequency cepstrum coefficients. The feature extraction unit 110 sends the extracted feature vector to the label conversion unit 120.

[0080] In step S103, the label conversion unit 120 of the model learning device 10 receives the feature vector from the feature extraction unit 110. The label conversion unit 120 converts the feature vector into a self-supervised label in accordance with a predetermined conversion rule. The label conversion unit 120 sends the self-supervised label to the model generation unit 130.

[0081] In step S104, the model generation unit 130 of the model learning device 10 reads the unlabeled data read by the feature extraction unit 110 in step S101 from the unlabeled data storage unit 101. The model generation unit 130 receives the self-supervised labels from the label conversion unit 120. The model generation unit 130 generates a pre-training model based on the unlabeled data and the self-supervised labels. The model generation unit 130 sends the pre-training model to the additional training unit 140.

[0082] Specifically, the model generation unit 130 inputs unlabeled data into a pre-trained model under training. The pre-trained model executes a speech processing task on the input unlabeled data and outputs the execution result of the speech processing task. The model generation unit 130 calculates the error between the output of the pre-trained model and the self-supervised label. The model generation unit 130 updates the parameters of the pre-trained model based on the error between the output of the pre-trained model and the self-supervised label.

[0083] The model learning device 10 may repeatedly execute the processes from step S101 to step S104. For example, the model learning device 10 may repeatedly update the parameters of the pre-learning model until a termination condition for terminating the pre-learning is met. The termination condition may be, for example, that the number of parameter updates is equal to or greater than a predetermined threshold. Alternatively, the termination condition may be, for example, that the amount of parameter update has converged.

[0084] In step S105, the additional learning unit 140 of the model learning device 10 reads labeled data from the labeled data storage unit 102. The additional learning unit 140 may read one or more pieces of labeled data that have not yet been learned from the labeled data stored in the labeled data storage unit 102.

[0085] In step S106, the additional learning unit 140 of the model learning device 10 receives the pre-trained model from the model generation unit 130. The additional learning unit 140 additionally learns the pre-trained model based on the labeled data read in step S105. As a result, a trained model is generated.

[0086] The model learning device 10 may repeatedly execute the processes from step S105 to step S106. For example, the model learning device 10 may repeatedly update the parameters of the pre-learning model until a termination condition for terminating the additional learning is satisfied. The termination condition for terminating the additional learning may be the same as or different from the termination condition for terminating the pre-learning.

[0087] In step S107, the additional learning unit 140 of the model learning device 10 outputs the trained model. The additional learning unit 140 may transmit the trained model to the speech processing device 20. The speech processing device 20 may receive the trained model from the model learning device 10 and store it in the model storage unit 201.

[0088] The additional learning unit 140 may store the trained model in a storage device such as the HD 504 of the model learning device 10. The trained model stored in the storage device of the speech processing device 20 may be read by the speech processing device 20. In addition, the model learning device 10 may transmit the trained model stored in the storage device to the speech processing device 20 in response to a request from the speech processing device 20.

[0089] <Label conversion process> The label conversion process (step S103 in Fig. 5) will be described in more detail with reference to Fig. 6. Fig. 6 is a flowchart showing an example of the label conversion process according to one embodiment of the present invention.

[0090] In step S131, the label conversion unit 120 acquires elements of d dimensions or less from the feature vector. Specifically, the label conversion unit 120 acquires elements of the first dimension to the dth dimension from the 80-dimensional Mel-frequency cepstrum coefficients. In this embodiment, d=10.

[0091] In step S132, the label conversion unit 120 normalizes the d-dimensional feature vector acquired in step S131 to a mean of 0 and a variance of 1. Specifically, for each dimension of the d-dimensional feature vector, the label conversion unit 120 subtracts the minimum value and divides the result by the difference between the maximum and minimum values.

[0092] In step S133, the label conversion unit 120 converts each dimension of the d-dimensional feature vector normalized in step S132 into a single-digit β-ary number. For example, when converting each dimension of the feature vector into a binary number, the label conversion unit 120 sets λ1=0 and calculates equation (2).

[0093] In step S134, the label conversion unit 120 concatenates the d β-adic numbers converted in step S132 according to the number of dimensions. As a result, an integer expressed in a d-digit β-adic number is generated. The label conversion unit 120 converts the d-digit β-adic number into a decimal number. The label conversion unit 120 obtains the integer expressed in a decimal number as a self-supervised label.

[0094] We will now provide specific examples of the relationship between feature vectors and self-supervised labels. In the first example, we set d=10 and β=2, convert a 10-dimensional feature vector x into a 10-digit binary number ^x, and obtain a self-supervised label C, which is a decimal integer. Note that the threshold is λ1=0. In this case, x, ^x, and C are as follows:

[0095] ·x=[0.8,-0.4,0.3,0.8,0.2,-0.5,-0.1,0.6,-0.3,0.1] ^x=[1,0,1,1,1,0,0,1,0,1] C={1011100101}2=741

[0096] In the second example, d = 6 and β = 3 are used, and a six-dimensional feature vector x is converted into a six-digit ternary number ^x to obtain a self-supervised label C, which is a decimal integer. The thresholds are λ1 = -0.5 and λ2 = 0.5. In this case, x, ^x, and C are as follows:

[0097] ·x=[0.8,-0.4,0.3,0.8,0.2,-0.5] ^x=[2,1,1,2,1,1] C={211211}3=616

[0098] In the third example, we set d=5 and β=4, convert a five-dimensional feature vector x into a five-digit quaternary number ^x, and obtain a self-supervised label C, which is a decimal integer. The thresholds are λ1=-0.5, λ2=0, and λ3=0.5. In this case, x, ^x, and C are as follows:

[0099] x=[0.8,-0.4,0.3,0.8,0.2] ^x=[3,1,2,3,2] C={31232}4=878

[0100] <<Task execution process>> The task execution process is a process of executing a voice processing task based on a trained model. Fig. 7 is a flowchart showing an example of the task execution process according to one embodiment of the present invention.

[0101] In step S201, the audio input unit 210 of the audio processing device 20 accepts input of audio data to be processed. If the audio data is a time-domain audio signal, the audio input unit 210 converts the audio signal into a logarithmic mel spectrogram. The audio input unit 210 sends the audio data to the task execution unit 220.

[0102] In step S202, the task execution unit 220 of the voice processing device 20 receives voice data from the voice input unit 210. The task execution unit 220 reads out a trained model from the model storage unit 201.

[0103] In step S203, the task execution unit 220 of the voice processing device 20 executes a voice processing task based on the voice data input in step S201 and the trained model read out in step S202. Specifically, the task execution unit 220 inputs the voice data to the trained model. The trained model executes the voice processing task on the input voice data and outputs the execution result of the voice processing task. The task execution unit 220 acquires the execution result output from the trained model. The task execution unit 220 sends the execution result of the voice processing task to the result output unit 230.

[0104] In step S204, the result output unit 230 of the voice processing device 20 receives the execution result of the voice processing task from the task execution unit 220. The task execution unit 220 may display the execution result of the voice processing task on the display 506 of the voice processing device 20. The task execution unit 220 may transmit the execution result of the voice processing task to a terminal device equipped with a display via the communication network N.

[0105] <Effects of the embodiment> The speech processing device 20 according to this embodiment executes a speech processing task based on a trained model. The trained model is trained using speech data and labels obtained by converting feature vectors extracted from the speech data according to predetermined rules. In one aspect, this embodiment derives labels to be used in self-supervised learning using deterministic operations according to predetermined rules, without using statistical distributions of the dataset, thereby enabling efficient execution of the speech processing task.

[0106] The rule may be to quantize the feature vector to a predetermined number of integers. The rule may be to quantize the feature vector to an integer by converting each element of the feature vector to a single-digit β-adic number, where β is an integer equal to or greater than 2. In one aspect, according to this embodiment, the integers obtained by quantizing the elements of the feature vector are used as self-supervised labels, so that the self-supervised labels can be derived with a small amount of calculation.

[0107] The rule may quantize a portion of the feature vector to an integer. The portion of the feature vector may include elements of the feature vector with d dimensions or less, where d is an integer less than the number of elements in the feature vector. In one aspect, this embodiment allows for the type of self-supervised label to be adjusted, thereby enabling efficient generation of pre-trained models.

[0108] A part of the feature vector may include an element of the feature vector that indicates linguistic information. The feature vector may be Mel-frequency cepstral coefficients. In one aspect, according to this embodiment, the Mel-frequency cepstral coefficients store linguistic information in low-dimensional elements, so that a pre-trained model suitable for speech processing can be generated.

[0109] The trained model may be additionally trained using speech data and text data indicating the content of utterances included in the speech data. According to one aspect, the present embodiment makes it possible to efficiently generate trained models for performing various speech processing tasks.

[0110] The speech processing device 20 may execute a task of speech recognition of speech data by inputting the speech data into a trained model. According to one aspect, the present embodiment enables efficient speech recognition.

[0111] This embodiment enables self-supervised learning with high compatibility with speech processing tasks, enabling highly accurate recognition of, for example, speech containing noise or reverberation far from the microphone, as well as casual speech between people. Therefore, it can be used in business settings where accuracy is essential. For example, in workplaces where many people share tasks, it can support diverse work styles as a tool for automating voice communication processes, such as automatically creating minutes or reports, displaying subtitles during meetings, and even voice dialogue with AI agents. When applied to voice dialogue with AI agents, it can instantly recognize and analyze customer utterances and dynamically generate the next question, accurately grasping customer needs and providing appropriate recommendations.

[0112] [supplement] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each of the above-described functions.

[0113] The devices described in the examples are merely illustrative of one of several computing environments for implementing the embodiments disclosed herein. In some embodiments, model learning device 10 or speech processing device 20 includes multiple computing devices, such as a server cluster, configured to communicate with each other via any type of communications link, including a network, shared memory, etc., and to perform the processes disclosed herein.

[0114] For example, aspects of the present invention are as follows.

[0115] (Appendix 1) a task execution unit that executes tasks related to speech processing based on the trained model; The trained model is trained using voice data and labels obtained by converting feature vectors extracted from the voice data in accordance with predetermined rules. Audio processing device.

[0116] (Appendix 2) The rule quantizes the feature vector to a predetermined number of integers. 2. The audio processing device of claim 1.

[0117] (Appendix 3) The rule is that β is an integer equal to or greater than 2, and each element of the feature vector is converted into a single-digit β-ary number, thereby quantizing the feature vector to the integer. 3. The audio processing device of claim 2.

[0118] (Appendix 4) The rule quantizes a portion of the feature vector to the integer. 4. The audio processing device according to claim 2 or 3.

[0119] (Appendix 5) a part of the feature vector includes an element of the feature vector that indicates linguistic information; 5. The audio processing device of claim 4.

[0120] (Appendix 6) The part of the feature vector includes elements of the feature vector having d dimensions or less, where d is an integer less than the number of elements of the feature vector. 6. The audio processing device according to claim 4 or 5.

[0121] (Appendix 7) The feature vector is a Mel-frequency cepstral coefficient. 7. A speech processing device according to any one of appendices 1 to 6.

[0122] (Appendix 8) The trained model is additionally trained using voice data and text data indicating the content of utterances included in the voice data. 8. A speech processing device according to any one of claims 1 to 7.

[0123] (Appendix 9) further comprising a voice input unit that accepts input of second voice data; the task execution unit executes the task of speech recognition of the second speech data by inputting the second speech data to the trained model. 9. The audio processing device of claim 8.

[0124] (Appendix 10) A speech processing system including a model learning device and a speech processing device, The audio processing device includes: a task execution unit that executes tasks related to speech processing based on the trained model; The model learning device a feature extraction unit that extracts a feature vector from the speech data; a label conversion unit that converts the feature vector into a label according to a predetermined rule; a model generation unit that generates the trained model using the voice data and the label; A voice processing system comprising:

[0125] (Appendix 11) The computer Execute steps to perform a task related to speech processing based on the trained model; The trained model is trained using voice data and labels obtained by converting feature vectors extracted from the voice data in accordance with predetermined rules. Audio processing methods.

[0126] (Appendix 12) On the computer, Execute steps to perform a speech processing task based on the trained model, The trained model is trained using voice data and labels obtained by converting feature vectors extracted from the voice data in accordance with predetermined rules. program.

[0127] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0128] 10: Model learning device 20: Audio processing device 101: Unlabeled data storage unit 102: Labeled data storage unit 110: Feature extraction unit 120: Label conversion unit 130: Model generation unit 140: Additional Learning Section 201: Model memory unit 210: Audio input section 220: Task execution unit 230: Result output section 1000: Audio processing system [Prior art documents] [Patent documents]

[0129] [Patent Document 1] Japanese Patent Application Publication No. 2023-64381

Claims

1. a task execution unit that executes tasks related to speech processing based on the trained model; The trained model is trained using voice data and labels obtained by converting feature vectors extracted from the voice data in accordance with predetermined rules. Audio processing device.

2. The rule quantizes the feature vector to a predetermined number of integers. The audio processing device according to claim 1 .

3. The rule is that β is an integer equal to or greater than 2, and each element of the feature vector is converted into a single-digit β-ary number, thereby quantizing the feature vector to the integer. The audio processing device according to claim 2 .

4. The rule quantizes a portion of the feature vector to the integer. The audio processing device according to claim 2 .

5. a part of the feature vector includes an element of the feature vector that indicates linguistic information; The audio processing device according to claim 4 .

6. the part of the feature vector includes elements of the feature vector having d dimensions or less, where d is an integer less than the number of elements of the feature vector; The audio processing device according to claim 4 .

7. The feature vector is a Mel-frequency cepstral coefficient.

7. The audio processing device according to claim 1.

8. The trained model is additionally trained using voice data and text data indicating the content of utterances included in the voice data.

7. The audio processing device according to claim 1.

9. further comprising a voice input unit that accepts input of second voice data; the task execution unit executes the task of speech recognition of the second speech data by inputting the second speech data to the trained model. The audio processing device according to claim 8 .

10. A speech processing system including a model learning device and a speech processing device, The audio processing device includes: a task execution unit that executes tasks related to speech processing based on the trained model; The model learning device a feature extraction unit that extracts a feature vector from the speech data; a label conversion unit that converts the feature vector into a label according to a predetermined rule; a model generation unit that generates the trained model using the voice data and the label; A voice processing system comprising:

11. The computer Execute steps to perform a task related to speech processing based on the trained model; The trained model is trained using voice data and labels obtained by converting feature vectors extracted from the voice data in accordance with predetermined rules. Audio processing methods.

12. On the computer, Execute steps to perform a speech processing task based on the trained model, The trained model is trained using voice data and labels obtained by converting feature vectors extracted from the voice data in accordance with predetermined rules. program.

Images