Audio processing device, audio processing method, and audio processing program

Abstract

A first training device (10) calculates a first loss function that becomes smaller as a vector into which a model has quantized the feature amount of speech becomes closer to a context representation which the model has acquired from the feature amount, and a second loss function that becomes smaller as an accuracy with which the model identifies meta information of the speech on the basis of the context representation becomes higher. The first training device (10) updates the parameter of the model such that the first loss function becomes smaller and the second loss function becomes larger.

Description

[Technical Field]

[0001] The present invention relates to an audio processing device, an audio processing method, and an audio processing program. [Background technology]

[0002] It has been known that the accuracy of a specific subsequent task can be improved by transferring the parameters of a neural network trained by self-supervised learning using speech data to the subsequent task (see, for example, Non-Patent Document 1).

[0003] Here, the latter task refers to a task that uses speech as input, such as speech recognition. In self-supervised learning, parameters are learned so that a contextual representation can be acquired from speech, taking into account the preceding and following inputs. Transformer is known as a neural network that can acquire a contextual representation (see, for example, Non-Patent Document 2). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” NeurIPS, 2020. [Non-patent document 2] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, AN Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need", In Advances in NeurIPS, pp. 5998-6008, 2017. Summary of the Invention [Problem to be solved by the invention]

[0005] However, the conventional techniques have a problem in that the accuracy of later tasks in self-supervised learning may decrease.

[0006] For example, in the technology described in Non-Patent Document 1, the self-supervised learning model for speech may over-fit to the training data for self-supervised learning. In this case, a mismatch occurs between the self-supervised learning model and the data used in the subsequent task, making it impossible to acquire an effective representation for the subsequent task. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the object, the speech processing device is characterized by having a loss function calculation unit that calculates a first loss function that becomes smaller as a vector obtained by quantizing speech features by a model becomes closer to a context representation obtained by the model from the features, and a second loss function that becomes smaller as the accuracy with which the model identifies meta information of the speech based on the context representation increases, and an update unit that updates parameters of the model so that the first loss function becomes smaller and the second loss function becomes larger. [Effects of the Invention]

[0008] According to the present invention, it is possible to prevent a decrease in accuracy of subsequent tasks in self-supervised learning. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a first learning device. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of the second learning device. [Figure 3] FIG. 3 is a diagram illustrating an example of the configuration of the estimation device. [Figure 4] FIG. 4 is a flowchart showing the overall flow of the learning process. [Figure 5] FIG. 5 is a flowchart showing the flow of the self-supervised learning process. [Figure 6] FIG. 6 is a flowchart showing the flow of the re-learning process. [Figure 7] FIG. 7 is a flowchart showing the flow of the inference process. [Figure 8] FIG. 8 is a diagram illustrating an example of a computer that executes a learning program. DETAILED DESCRIPTION OF THE INVENTION

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A voice processing device, a voice processing method, and a voice processing program according to the present invention will be described in detail below with reference to the accompanying drawings. However, the present invention is not limited to the following embodiments.

[0011] [First embodiment] In the first embodiment, multiple models are trained. The models are, for example, neural networks, and include a speech encoder, a context network, a quantization network, a classification network, and an additional network. Details of each network will be described later.

[0012] The additional network is a neural network for calculating the final output in the subsequent task, such as classification, generation, or prediction.

[0013] This embodiment is particularly concerned with tasks targeting speech, including speech recognition for obtaining text from speech, speech classification for classifying speech into predetermined types (e.g., speaker attributes or emotions), and speaker identification for identifying speakers of speech.

[0014] The process of optimizing model parameters to improve task accuracy is called the training process, and the process of actually performing a task using one or more models, including additional networks trained by the training process, is called the inference process.

[0015] The learning process of this embodiment is made up of two stages: a self-supervised learning process and a re-learning process.

[0016] Here, the speech encoder and context network are referred to as a self-supervised learning model, which can be used for multiple different speech tasks, while the additional network is a model specialized for a particular task.

[0017] In the self-supervised learning process, a self-supervised model is trained, and in the retraining process, an additional network is trained using the self-supervised model that was trained in the self-supervised learning process.

[0018] In this embodiment, a first learning device 10 performs self-supervised learning processing. A second learning device 20 performs relearning processing. An inference device 30 performs inference processing. The first learning device 10, the second learning device 20, and the inference device 50 may be realized by different computers, or may be realized by a single computer.

[0019] The first learning device 10 is an example of a speech processing device. In addition, any one or more of the first learning device 10, the second learning device 20, and the inference device 50 can function as a speech processing device.

[0020] The configuration of the first learning device 10 will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the first learning device.

[0021] As shown in FIG. 1, the first learning device 10 stores a set of acoustic feature sequences X and classification labels l of meta information {(X1, l1), ..., (X M ,l M );l M ∈{l 1 ,…,l L}} is input as training data. The classification label l is the correct label.

[0022] Here, M is the number of pairs of speech feature sequences and classification labels included in the training data, and is an integer equal to or greater than 1. l is the lth classification label. L is the number of classification labels provided and is an integer of 2 or greater.

[0023] The meta information is information that indicates the domain of the voice (such as conversational voice at a call center, voice at an online conference, or voice reading), language, gender, and the like.

[0024] The acoustic features (elements of the acoustic feature sequence X) are, for example, log Mel filterbank coefficients (FBANK). The acoustic features are not limited to log Mel filterbanks, but may be Mel frequency cepstral coefficients (MFCCs), ΔMFCCs (first-order derivatives of MFCCs), ΔΔMFCCs (second-order derivatives of MFCCs), log powers, Δ log powers (first-order derivatives of log powers), etc. The acoustic features may also be raw speech samples.

[0025] Alternatively, the classification label may be represented by an L-dimensional 1-hot vector.

[0026] As shown in FIG. 1, the first learning device 10 includes a speech encoder unit 11, a context network unit 12, a quantization network unit 13, a classification network unit 14, a classification learning loss calculation unit 15, a context representation learning loss calculation unit 16, a learning parameter update unit 17, and model information 10a.

[0027] The model information 10a is the parameters of the model used in the first learning device 10. The parameters are the weights and biases of the neural network, etc. Furthermore, the model information 10a is updated as appropriate during the learning process.

[0028] The speech encoder unit 11 generates an acoustic feature sequence X={x1,...,x I}, the intermediate speech representation vector sequence Z={z1,…,z THere, I is the sequence length of the acoustic feature and is an integer equal to or greater than 1. Furthermore, T is the sequence length of the intermediate feature vector sequence of speech and is an integer equal to or greater than 1.

[0029] The speech encoder unit 11 calculates the speech intermediate feature vector sequence Z as shown in equation (1).

[0030]

number

[0031] Here, SpeechEncoder() is a function that has the functionality of a neural network, for example, a convolutional neural network.

[0032] θ se1 are parameters of the speech encoder and can be trained. se1 is read from the model information 10a.

[0033] The context network unit 12 applies masking to the intermediate feature vector sequence Z, which is the output of the speech encoder unit 11, as shown in equation (2).

[0034]

number

[0035] The context network unit 12 converts the masked intermediate feature vector sequence (the bar above C) into a context representation C={c1,...,c I}.

[0036]

number

[0037] Here, Masking() is a function that performs masking in the time direction.

[0038] ContextNetwork() is a function having the functionality of a neural network, such as the Transformer described in Non-Patent Document 2.

[0039] θ se2 are the parameters of the context network and can be learned. se2 is read from the model information 10a.

[0040] The quantization network unit 13 generates a quantized speech representation vector sequence Q={q1,...,q I}.

[0041]

number

[0042] Here, QuantizationNetwork() is a function having the functionality of a neural network, and is composed of, for example, a fully connected neural network and a Gumbel softmax function.

[0043] The Gumbel softmax function is a differentiable function used to propagate the output of a classifier (e.g., a fully connected neural network) to a subsequent network. The Gumbel softmax function is described in, for example, Reference 1.

[0044] θ qn are the parameters of the quantization network and can be learned. qn is read from the model information 10a.

[0045] Reference 1: E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with Gumbel-softmax,” ICLR, 2017.

[0046] The classification network unit 14 derives a probability sequence O={o l1 ,…,o lL The number of dimensions of the probability sequence O is L. Each element of the probability sequence O is calculated by dividing each element of the classification label l by {l 1 ,…,l L} is supported.

[0047]

number

[0048]

number

[0049] Here, GRL() is a function that represents the Gradient reversal layer (see, for example, Reference 1), and is a function that inverts the sign of the gradient when going backward in the backpropagation method.

[0050] ClassNetwork() (classification network) is a function that has the functionality of a neural network, and is composed of, for example, a fully connected neural network and a softmax function.

[0051] θ qn are the parameters of the quantization network and can be learned. qn is read from the model information 10a.

[0052] The classification learning loss calculation unit 15 calculates the classification learning loss L for the classification label l. class is calculated as in equation (7).

[0053]

number

[0054] ClassLoss() is a function that calculates the loss to identify the classification label l, for example, cross-entropy loss.

[0055] The context representation learning loss calculation unit 16 calculates the loss L for learning the context representation. context (Context representation learning loss) is calculated as in equation (8).

[0056]

number

[0057] ContextLoss() is a function that calculates the loss for learning contextual representations, such as contrastive loss.

[0058] We will explain contrastive loss. Sim() in equation (8) is a function that calculates the similarity between two vectors, for example, cosine similarity. ^Q (^ over Q) represents a set of negative examples of quantized vectors. τ is a temperature parameter that is set in advance.

[0059] In the numerator of the logarithm of the third side of equation (8), a pair (positive example) of an element of the quantized representation vector sequence Q and an element of the corresponding context representation vector sequence is used. For example, q t and c t are pairs of corresponding elements.

[0060] On the other hand, in the denominator of the logarithm of the third side of equation (8), a pair (negative example) of an element of the quantized representation vector sequence Q and an element of the non-corresponding context representation vector sequence is used. For example, q t and c t´(where t≠t´) is a pair of non-corresponding elements.

[0061] The learning parameter update unit 17 updates the classification learning loss L class and the context representation learning loss L context Based on this, the model parameters are updated.

[0062] Classification learning loss L class and loss L for learning context representation context The calculation of is performed for each mini-batch. Therefore, the learning parameter update unit 17 updates the parameters for each mini-batch.

[0063] Let the learning parameter Θ se ={θ se1 ,θ se1}, Θ qn ={θ qn}, Θ cn ={θ cn}, the learning parameter update unit 17 updates the parameters using equations (9), (10), and (11).

[0064]

number

[0065]

number

[0066]

number

[0067] Here, ε is the learning rate, β is the weight for the context representation learning loss, and γ is the weight for the classification learning loss. Also, α represents the weight, which adjusts the influence of the loss by changing it significantly as learning progresses (updates are repeated in mini-batch units).

[0068] As shown in equation (5), the function GRL() is introduced in the calculation of the classification network unit 14, so the sign of the term with α in equation (11) is inverted.

[0069] In equation (11), the learning parameters are updated so that they become smaller as the vector quantized by the model for the input approaches the acquired context representation and the accuracy of identifying meta-information based on the acquired context representation increases.

[0070] In this way, the learning parameter update unit 17 uses a first loss function (context representation learning loss L ) that becomes smaller as the vector obtained by quantizing the voice feature quantity by the model approaches the context representation acquired by the model from the feature quantity. context ), and the second loss function (classification learning loss L class ) is calculated. Then, the learning parameter update unit 17 updates the parameters of the model so that the first loss function becomes smaller and the second loss function becomes larger (Equation (11)). In this case, the learning parameter update unit 17 corresponds to a loss function calculation unit and an update unit.

[0071] More specifically, the part in the parentheses on the right side of equation (11) is a loss function (third loss function) obtained by subtracting the second term (term marked with α) of the second loss function to which a weight α, which increases as the number of times the parameter update by the update unit is repeated, is added from the first term (term marked with β) which is the first loss function.

[0072] In this embodiment, in order to prevent overfitting of the model to the training data in the self-supervised learning process, an adversarial neural network (ANN) is implemented using equation (11). ANN is described in Reference 1.

[0073] In this embodiment, the ANN is trained to prevent the context network from identifying meta-information about speech, which allows the context network to acquire a universal representation without overfitting to the training data.

[0074] For example, if the classification label l indicates a domain of speech, according to this embodiment, the context network operates robustly even for speech in an unknown domain.

[0075] Meta-information is information such as the domain of the voice, characteristics of the voice (language, etc.), and attributes of the speaker of the voice (gender, age), and is different from the content of the utterance expressed in text, etc. The content of the utterance may also be rephrased as the content of the information conveyed by the voice.

[0076] The learning parameter update unit 17 updates the parameter θ se1 , θ se2 , θ qn , θ cn After the parameter is updated, the process is repeated using the updated parameter. If a predetermined condition (for example, the number of repetitions) is met, the repetitive process ends.

[0077] Here, the context network unit 12 is an example of a context representation calculation unit that inputs audio features into a model and calculates a context representation. The classification network unit 14 is an example of a meta information label calculation unit that inputs a context representation into a model and calculates a label that identifies audio meta information. The quantization network unit 13 is an example of a quantization vector calculation unit that inputs audio features into a model and calculates a quantized vector.

[0078] Thus, the learning parameter update unit 17 calculates the first loss function so that the closer the vector calculated by the quantization vector calculation unit is to the context representation calculated by the context representation calculation unit, the smaller the first loss function becomes, and calculates the second loss function so that the higher the accuracy of the label calculation by the meta information label calculation unit, the smaller the second loss function becomes.

[0079] The configuration of the second learning device 20 will be described with reference to FIG. 2. FIG. 2 is a diagram illustrating an example of the configuration of the second learning device. The second learning device 20 uses parameters updated by the first learning device 10 to train a re-learning model that performs a task related to speech. The model in the first learning device 10 is a model that combines a speech encoder, a context network, a quantization network, and a classification network. On the other hand, the re-learning model is a model that combines a speech encoder, a context network, and an additional network.

[0080] 2, a set of an acoustic feature sequence X and a subsequent task label l' is input as training data to the second learning device 20. The subsequent task label l' is a correct label.

[0081] Here, the subsequent task label l' corresponds to information according to the task, and does not necessarily indicate meta information.

[0082] For example, if the task is speech recognition to obtain text from speech, the subsequent task label l' is text corresponding to the speech. Furthermore, the subsequent task label l' may indicate meta-information, similar to the classification label l. Note that the processing unit of the text corresponding to the speech in the subsequent task label l' may be a phoneme, a character, or a word.

[0083] As shown in FIG. 2, the second learning device 20 includes a speech encoder unit 21, a context network unit 22, an additional network unit 23, a subsequent task learning loss calculation unit 24, a learning parameter update unit 25, and model information 20a.

[0084] The model information 20a is the parameters of the model learned by the first learning device 10. The model information 20a includes at least the parameter θ se1 and θ se2 Furthermore, the model information 20a includes parameters θ of an additional network according to the task.add Includes.

[0085] Similar to the speech encoder 11, the speech encoder 21 encodes an acoustic feature sequence X={x1,...,x I}, calculate the intermediate representation vector sequence Z of the audio.

[0086] The speech encoder unit 21 calculates the speech intermediate feature vector sequence Z as shown in equation (1).

[0087] θ se1 are parameters of the speech encoder updated by the first learning device 10 and are read from the model information 20a.

[0088] Similar to the context network unit 12, the context network unit 22 converts the intermediate feature vector sequence Z, which is the output of the speech encoder unit 21, into a context representation C as shown in equation (12). However, unlike the context network unit 12, the context network unit 22 does not perform masking.

[0089]

number

[0090] θ se2 are parameters of the context network updated by the first learning device 10 and are read from the model information 10a.

[0091] The additional network unit 23 calculates a probability sequence P (sequence of predicted probabilities) for the subsequent task label from the context expression vector sequence C that is the output of the context network unit 22, as in equation (13).

[0092]

number

[0093] The ClassNetwork( ) (classification network) in equation (13) is different from the classification network of the first learning device 10, and learning is performed in the second learning device 20.

[0094] For example, the classification network of the second learning device 20 is a function having the functionality of a neural network, and is configured by, for example, a bidirectional LSTM and a softmax function.

[0095] θ add are the parameters of the classification network for the subsequent task and can be trained. addn is read from the model information 20a.

[0096] The subsequent task learning loss calculation unit 24 calculates the subsequent task learning loss L for the subsequent task label l'. down is calculated as in equation (14).

[0097]

number

[0098] Loss() is a function that calculates the loss of a subsequent task (e.g., classification loss), such as cross-entropy loss. Note that Loss() is changed as appropriate depending on the type of subsequent task (e.g., classification task, generation task, prediction task, etc.).

[0099] The learning parameter update unit 25 updates the loss L down Based on this, the model parameters are updated.

[0100] The learning parameter update unit 25 may fix some parameters and update other parameters. For example, the learning parameter update unit 25 may update the parameter θ se1 and θ se2 Without updating the parameter θ add Update.

[0101] Loss of subsequent tasks L downThe calculation of is performed for each mini-batch. Therefore, the learning parameter update unit 25 updates the parameters for each mini-batch.

[0102] After the parameters are updated by the learning parameter update unit 25, the process is further repeated using the updated parameters. When a predetermined condition (for example, the number of repetitions) is satisfied, the repetitive process ends.

[0103] An inference process using a trained model will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the configuration of an estimation device. The inference device 50 executes a task using a re-trained model.

[0104] 3, an acoustic feature sequence X is input as training data to the inference device 50. For example, the inference device 50 estimates a label corresponding to the acoustic feature sequence X.

[0105] As shown in FIG. 3, the inference device 50 includes a speech encoder unit 51, a context network unit 52, an additional network unit 53, and model information 50a.

[0106] The model information 50a is the parameters of each model trained by the first training device 10 and the second training device 20. The model information 50a includes trained parameters θ se1 and the parameters θ of the trained context network se2 The model information 50a also includes the parameters θ of the trained additional network. add Includes.

[0107] Similar to the speech encoder 21, the speech encoder 51 encodes an acoustic feature sequence X={x1,...,x I}, calculate the intermediate representation vector sequence Z of the audio.

[0108] The context network unit 52 converts the intermediate feature vector sequence Z, which is the output of the speech encoder unit 51, into a context representation C, similar to the context network unit 22.

[0109] The additional network unit 23 calculates a probability sequence P (a sequence of predicted probabilities) for the subsequent task label from the context expression vector sequence C that is the output of the context network unit 52.

[0110] The additional network unit 53 outputs a classification result based on the probability sequence P. The additional network unit 53 may output the probability sequence P, or may output information specifying a subsequent task label corresponding to the element with the largest value among the elements of the probability sequence P.

[0111] [Processing of the first embodiment] The flow of the learning process and the inference process of the first embodiment will be described with reference to FIGS.

[0112] 4 is a flowchart showing the overall flow of the learning process. As shown in Fig. 4, first, the first learning device 10 performs pre-learning of the speech encoder, the context network, the quantization network, and the classification network (step S1).

[0113] Next, the second training device 20 trains an additional network using the trained speech encoder and context network (step S2). At this time, it is also possible to retrain the speech encoder and context network.

[0114] 5 is a flowchart showing the flow of the self-supervised learning process, which corresponds to the process in step S1 in FIG.

[0115] As shown in FIG. 5, first, the first learning device 10 inputs an acoustic feature sequence to a speech encoder and calculates an intermediate representation vector sequence (step S101).

[0116] Next, the first learning device 10 applies masking to the intermediate representation vector sequence and inputs it to the context network to calculate a context representation vector sequence (step S102).

[0117] Furthermore, the first learning device 10 inputs the intermediate representation vector sequence to the quantization network and calculates a quantized representation vector sequence (step S103).

[0118] Next, the first learning device 10 applies GRL to the context expression vector sequence, inputs it to the classification network, and calculates a probability sequence for the classification label of the meta information (step S104).

[0119] Then, the first learning device 10 calculates the classification learning loss based on the calculated probability sequence and the correct classification label of the meta information (step S105).

[0120] Furthermore, the first learning device 10 calculates a context representation learning loss based on the context representation vector sequence and the quantized representation vector sequence (step S106).

[0121] Furthermore, the first learning device 10 updates the parameters of the speech encoder, the context network, the quantization network, and the classification network based on the classification learning loss and the context representation learning (step S107).

[0122] If the termination condition is satisfied (step S108, Yes), the first learning device 10 terminates the processing. On the other hand, if the termination condition is not satisfied (step S108, No), the first learning device 10 returns to step S101 and repeats the processing using the model with updated parameters.

[0123] The termination condition is, for example, that the process has been repeated a certain number of times, that the update amount of the parameters has converged, or the like.

[0124] 6 is a flowchart showing the flow of the re-learning process, which corresponds to the process in step S2 in FIG.

[0125] As shown in FIG. 6, first, the second learning device 20 inputs an acoustic feature sequence to a speech encoder and calculates an intermediate representation vector sequence (step S201).

[0126] Next, the second learning device 20 inputs the intermediate representation vector sequence into the context network and calculates a context representation vector sequence (step S202).

[0127] Next, the second learning device 20 inputs the context expression vector sequence into the additional network, and calculates a probability sequence for the classification label according to the task (step S203).

[0128] Then, the second learning device 20 calculates the additional learning loss based on the calculated probability sequence and the correct classification label according to the task (step S204).

[0129] Furthermore, the second learning device 20 updates the parameters of the additional network based on the additional learning loss (step S205). At this time, it is also possible to re-learn the speech encoder and the context network.

[0130] If the termination condition is satisfied (step S206, Yes), the second learning device 20 terminates the processing. On the other hand, if the termination condition is not satisfied (step S206, No), the second learning device 20 returns to step S201 and repeats the processing using the model with updated parameters.

[0131] The termination condition is, for example, that the process has been repeated a certain number of times, that the update amount of the parameters has converged, or the like.

[0132] FIG. 7 is a flowchart showing the flow of the inference process.

[0133] As shown in FIG. 7, first, the inference device 50 inputs an acoustic feature sequence into a speech encoder and calculates an intermediate representation vector sequence (step S501).

[0134] Next, the inference device 50 inputs the intermediate representation vector sequence into the context network and calculates a context representation vector sequence (step S502).

[0135] Next, the inference device 50 inputs the context expression vector sequence into the additive network and calculates a probability sequence for the classification label according to the task (step S503).

[0136] Then, inference device 50 outputs the classification result based on the calculated probability sequence (step S504).

[0137] [Effects of the first embodiment] As described above, the first learning device 10 calculates a first loss function that decreases as the vector obtained by quantizing speech features by the model approaches the context representation acquired by the model from the features, and a second loss function that decreases as the accuracy with which the model identifies speech meta information based on the context representation increases. The first learning device 10 updates the model parameters so that the first loss function decreases and the second loss function increases. This prevents overfitting of the context network to the learning data and suppresses a decrease in the accuracy of subsequent tasks in self-supervised learning.

[0138] The first learning device 10 calculates a third loss function by subtracting a second term, which is a second loss function with a weight that increases the more times parameter updates are repeated, from the first term, which is the first loss function, and updates the parameters of the model so that the third loss function becomes smaller. This makes it possible to reduce the impact of improving the accuracy of the classification network as learning progresses.

[0139] The first learning device 10 uses the updated parameters to retrain a model that executes a speech-related task, thereby enabling the additional network to execute subsequent tasks with high accuracy.

[0140] The first learning device 10 inputs audio features into a model, calculates a context representation, inputs the context representation into the model, calculates a label that identifies audio meta information, inputs the audio features into the model, and calculates a quantized vector. The first learning device 10 calculates a first loss function so that the closer the calculated vector is to the calculated context representation, the smaller the loss function becomes, and calculates a second loss function so that the higher the accuracy of the label calculation, the smaller the loss function becomes. This allows the first learning device 10 to consistently perform calculations using the model and update parameters.

[0141] The speech processing device according to the first embodiment provides certain improvements over conventional machine learning techniques such as those described in Non-Patent Document 1, and represents an advancement in the technical field of speech tasks using machine learning models.

[0142] [System configuration, etc.] Furthermore, the components of each device shown in the figure are functional concepts and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of the devices can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic. Note that the program may be executed not only by the CPU but also by other processors such as a GPU.

[0143] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method.In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[0144] [program] In one embodiment, the speech processing device (first learning device 10, second learning device 20, or inference device 50) can be implemented by installing a program that executes the above-described processing as package software or online software on a desired computer. For example, by executing the above-described program on an information processing device, the information processing device can function as a speech processing device. The information processing device referred to here includes desktop and notebook personal computers. In addition, information processing devices also include mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as slate terminals such as PDAs (Personal Digital Assistants).

[0145] The speech processing device can also be implemented as a learning server device that uses a terminal device used by a user as a client and provides services related to the above-mentioned learning process to the client. For example, the learning server device is implemented as a server device that provides a learning service that receives learning data as input and outputs trained model parameters. In this case, the learning server device may be implemented as a web server or as a cloud that provides services related to the above-mentioned learning process through outsourcing.

[0146] 8 is a diagram showing an example of a computer that executes a learning program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0147] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0148] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the program that defines each process of the learning device 5 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing processes similar to those of the functional configuration of the learning device 5 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0149] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.

[0150] The program module 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, but may also be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0151] The following additional notes are provided regarding the above-described embodiments.

[0152] (Additional note 1) Memory and at least one processor coupled to said memory; Including, The processor: calculating a first loss function that decreases as the vector obtained by quantizing the features of the audio by the model approaches the context representation acquired by the model from the features, and a second loss function that decreases as the accuracy with which the model identifies the meta information of the audio based on the context representation increases; Update the parameters of the model so that the first loss function is reduced and the second loss function is increased. Audio processing device.

[0153] (Additional note 2) Item 10. The speech processing device according to item 1, wherein the processor: calculating a third loss function by subtracting a second term, which is a weight assigned to the second loss function and which increases as the number of times parameter updates are repeated, from a first term, which is the first loss function; Update the parameters of the model so that the third loss function becomes smaller. Audio processing device.

[0154] (Additional note 3) Item 10. The speech processing device according to item 1, wherein the processor: Retrain the model to perform speech tasks using the updated parameters. Audio processing device.

[0155] (Additional note 4) Item 10. The speech processing device according to item 1, wherein the processor: inputting the speech features into the model to calculate a context representation; inputting the context representation into the model and computing labels identifying meta-information of the audio; Inputting the speech features into the model, calculating a quantized vector, The first loss function is calculated so that the closer the calculated vector is to the calculated context representation, the smaller the first loss function is calculated, and the higher the accuracy of the label calculation, the smaller the second loss function is calculated. Audio processing device.

[0156] (Additional note 5) Item 10. The speech processing device according to item 1, wherein the processor: Perform the task using the retrained model. Audio processing device.

[0157] (Additional note 6) A non-transitory storage medium storing a program executable by a computer to perform audio processing, The audio processing calculating a first loss function that decreases as the vector obtained by quantizing the features of the audio by the model approaches the context representation acquired by the model from the features, and a second loss function that decreases as the accuracy with which the model identifies the meta information of the audio based on the context representation increases; Update the parameters of the model so that the first loss function is reduced and the second loss function is increased. Non-transitory storage medium.

[0158] (Additional note 7) an inference unit that calculates a first loss function that decreases as a vector obtained by quantizing audio features by a model approaches a context representation acquired by the model from the features, and a second loss function that decreases as the accuracy with which the model identifies meta information of the audio based on the context representation increases; and that performs inference processing related to audio using a re-trained model that has been trained using parameters of the model that has been trained through a pre-training process that updates parameters of the model so that the first loss function decreases and the second loss function increases. [Explanation of symbols]

[0159] 10 First Learning Device 10a, 20a, 50a model information 11, 21, 51 Audio encoder section 12, 22, 52 Context Network Section 13 Quantization Network Section 14 Classification Network Division 15 Classification learning loss calculation unit 16 Context Representation Learning Loss Calculation Unit 17, 25 Learning parameter update section 24 Post-task learning loss calculation unit 23, 53 Additional Network Section

Claims

1. a loss function calculation unit that calculates a first loss function that decreases as a vector obtained by quantizing speech features by a model approaches a context representation acquired by the model from the features, and a second loss function that decreases as the accuracy with which the model identifies meta information of the speech based on the context representation increases; an update unit that updates parameters of the model so that the first loss function becomes smaller and the second loss function becomes larger; 10. A voice processing device comprising:

2. the loss function calculation unit calculates a third loss function by subtracting, from a first term that is the first loss function, a second term that is a weight assigned to the second loss function, the weight being larger as the number of times that the parameter update by the update unit is repeated is larger; The update unit updates parameters of the model so that the third loss function becomes smaller.

2. The audio processing device according to claim 1, wherein:

3. 2. The speech processing device according to claim 1, further comprising an additional learning unit that uses the parameters updated by the update unit to learn a re-learning model that executes a speech-related task.

4. a context expression calculation unit that inputs the feature of the speech into the model and calculates a context expression; a meta-information label calculation unit that inputs the context representation into the model and calculates a label that identifies meta-information of the audio; a quantization vector calculation unit that inputs the feature of the speech into the model and calculates a quantized vector; and The loss function calculation unit calculates the first loss function so that the closer the vector calculated by the quantization vector calculation unit is to the context representation calculated by the context representation calculation unit, the smaller the first loss function becomes, and calculates the second loss function so that the higher the calculation accuracy of the label of the meta information label calculation unit is, the smaller the second loss function becomes.

2. The audio processing device according to claim 1, wherein:

5. The speech processing device according to claim 3 , further comprising an inference unit that executes the task using the re-learning model.

6. 1. A method of audio processing performed by an audio processing device, comprising: a loss function calculation step of calculating a first loss function that decreases as a vector obtained by quantizing speech features by a model approaches a context representation acquired by the model from the features, and a second loss function that decreases as the accuracy with which the model identifies meta information of the speech based on the context representation increases; an updating step of updating parameters of the model so that the first loss function is reduced and the second loss function is increased; 10. A speech processing method comprising:

7. a loss function calculation step of calculating a first loss function that decreases as a vector obtained by quantizing speech features by a model approaches a context representation acquired by the model from the features, and a second loss function that decreases as the accuracy with which the model identifies meta information of the speech based on the context representation increases; an updating step of updating parameters of the model so that the first loss function is reduced and the second loss function is increased; A speech processing program characterized by causing a computer to execute the above.

Images