Speech recognition device, speech recognition method, and speech recognition program

Abstract

A speech recognition device (10) includes a generation unit that inputs speech to one machine learning model trained in advance and outputs a plurality of types of a text corresponding to the speech.

Description

Speech recognition device, speech recognition method, and speech recognition program

[0001] The present invention relates to a voice recognition device, a voice recognition method, and a voice recognition program.

[0002] With the development of deep learning, end-to-end neural speech recognition, which takes speech features as input and outputs a transcription of the speech, has achieved high performance. The speech text output from speech recognition includes disfluencies such as fillers and hesitations.

[0003] However, these disfluencies are important elements for analyzing speaker interactions, so their removal is not necessarily required, and they are an important phenomenon, especially in applications such as dialogue.On the other hand, speech recognition results are often used as input for natural language processing techniques such as summarization, translation, and template filling.

[0004] Natural language processing technology assumes that input text includes punctuation and does not include disfluent expressions. Therefore, if spoken text is input as is, performance may be significantly degraded. For this reason, there are cases where it is necessary to convert spoken text into written text. Therefore, technologies have been proposed that convert the spoken language, which is the result of speech recognition, into written language.

[0005] Specifically, in conventional techniques, written text is generated by rewriting spoken text using a spoken-to-written conversion model for converting spoken text into written text (Patent Document 1).

[0006] International Publication No. 2021 / 166145

[0007] However, the method described in Patent Document 1 has a problem in that it is not possible to optimize written text directly from speech, and therefore the generation of written text is not robust against erroneous conversion to spoken text.

[0008] In addition, applications for processing speech may be required to output both spoken and written text. This task of outputting multiple types of text from speech input is not limited to outputting spoken and written text, but can also occur when outputting, for example, the results of speech recognition and speech translation.

[0009] The present invention has been made in consideration of the above, and aims to provide a speech recognition device, a speech recognition method, and a speech recognition program that are simply configured and can accurately output multiple types of text (e.g., both spoken and written language) from input speech.

[0010] In order to solve the above-mentioned problems and achieve the object, the speech recognition device of the present invention is characterized by having a generation unit that inputs speech into a single pre-trained machine learning model and outputs multiple types of text corresponding to the speech.

[0011] According to the present invention, a plurality of types of text can be accurately output from input speech with a simple configuration.

[0012] FIG. 1 is a diagram illustrating the configuration of a speech recognition device according to an embodiment. FIG. 2 is a diagram illustrating the system configuration of the speech recognition device shown in FIG. 1 during learning. FIG. 3 is a diagram illustrating speech recognition processing by the speech recognition device shown in FIG. 1. FIG. 4 is a flowchart illustrating the processing procedure of the speech recognition processing according to an embodiment. FIG. 5 is a flowchart illustrating the processing procedure of the learning processing according to an embodiment. FIG. 6 is a diagram illustrating an example of the configuration of the speech recognition device shown in FIG. 1. FIG. 7 is a diagram illustrating the configuration of a speech recognition device according to the prior art. FIG. 8 is a diagram illustrating an example of a computer on which a speech recognition device is realized by executing a program.

[0013] An embodiment of the present invention will be described in detail below with reference to the drawings. However, the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are denoted by the same reference numerals. In the following, when "^A" is written for A, it is considered to be equivalent to "a symbol with "^" written immediately above "A".

[0014] [Embodiment] In the embodiment, a speech recognition device is described that, when speech is input, outputs both spoken text (first text) and written text (second text) different from the spoken text using one model (machine learning model).

[0015] The speech recognition device according to the embodiment functions as a generator that inputs speech into a single pre-trained machine learning model (hereinafter referred to as a model) and outputs multiple types of text corresponding to the speech. Here, the types of text include language types such as Japanese and English, as well as spoken and written forms and dialects within the same language. The types of text may also include markup languages ​​such as HTML and programming languages. These various types of text may hereinafter be referred to as "text domains."

[0016] Specifically, the speech recognition device of the embodiment outputs, as the output of the model when speech is input, a text that concatenates two texts: a spoken text and a written text that is different from the spoken text (hereinafter referred to as spoken / written text).

[0017] 1 is a diagram illustrating the configuration of a speech recognition device according to an embodiment. The speech recognition device 10 according to the embodiment is realized by, for example, loading a predetermined program into a computer or the like including a ROM (Read Only Memory), a RAM (Random Access Memory), a CPU (Central Processing Unit), etc., and causing the CPU to execute the predetermined program. The speech recognition device 10 also has a communication interface for transmitting and receiving various information to and from other devices connected via a network, etc.

[0018] The speech recognition device 10 (generation unit) inputs acoustic features extracted from speech into a single pre-trained machine learning model and outputs two types of text, spoken language text and written language text, as multiple types of text corresponding to the speech. The speech recognition device 10 is configured, for example, with a neural network. The speech recognition device 10 includes, for example, a feature conversion unit 11 and an output generation unit 12.

[0019] The feature transform unit 11 receives an input of acoustic features extracted from speech and transforms the features. The feature transform unit 11 is, for example, an encoder.

[0020] When the output generation unit 12 receives input of the features converted by the feature conversion unit 11, it concatenates spoken text and written text as multiple types of text and outputs the concatenated spoken text and written text as estimated text. The output generation unit 12 is, for example, a decoder. The spoken and written text is, for example, a series concatenation of spoken text and written text.

[0021] The speech recognition device 10 has a configuration in which, for example, one model (the feature conversion unit 11 and the output generation unit 12) is shared for outputting spoken text and written text. In other words, when speech is input, the speech recognition device 10 uses one model to output spoken and written text. That is, the speech recognition device 10 outputs both spoken and written text by outputting spoken and written text.

[0022] Although an encoder-decoder model has been used as an example of the model structure of the feature conversion unit 11 and the output generation unit 12 of the speech recognition device 10, the model is not limited to the encoder-decoder model as long as it is a model that can perform sequence conversion.

[0023] Furthermore, by setting special tokens, the speech recognition device 10 can switch between three output patterns: spoken text, written text, and spoken and written text when generating estimated text. The special tokens are tokens separate from the tokens to be processed, and are used to define the output of the output generation unit 12 as either spoken text, written text, or spoken and written text. Note that there are as many patterns of special tokens for defining the output as there are types of text to be output and their combinations.

[0024] [Learning Phase] Next, a description will be given of the learning phase in the speech recognition device 10. Fig. 2 is a diagram showing the system configuration during learning of the speech recognition device 10 shown in Fig. 1. Here, a method for simultaneously learning spoken language only, written language only, and both spoken and written language text will be described.

[0025] The speech recognition system 100 is realized, for example, by loading a predetermined program into a computer or the like including a ROM, a RAM, a CPU, etc., and having the CPU execute the predetermined program. The speech recognition system 100 also has a communication interface for transmitting and receiving various information to and from other devices connected via a network, etc.

[0026] As shown in FIG. 2, the speech recognition system 100 includes a learning unit 30 that uses training data D for the speech recognition device 10 to update model parameters Θ of the model included in the speech recognition device 10 .

[0027] The training data D includes at least acoustic features extracted from speech (which may be speech) and paired data of spoken text and written text corresponding to the acoustic features, which are correct answer texts (token sequences). The training data D includes the spoken and written texts as correct answer texts. The training unit 30 optimizes the model parameters Θ of the model of the speech recognition device 10 by performing supervised learning of the model held by the speech recognition device 10 using each paired data of the training data D.

[0028] The speech recognition device 10 estimates spoken / written text (estimated text (token sequence)) using as input the acoustic features of training data D, estimated text estimated by the speech recognition device 10, and special tokens that define the output of the output generation unit 12. The model parameter training unit 20 updates the model parameters Θ so that the spoken / written text estimated by the speech recognition device 10 approaches the correct text corresponding to the acoustic features input as training data.

[0029] Here, in addition to pairs of speech and spoken / written text, paired data of speech and spoken text, speech and written text, and speech and spoken / written text may be prepared as the training data D. Also, a special token may be associated with each paired data as the training data D. In this case, since it is sufficient to input the training acoustic features and the special tokens corresponding to these acoustic features to the speech recognition device 10 as the training data D, there is no need to separately input the special tokens.

[0030] As described above, when speech is input, the speech recognition device 10 uses a single model to output both spoken and written text, i.e., both spoken and written text. Training long texts such as spoken and written texts is generally more difficult than training short texts. For this reason, multitask training, which allows both spoken and written texts to be used as training data, may be employed for modeling the speech recognition device 10.

[0031] From here, we will explain how to study spoken text only, written text only, and both spoken and written text simultaneously.

[0032] First, we prepare paired data of speech and spoken text, speech and written text, and speech and spoken / written text as training data. At this time, we distinguish each text by giving special tokens, such as [spoken] for spoken text, [written] for written text, and [spoken2written] for spoken / written text.

[0033] The training unit 30 uses these special tokens to train all data into one common model (e.g., an encoder-decoder model, a decoder-only model, etc.) of the speech recognition device 10. At this time, the [spoken] and [written] data can be used whether they are paired or not. The model parameter training unit 20 updates the model parameter Θ so that the spoken text, written text, or spoken and written text estimated by the speech recognition device 10 based on the special tokens approaches each correct answer text corresponding to the acoustic features input as training data.

[0034] [Estimation Phase] Next, the estimation phase in the speech recognition device 10 will be described with reference to Fig. 3. Fig. 2 is a diagram for explaining the speech recognition process by the speech recognition device 10 shown in Fig. 1.

[0035] 3, the speech recognition device 10 estimates and outputs spoken and written text using the acoustic features of the recognition target, estimated text estimated by the speech recognition device 10, and special tokens that define the output of the output generation unit 12. The model parameters Θ optimized in the training phase are applied to the model of the speech recognition device 10.

[0036] If the speech recognition device 10 is required to output estimated text of a fixed pattern, there is no need to provide a special token from outside the speech recognition device 10. For example, if the output pattern of estimated text is a spoken / written text, the estimated text can be fixed to be defined as a spoken / written text, and a special token that defines the output pattern of estimated text as a spoken / written text can be used each time. In this case, the speech recognition device 10 can always use [spoken2written] as the special token.

[0037] On the other hand, when the output pattern of the speech recognition device 10 is to be switched arbitrarily, a special token may be provided from outside the speech recognition device 10, or the speech recognition device 10 may read the output switching setting. For example, there may be a case where it is desired to switch between [spoken2written] and [written].

[0038] [Example] [Combination of Input and Output] Next, an example will be described. In this example, the output of the speech recognition device 10 is changed as follows depending on the special token that is input. Therefore, the combination of the special token that is input and the estimated text that is output will be described.

[0039] In the following example, an encoder-decoder model will be described as an example of the model structure of the feature conversion unit 11 and the output generation unit 12 of the speech recognition device 10. The model applied to the speech recognition device 10 may be any model that can perform sequence conversion.

[0040] When a first input is a sequence of acoustic features X, a special token p=[spoken] for spoken text, and model parameters Θ, a token sequence O=Y for spoken text is output as a first output from the output generation unit 12. Note that the sequence of acoustic features X is an input to the feature conversion unit 11. The special token p is an input to the output generation unit 12.

[0041] The model parameter Θ is used in common by the neural network components of the speech recognition device 10, and it only needs to be read once at the beginning of the inference phase. Therefore, it is not necessary to input the model parameter Θ for each inference process. This also applies to the following cases.

[0042] When the second input is a sequence of acoustic features X, a special token p=[written] for spoken text, and model parameters Θ, the output generator 12 outputs a token sequence O=Z for written text as a second output.

[0043] When the third input is a sequence of acoustic features X, a special token for spoken text p=[spoken2written], and model parameters Θ, the output generator 12 outputs a third output, a sequence of tokens O=Y[SEP]Z for spoken and written text.

[0044] The speech recognition device 10 receives an acoustic feature X and a special token p as input, and outputs O, which is the text specified by the special token p, through probability calculation according to the model parameter Θ.

[0045] Here, the input X is X = {x １ , ..., x Ｍ}, where m (1≦m≦M) indicates the order of the sequence of the input acoustic feature X. Also, the output O is expressed as O={^o １ , ..., ^o Ｎ}, where n (1≦n≦N) indicates the order of the word sequence in the estimated text, and p∈{[spoken], [written], [spoken2written]}.

[0046] In this embodiment, 80 log-mel scale filter bank features with a frame shift of 10 ms are used as acoustic features. However, any input format that can be used in deep learning may be used as the acoustic features, and other acoustic features may also be used. Furthermore, the signal waveform may be input directly to the speech recognition device 10.

[0047] Furthermore, the token sequence may be, for example, a space-delimited expression in the case of English, or, for example, a character-delimited expression in the case of Japanese. The estimated text output by the speech recognition device 10 is a vector sequence corresponding to a posterior probability distribution, or a text sequence. Specifically, the estimated text sequence is either a sequence of one-hot vectors representing the words of the estimated text, or a sequence of the words themselves of the estimated text.

[0048] The processing of the speech recognition device 10 can be realized by inputting the input acoustic features and special tokens to the next speech recognition device and learning the model parameters Θ (FIG. 2).

[0049] The model used by the speech recognition device 10 may be any neural speech recognition model. For example, the model used by the speech recognition device 10 may be, for example, a method using an attention-based encoder-decoder model (Reference 1) or a method using a Transformer (Reference 2). Reference 1: D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio. “END-TO-END ATTENTION-BASED LARGE VOCABULARY SPEECH RECOGNITION”, In Proc. ICASSP, pp. 4945-4949, 2016. Reference 2: L. Dong, S. Xu, and B. Xu., “SPEECH-TRANSFORMER: A NO-RECURRENCE SEQUENCE-TO-SEQUENCE MODEL FOR SPEECH RECOGNITION”, In Proc. ICASSP, pp. 5884-5888, 2018.

[0050] [Processing of the Speech Recognition Device] The output process of the estimated text by the speech recognition device 10 will be described. The input to the speech recognition device 10 is a special token p, a sequence of acoustic features X, and a token sequence ^o of the estimated text specified by the special tokens p from the 1st to the (n-1)th tokens already obtained as a result of speech recognition. １ , ..., ^o ｎ－１ , and model parameters Θ. The output of the speech recognition device 10 is the n-th estimated text word ^o ｎ The word ^o in the estimated text ｎ is output in the output pattern defined by the special token p.

[0051] A sequence of special tokens and acoustic features is input to the speech recognition device 10. The speech recognition device 10 also recognizes the n-th estimated text ^o ｎ When estimating the token sequence {^o １ , ..., ^o ｎ－１} is used.

[0052] Then, the speech recognition device 10 performs feature conversion on the input data, and then calculates the probability according to the model parameter Θ to obtain a posterior probability distribution P(o ｎ |^o １ , ..., ^o ｎ－１ , X, p; Θ). The speech recognition device 10 calculates the posterior probability distribution P(o ｎ |^o １ , ..., ^o ｎ－１ , X, p; Θ), the nth speech recognition result word ^o is calculated by the maximum likelihood criterion. ｎ The determination by the maximum likelihood criterion follows equation (1).

[0053]

[0054] The speech recognition device 10 performs the above process recursively to obtain a token sequence O={^o １ , ..., ^o Ｎ} is obtained. ｎ |^o １ , ..., ^oｎ－１ , X, p; Θ) and a detailed calculation method thereof will be described later.

[0055] [Speech Recognition Processing Procedure] Fig. 4 is a flowchart showing the speech recognition processing procedure according to an embodiment. When the speech recognition device 10 receives input of a sequence of acoustic features X and a special token p (step S1), it determines whether n>1 (step S2). The model parameter Θ is input to the speech recognition device 10 and applied to the model. Furthermore, if the output text pattern is not switched, external input of the special token p is not required. A fixed special token p may be provided each time.

[0056] If n>1 (step S2: Yes), the speech recognition device 10 converts the acoustic feature sequence X,1 to n-1 of the generated word sequence ^o into the acoustic feature sequence X,1. １ , ..., ^o ｎ－１ , and the special token p, feature conversion is performed (step S3). １ , ..., ^o ｎ－１ corresponds to the estimated text in Figure 2, and hereafter, this generated word sequence ^o １ , ..., ^o ｎ－１ is also referred to as the "generated word sequence." Note that the generated word sequence does not necessarily include all generated words ^o １ , ..., ^o ｎ－１ does not need to be included in the series.

[0057] If n>1 is not satisfied (step S2: No), that is, if n=1, the speech recognition device 10 performs feature conversion in the feature conversion unit 11 based on the acoustic feature sequence X and the special token p (step S4).

[0058] The output generation unit 12 generates a posterior probability distribution P(o) for the n-th word in the estimated text based on the feature converted by the feature conversion unit 11. ｎ |^o １ , ..., ^o ｎ－１ , X, p; Θ) (step S5). Then, the output generation unit 12 calculates the obtained posterior probability distribution P(o ｎ |^o １, ..., ^o ｎ－１ , X, p; Θ), the nth speech recognition result word ^o ｎ is determined (step S6).

[0059] The speech recognition device 10 sets n=n+1 (step S7) and determines whether n=N (step S8).

[0060] If n=N is not satisfied (step S8: No), the process returns to step S2. In this way, the speech recognition device 10 recursively performs the processes from step S2 onward to generate a token sequence O={^o １ , ..., ^o Ｎ If n=N (step S8: Yes), the speech recognition device 10 obtains a token sequence O={^o １ , ..., ^o Ｎ} is output.

[0061] [Optimization of Model Parameters] Next, optimization of the learning data D and the model parameters Θ used by the learning unit 30 will be described.

[0062] In this case, in the system (FIG. 2) during training of the speech recognition device 10, the input is a special token p, and a set D of pairs of a sequence of acoustic features and a token sequence of a spoken text (correct text) as training data. ａｓｒ (Equation (2)), a set D of pairs of a sequence of acoustic features and a token sequence of a written text (correct text) ｗ (Equation (3)), a set D of pairs of a sequence of acoustic features and a token sequence of spoken / written text (correct text) ｓ２ｗ (Equation (4)), and the output is the model parameter Θ. Note that if the special token is included in the correct text in advance, it does not need to be provided as a separate input.

[0063]

[0064]

[0065]

[0066] The model parameter training unit 20 updates the model parameters Θ so that the estimated text and the correct text are closer to each other. The model parameter training unit 20 trains a special token p, a set of pairs of a sequence X of acoustic features and a token sequence of spoken text, a token sequence of written text, or a token sequence of spoken and written text, and a posterior probability distribution P(o ｎ |^o １ , ..., ^o ｎ－１ , X, p; Θ), and then learning of the model parameters Θ is performed based on D={D ａｓｒ , D ｗ , D ｓ２ｗ The model parameters Θ optimized by} follow equation (5).

[0067]

[0068] The model parameter Θ learned here is used as Θ in the speech recognition device.

[0069] The model parameter learning unit 20 may use a general optimization method in neural network learning. Specifically, for an input speech (a sequence X of its acoustic features), one of a token sequence of spoken text, a token sequence of written text, and a token sequence of spoken and written text that corresponds to the speech and a special token p is used as a correct answer text, and a set of pairs of speech and correct answer text is used as learning data D, thereby performing supervised learning to optimize the model parameter Θ.

[0070] The training data D is a combination of speech (sequence X of acoustic features) and training data D of spoken language text. ａｓｒ , speech (sequence of acoustic features X) and written text training data D ｗ , speech (sequence of acoustic features X) and training data D of spoken and written text ｓ２ｗ Among these, the model parameter learning unit 20 at least selects a set D of pairs of (a sequence X of acoustic features of) speech and a token sequence of spoken / written text. ｓ２ｗHowever, if you want to switch the output using a special token in the inference phase, you need to use learning data that includes the output to be switched.

[0071] [Learning Processing Procedure] FIG. 5 is a flowchart showing the learning processing procedure according to the embodiment.

[0072] The model parameter learning unit 20 sets k=1 (step S21), where k (1≦k≦K) indicates the number of the token sequence. The speech recognition device 10 calculates the acoustic feature sequence X of the training data D. ｋ , special token p ｋ (Step S22). If the pattern of the output text is not to be switched, the special token p ｋ External input of the fixed special token p is not required. ｋ The speech recognition device 10 determines whether n>1 (step S23).

[0073] If n>1 (step S23: Yes), the speech recognition device 10 converts the acoustic feature sequence X ｋ , the n-1th word sequence o of the correct text ｋ １ , ..., o ｋ ｎ－１ , and special token p ｋ The feature transformation is performed based on the n-1th word sequence of the correct text (step S24). ｋ １ , ..., o ｋ ｎ－１ is also called the "correctly generated word sequence."

[0074] If n>1 is not satisfied (step S23: No), that is, if n=1, the speech recognition device 10 converts the acoustic feature sequence X ｋ , and special token p ｋ Feature transformation is performed based on the above (step S25).

[0075] Step S26 is the same process as step S5 shown in FIG. 4. Then, the model parameter learning unit 20 calculates a predetermined loss function (step S27). The loss function may be a general one used in neural network training, and may represent, for example, the error between the calculation result in step S26 and the correct text. Then, the speech recognition device 10 sets n=n+1 (step S28) and determines whether n=N (step S29). If n=N is not true (step S29: No), the process returns to step S23.

[0076] If n=N (step S29: Yes), the model parameter learning unit 20 sets k=k+1 (step S30) and determines whether k=K (step S31). If k=K is not true (step S31: No), the process returns to step S22.

[0077] If k=K (step S31: Yes), the model parameter learning unit 20 updates the model parameter Θ based on the calculation result of the loss function so that the token sequence of the estimated text approaches the ground truth text corresponding to the acoustic features input as training data (step S32).

[0078] If the predetermined termination condition is satisfied (step S33: Yes), the model parameter learning unit 20 outputs the optimized model parameters Θ to the speech recognition device 10 and terminates the learning process. The predetermined termination condition is, for example, when the number of parameter updates reaches a predetermined number, when the parameter update amount is equal to or less than a predetermined threshold, when the calculated loss is equal to or less than a predetermined threshold, etc.

[0079] If the predetermined termination condition is not satisfied (step S33: No), the speech recognition device 10 and the model parameter learning unit 20 return to step S21 and execute the processes from step S21 onward. ｋ , the word sequence generated one step before, and the special token p ｋ The feature transformation may be performed using the following.

[0080] [Detailed Configuration of Speech Recognition Apparatus] Next, a detailed configuration example of the speech recognition apparatus 10 will be described. Fig. 6 is a diagram showing an example of the configuration of the speech recognition apparatus 10 shown in Fig. 1 .

[0081] 1, the feature conversion unit 11 of the speech recognition device 10 includes an acoustic feature embedding vector calculation unit 111, an output token embedding vector calculation unit 112, and a context vector calculation unit 113. The output generation unit 12 of the speech recognition device 10 includes a posterior probability calculation unit 121 and a text generation unit 122.

[0082] As described above, the speech recognition device 10 calculates the posterior probability distribution P(o ｎ |^o １ , ..., ^o ｎ－１ , X, p; Θ) (Equation (1)) is calculated (FIG. 3). This calculation is realized by the acoustic feature embedding vector calculation unit 111, the acoustic feature embedding vector calculation unit 112, the output token embedding vector calculation unit 112, and the posterior probability calculation unit 121. Then, the text generation unit 122 calculates the posterior probability distribution P(o ｎ |^o １ , ..., ^o ｎ－１ , X, p; Θ), and the nth speech recognition result word ^o ｎ Determine.

[0083] Here, the special token p is input to the output token embedding vector calculation unit 112. Hereinafter, the posterior probability distribution P(o ｎ |^o １ , ..., ^o ｎ－１ , X, p; Θ) where the model parameters are Θ = {θ ｈ , θ ｕ , θ ｓ , θ ｐ}.

[0084] [Acoustic Feature Embedding Vector Calculation Unit] The acoustic feature embedding vector calculation unit 111 will be described. The acoustic feature embedding vector calculation unit 111 calculates an acoustic feature sequence X, a model parameter θ ｈis input, and an acoustic feature embedding vector sequence h is output according to the model in the acoustic feature embedding vector calculation unit 111.

[0085] The acoustic feature embedding vector calculation unit 111 calculates the acoustic feature sequence X using the model parameters θ ｈ The acoustic feature vectors are converted into an embedding vector sequence h by a conversion function based on

[0086] The embedding vector represents a vector in which information contained in acoustic features is embedded, and includes semantic information contained in the acoustic features in each time series, and is necessary for converting the acoustic features into estimated text. The larger the number of dimensions of the vector, the more information can be embedded. For example, the number of dimensions is manually determined as a 512-dimensional vector.

[0087] Any conversion function can be used as long as it converts a variable-length string into a variable-length vector. For example, a function that constructs a frequency vector of tokens in the text, a recurrent neural network, or a bidirectional recurrent neural network can be used. A method using a self-attention mechanism can also be applied (Reference 3). Reference 3: A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin. "Attention Is All You Need", In Proc. NIPS, pp. 5998-6008, 2017.

[0088] [Output Token Embedding Vector Calculation Unit] The output token embedding vector calculation unit 112 will be described. The output token embedding vector calculation unit 112 calculates a special token p, the (n-1)th word sequence ^o of the estimated text specified by the special token p, and １ , ..., ^o ｎ－１ , model parameters θ ｕ is used as input, and the token embedding vector u of the special token and the output text is ｎ Output.

[0089] The output token embedding vector calculation unit 112 calculates the (n-1)th word sequence of the output text specified by the special token p using the model parameters θ ｕ The token embedding vector u of the estimated text is calculated by a transformation function based on ｎ Convert to.

[0090] The embedding vector represents a vector in which information of each word is embedded. The embedding vector contains semantic information of the word in the text, and is necessary for converting acoustic features into output text. The details of the embedding vector are the same as those described in the acoustic feature embedding vector calculation unit 111, and therefore will be omitted here.

[0091] [Context Vector Calculation Unit] The context vector calculation unit 113 will be described. The context vector calculation unit 113 calculates the token embedding vectors u ｎ , an embedding vector sequence h of acoustic features, and model parameters θ ｓ is used as input, and the context vector s ｎ Output.

[0092] The context vector calculation unit 113 calculates the special token and the (n-1)th token embedding vector u ｎ and the embedding vector sequence h of the acoustic features and the model parameters θ ｓ The context vector s is converted into ｎ Convert to.

[0093] This context vector s ｎ contains semantic information necessary to convert the n-th word of the estimated text, taking into account the acoustic features of the input.

[0094] In this case, any conversion function can be used as long as it converts two types of variable length vectors into a single vector.

[0095] For example, a function that expresses a word sequence of the input text and a word sequence of the estimated text as a single context vector by providing a recurrent neural network for each and adding an attention mechanism can be used as the conversion function.

[0096] In the simplest case, a conversion function can be used that constructs a vector obtained by averaging the frequency vectors of the special token and the first n tokens in the output text, and the frequency vectors of the acoustic feature sequence.

[0097] Furthermore, as described in Reference 1, a self-attention mechanism can be introduced into each of the acoustic feature sequence, the special token, and the token sequence of the estimated text, and a function that expresses them as a single context vector by adding the attention mechanism can be used as the transformation function.

[0098] [Posterior Probability Calculation Unit] The posterior probability calculation unit 121 will be described. The posterior probability calculation unit 121 calculates the context vector s ｎ , model parameters θ ｐ is input, and the posterior probability distribution P(o ｎ |^o １ , ..., ^o ｎ－１ , X, p; Θ).

[0099] The posterior probability calculation unit 121 calculates the context vector s ｎ The model parameters θ ｐ The posterior probability distribution P(o) for the n-th word of the output text controlled by the special talk is calculated by the transformation function based on ｎ |^o １ , ..., ^o ｎ－１ , X, p; Θ).

[0100] The posterior probability can be expressed as a vector with each word as an element, and the posterior probability distribution can be expressed by a vector conversion function.

[0101] In this case, the conversion function may be any function that converts a vector into a posterior probability distribution. For example, the conversion function may be realized by a function that performs conversion using a softmax function. Alternatively, any other function may be applied that can convert the sum of the elements of the estimated vector corresponding to the posterior probability distribution into 1.0.

[0102] [Text Generation Unit] The text generation unit 122 will be described. The text generation unit 122 generates a posterior probability distribution P(o ｎ |^o １ , ..., o ｎ－１ , X, p; Θ) as input, and outputs a token sequence Y of the estimated text of the pattern defined by the special token.

[0103] The text generator 122 calculates the posterior probability distribution P(o １ |X,p;Θ),...,P(o ｎ |^o １ , ..., ^o ｎ－１ , X, p;Θ) so that the product of the posterior probability distribution P(o ｎ |^o １ , ..., ^o ｎ－１ , X, p; Θ), a word sequence O of the estimated text of the pattern defined by the special token is output.

[0104] In this case, the posterior probability distribution P(o ｎ |^o １ , ..., ^o ｎ－１ , X, p; Θ), any method that maximizes the probability can be used for element selection. For example, element selection can be achieved using an approximation algorithm such as a greedy algorithm.

[0105] As shown in Fig. 4, these processes are repeated recursively until all words in the estimated text are generated. However, during model training of the speech recognition device 10 (Fig. 5), the process of the text generation unit 122 is omitted, and the output of the posterior probability calculation unit 121 is output as the estimated text (token sequence).

[0106] Possible tasks executed by the speech recognition device 10, which take speech (acoustic features extracted from speech) as input and output multiple types of text, include speech translation, speech summarization, response generation, and conversion to language other than spoken or written. Note that, in the following description, an example is given in which two types of text, a first text (specifically, spoken text) and a second text (written text), are output, but two or more texts may be output.

[0107] A speech translation task is a task that outputs a speech recognition result (spoken text) in a language corresponding to input speech and a translation result into a predetermined language. The speech translation task outputs a translation result of the spoken language into a predetermined language different from the input speech, and a translation result into written text in that language.

[0108] The speech summarization task is a task to output the speech recognition result (spoken words) of the input speech and the summary result (written words).

[0109] The response generation task is a task for outputting the speech recognition result (spoken language) of the input speech and a response sentence (written language) in response to the speech recognition result. The response sentence may be in the same language or in a different language.

[0110] Conversion tasks other than spoken and written text include converting from a polite tone to a casual tone, or converting to a specific character.

[0111] One of the two output texts may be the speech recognition result (spoken language) and the other may be for a different task, or it may be something else. In the case of spoken language-to-written language conversion, the two output texts are in the same language, but they may be in different languages.

[0112] [Evaluation Results] Next, the results of evaluation of the speech recognition device 10 according to the embodiment will be described.

[0113] First, we will explain the datasets used in the evaluation. This evaluation used the Corpus of Spontaneous Japanese (CSJ) for the Japanese ASR (Automatic Speech Recognition) task and the S2WT (Speech-to-Written Text) task. The CSJ is divided into training, validation, and testing sections. The CSJ contains pairs of speech-level audio signals and spoken text. Therefore, written text was created using parts of the spoken text, and a set of pairs of spoken text and written text was used for training, validation, and testing of the speech recognition device 10.

[0114] As an evaluation index, the Word Error Rate (WER) was calculated for the ASR task, and the 4-gram BLEU (hereafter referred to as BLEU) was calculated for the S2WT task (Reference 4). Reference 4: K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, "BLEU: a Method for Automatic Evaluation of Machine Translation," in Proc. Annual Meeting on Association for Computational Linguistics (ACL), 2002, pp. 311-318.

[0115] For comparison, we compared our method with conventional cascading models, individual models, and interactive learning models (Reference 5). Reference 5: Y. Liu, J. Zhang, H. Xiong, L. Zhou, Z. He, H. Wu, H. Wang, and C. Zong, “Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding,” in Proc. AAAI Conference on Artificial Intelligence (AAAI), 2020, pp. 8417-8424.

[0116] The cascade model includes a first model (Reference 6) that estimates spoken text from acoustic features, and a second model (Patent Document 1) that estimates written text based on the spoken text estimated by the first model. Reference 6: Linhao Dong, Shuang Xu, Bo Xu, “SPEECH-TRANSFORMER: A NO-RECURRENCE SEQUENCE-TO-SEQUENCE MODEL FOR SPEECH RECOGNITION,” IEEE, 2018.

[0117] The individual models include a first model (Reference 6) that estimates spoken text from acoustic features, and a second model (Reference 6; the training data is a pair of speech and written text) that estimates written text from acoustic features.

[0118] The interactive learning model, like the speech recognition device 10P illustrated in Figure 7, has a feature conversion unit 11P (encoder), a first output generation unit 12P-1 (decoder 1) (first model) that generates spoken text and written text, and a second output generation unit 12P-2 (decoder 2) (second model) that generates spoken text and written text, respectively.

[0119] Table 1 shows an example of output from a conventional individual model and an example of output from the speech recognition device 10 (proposed method).

[0120]

[0121] Table 2 shows the evaluation results of WER and BLUE for each model, as well as the parameter size of each model. The scores in Table 2 are the average of each score calculated using two test data sets.

[0122]

[0123] As shown in Table 2, the speech recognition device 10 achieved better results than the conventional device in both the WER and BLUE evaluation results.

[0124] As described above, conventional speech recognition devices using the cascade model, the individual model, and the interactive learning model have two models.

[0125] The cascade model, the individual model, and the dialogue-based learning model use two models, the first model and the second model, and therefore have a larger number of parameters than the speech recognition device 10, which uses only one model (Table 2). Specifically, the cascade model and the individual model have twice the number of parameters compared to the speech recognition device 10, which uses only one model. Furthermore, the dialogue-based learning model also requires an attention layer to be added, and therefore has a larger number of parameters than the speech recognition device 10.

[0126] Therefore, the cascade model, the individual model, and the dialogue-based learning model have a larger number of model parameters and require higher calculation costs than the speech recognition device 10. In contrast, the speech recognition device 10 can reduce the number of model parameters and the calculation costs compared to the cascade model, the individual model, and the dialogue-based learning model.

[0127] Furthermore, as shown in Table 1, in the individual models, only speech tokens that can be output by ASR were omitted, resulting in a coverage error. In contrast, the speech recognition device 10 uses a single shared model to estimate written text using both speech features and speech text, and it is clear from Table 1 that the error that occurred between the two models is improved.

[0128] Therefore, it is inferred that it is effective to generate spoken text and written text from speech using the speech recognition device 10. In fact, as shown in Table 2, it has become clear that the speech recognition device 10 is more effective than conventional devices.

[0129] [Effects of the embodiment] A conventional speech recognition device (for example, the speech recognition device 10P illustrated in FIG. 7) has a first model that outputs a spoken text and a second model that outputs a written text. That is, the conventional speech recognition device generates the spoken text and the written text separately using different models and outputs them in parallel.

[0130] Therefore, in the conventional speech recognition device, the entire spoken text cannot be taken into consideration when generating the written text. In other words, in the conventional speech recognition device, the effects of learning the spoken text and the written text cannot be shared between the first model that estimates the spoken text and the second model that estimates the written text, and there is a limit to the improvement of accuracy.

[0131] In contrast, the speech recognition device 10 according to the embodiment outputs a single shared model for spoken text and written text, and can therefore take the spoken text and the written text into consideration when generating the written text and the spoken text, respectively, and can share the effects of learning the spoken text and the written text.

[0132] In particular, when the speech recognition device 10 is applied to speech translation, by using the spoken text and the written text in the same language, it is thought that the effect of learning the spoken text will contribute more to the estimation of the written text.

[0133] Furthermore, conventional speech recognition devices use different models to estimate spoken text and written text, which results in a large number of model parameters, as shown in Table 2, and therefore has the problem of increasing calculation costs.

[0134] In contrast, the speech recognition device 10 estimates spoken text and written text using a common model, and therefore, as shown in Table 2, can reduce the number of model parameters compared to conventional speech recognition devices, thereby reducing calculation costs.

[0135] A conventional speech recognition device 10P (see FIG. 7) is, for example, a device that performs speech translation. That is, the conventional speech recognition device 10P outputs a speech recognition result in a language corresponding to an input speech and a translation result of the written language into a predetermined language.

[0136] In contrast, the speech recognition device 10 can also make the spoken text and the written text the same language. Therefore, by making the spoken text and the written text the same language, the speech recognition device 10 can further share the learning effects of the spoken text and the written text, thereby improving accuracy.

[0137] Furthermore, by setting a special token, the speech recognition device 10 can switch between three output patterns when generating estimated text: spoken text, written text, and spoken and written text.

[0138] Therefore, the speech recognition device 10 can switch the output pattern of estimated text depending on the usage scene of the service. For example, the speech recognition device 10 can output both when it is desired to obtain written text while checking whether speech recognition has been performed properly, and can output only the written text when only the written text is required. Therefore, by using the speech recognition device 10, flexible services can be provided according to the user's requests.

[0139] In this way, the speech recognition device 10 estimates spoken text and written text using a shared model, and therefore can accurately output both spoken text and written text from input speech while having a simple configuration and reducing computational costs.

[0140] The first text and the second text can also be defined as texts with different text domains. In other words, the first text and the second text are texts with the same semantic content but different domains. Furthermore, text corresponding to speech is, for example, "text related to the semantic content contained in speech" or "text expressing the semantic content contained in speech as a character string." However, this is not limited to this.

[0141] [System Configuration of the Embodiment] Each component of the speech recognition device 10 is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of the functions of the speech recognition device 10 is not limited to that shown in the figure, and all or part of the functions can be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc.

[0142] Furthermore, all or any part of the processes performed by the speech recognition device 10 may be realized by a CPU, a GPU (Graphics Processing Unit), and a program analyzed and executed by the CPU and the GPU. Furthermore, each process performed by the speech recognition device 10 may be realized as hardware using wired logic.

[0143] Furthermore, among the processes described in the embodiments, all or part of the processes described as being performed automatically can be performed manually. Alternatively, all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters described above and illustrated can be changed as appropriate unless otherwise specified.

[0144] 8 is a diagram showing an example of a computer in which the speech recognition device 10 is realized by executing a program. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0145] The memory 1010 includes a ROM 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0146] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, an application program 1092, a program module 1093, and program data 1094. That is, the program that defines each process of the speech recognition device 10 is implemented as a program module 1093 in which code executable by the computer 1000 is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, the program module 1093 for executing processes similar to those of the functional configuration of the speech recognition device 10 is stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced with an SSD (Solid State Drive).

[0147] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in memory 1010 or hard disk drive 1090. Then, CPU 1020 reads out program module 1093 or program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as necessary and executes them.

[0148] The program module 1093 and program data 1094 may not necessarily be stored in the hard disk drive 1090, but may also be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or a wide area network (WAN)). The program module 1093 and program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070.

[0149] The following additional notes are provided regarding the above-described embodiments.

[0150] (Supplementary Item 1) A speech recognition device including: a memory; and at least one processor connected to the memory, wherein the processor inputs speech into one pre-trained machine learning model and outputs multiple types of text corresponding to the speech.

[0151] (Supplementary Item 2) The speech recognition device according to Supplementary Item 1, wherein outputting the plurality of types of text includes outputting a first text and a second text different from the first text concatenated together.

[0152] (Supplementary Item 3) The speech recognition device according to Supplementary Item 2, wherein the first text and the second text are in the same language.

[0153] (Supplementary Item 4) The speech recognition device according to Supplementary Item 2, wherein outputting the plurality of types of text includes, upon receiving input of a special token that defines the output as either the first text only, the second text only, or a text in which the first text and the second text are concatenated in series, switching to an output of a pattern defined by the special token and outputting the text.

[0154] (Supplementary Item 5) The speech recognition device according to Supplementary Item 2, wherein supervised learning of the machine learning model is performed using a speech and paired data of the first text and the second text, which are correct texts corresponding to the speech, as learning data.

[0155] (Supplementary Item 7) A non-transitory storage medium storing a program executable by a computer to execute a speech recognition process, the speech recognition process inputting a speech into a single machine learning model that has been trained in advance, and generating multiple types of text corresponding to the speech as output.

[0156] Although the present invention has been described above as an embodiment, the present invention is not limited to the descriptions and drawings that form part of the disclosure of the present invention. In other words, other embodiments, examples, and operational techniques that can be made by those skilled in the art based on the present invention are all included in the scope of the present invention.

[0157] REFERENCE SIGNS LIST 10 Speech recognition device 11 Feature conversion unit 12 Output generation unit 20 Model parameter learning unit 111 Acoustic feature embedding vector calculation unit 112 Output token embedding vector calculation unit 113 Context vector calculation unit 121 Posterior probability calculation unit 122 Text generation unit

Claims

1. A speech recognition device comprising: a generation unit that inputs speech into a single pre-trained machine learning model and outputs multiple types of text corresponding to the speech.

2. The speech recognition device according to claim 1, characterized in that the generation unit outputs, as the multiple types of text, a first text and a second text different from the first text by concatenating them.

3. The speech recognition device according to claim 2, wherein the first text and the second text are in the same language.

4. The speech recognition device of claim 2, characterized in that when the generation unit receives input of a special token that specifies the output of the generation unit to be either the first text only, the second text only, or the first text and the second text concatenated in series, it switches to an output pattern specified by the special token and outputs the text.

5. The speech recognition device according to claim 2, further comprising a learning unit that performs supervised learning of the machine learning model using a speech and paired data of the first text and the second text, which are correct answer texts corresponding to the speech, as learning data.

6. A speech recognition method executed by a speech recognition device, comprising the steps of: inputting speech into one pre-trained machine learning model; and outputting multiple types of text corresponding to the speech.

7. A speech recognition program for causing a computer to execute the steps of: inputting speech into a single pre-trained machine learning model; and outputting multiple types of text corresponding to the speech.

Images