Voice recognition device and program

Abstract

To provide a voice recognition device capable of generating information on an intermediate layer of a transducer voice recognition model based on only text (namely, without depending on voice).SOLUTION: A first encoder calculates an intermediate layer feature amount based on an acoustic feature amount. A second encoder calculates a final layer feature amount based on the intermediate layer feature amount. A first loss calculation part calculates a first loss based on the final layer feature amount and a correct answer text. A second loss calculation part calculates a second loss based on the intermediate layer feature amount and the correct answer text. Learning of the first and second encoders can be performed based on the first loss and the second loss. A text encoder calculates a text feature amount based on the text. A third loss calculation part calculates a third loss based on the intermediate layer feature amount and the text feature amount. Learning of the text encoder can be performed based on the third loss.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a voice recognition device and a program. [Background technology]

[0002] A model called Transducer exists as a speech recognition model suitable for streaming speech recognition. Non-Patent Document 1 describes the Transducer technology. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Kanishka Rao, Hasim Sak, Rohit Prabhavalkar, “Exploring architectures, data and units for streaming end-to-end speech recognition with RNN-transducer”, in Proc. IEEE ASRU, 2017, pp. 193-199. Summary of the Invention [Problem to be solved by the invention]

[0004] The recognition accuracy of a speech recognition model also depends on the training data used when training the model. When a speech recognition model is trained using training data belonging to a certain domain, it is thought that there will be a limit to the accuracy when trying to recognize speech in another domain. Therefore, if a speech recognition model can be retrained (domain adaptation) using information from another domain (called a target domain), it is expected that the speech recognition accuracy in the target domain can be improved.

[0005] Retraining a speech recognition model using information on a target domain can be performed based on text belonging to the target domain. One possible form for this is to generate intermediate layer information (feature values, vectors output from an intermediate layer encoder, etc.) of the speech recognition model based on text belonging to the target domain.

[0006] The present invention has been made based on the above-mentioned problem recognition, and aims to provide a speech recognition device and a program therefor that can generate intermediate layer information (intermediate layer output from an encoder) of a transducer speech recognition model based only on text (i.e., without relying on speech) in order to enable learning using only text. [Means for solving the problem]

[0007] [1] In order to achieve the above object, a speech recognition device according to one aspect of the present invention includes a first encoder that calculates and outputs intermediate layer features based on input acoustic features, a second encoder that calculates and outputs final layer features based on the intermediate layer features output from the first encoder, a first loss calculation unit (55) that calculates a first loss based on the final layer features output from the second encoder and a correct answer text corresponding to the input acoustic features, a second loss calculation unit (62) that calculates a second loss based on the intermediate layer features output from the first encoder and a correct answer text corresponding to the input acoustic features, and and a third loss calculation unit (41) that calculates a third loss based on the intermediate layer features output by the first encoder and the text features output by the text encoder, wherein a parameter of a network of the second encoder and a parameter of a network of the first encoder are adjustable based on the first loss, a parameter of the network of the first encoder are adjustable based on the second loss, and a parameter of the network of the text encoder are adjustable based on the third loss.

[0008] [2] According to another aspect of the present invention, in the speech recognition device according to the above [1], based on an input text character string, a character string including a symbol n is generated. c times(n c ≧1) A repeat operation is performed so that the symbols in the text string are consecutively repeated, and n is added between the symbols in the text string. b times(n band a pseudo alignment generation unit that generates a pseudo symbol string by performing an operation of inserting consecutive blank symbols (≧0) into the pseudo symbol string, and when re-learning the trained second encoder based on the text string, the text encoder uses the pseudo symbol string generated by the pseudo alignment generation unit based on the text string as the text, and calculates and outputs text features based on the text, and the second encoder calculates and outputs the final layer features based on the text features output from the text encoder, in place of the intermediate layer features output from the first encoder.

[0009] [3] In addition, in one aspect of the present invention, in the speech recognition device according to [2] above, the pseudo alignment generation unit generates a pseudo alignment signal when the number of consecutive symbols is n c Probability density function P determined from given statistics for c (n) based on the above n c The value of is determined, and the number of consecutive blank symbols is n b Probability density function P determined from given statistics for b (n) based on the above n b This determines the value of

[0010] [4] Also, one aspect of the present invention includes a first encoder that calculates and outputs intermediate layer features based on input acoustic features, a second encoder that calculates and outputs final layer features based on the intermediate layer features output from the first encoder, a first loss calculation unit (55) that calculates a first loss based on the final layer features output from the second encoder and a correct answer text corresponding to the input acoustic features, a second loss calculation unit (62) that calculates a second loss based on the intermediate layer features output from the first encoder and a correct answer text corresponding to the input acoustic features, and a text feature calculation unit (63) that calculates and outputs text features based on an input text. a third loss calculation unit (41) that calculates a third loss based on the intermediate layer features output by the first encoder and the text features output by the text encoder, wherein a parameter of a network of the second encoder and a parameter of a network of the first encoder are adjustable based on the first loss, the parameter of the network of the first encoder is adjustable based on the second loss, and the parameter of the network of the text encoder is adjustable based on the third loss. Effect of the Invention

[0011] According to the present invention, the first encoder can be trained to output information about the correct text. Also, training can be performed so that the text features output by the text encoder approach the intermediate layer features output by the first encoder. This means that the second encoder can be trained based only on the text. [Brief description of the drawings]

[0012] [Figure 1] 1 is a block diagram showing a schematic functional configuration of a voice recognition device according to an embodiment of the present invention. [Diagram 2]FIG. 2 is a block diagram showing an outline of the configuration of a transducer voice recognition model corresponding to the “base technology 2” of the embodiment. [Diagram 3] FIG. 13 is a schematic diagram showing an example of an increment pattern of index values ​​(t, u) in a transducer voice recognition model corresponding to the “base technology 2” of the embodiment. [Figure 4] FIG. 2 is a schematic diagram for explaining a first stage of learning of a voice recognition model (pre-learning of an Intermediate CTC voice recognition model) corresponding to the "Related Art 1" of the embodiment. [Diagram 5] FIG. 13 is a schematic diagram for explaining second-stage learning (adapter learning) of the voice recognition model corresponding to the "Related Art 1" of the embodiment. [Figure 6] FIG. 13 is a schematic diagram showing a first domain adaptation method for third-stage training (re-training after Intermediate CTC) of a speech recognition model corresponding to “Related Art 1” of the embodiment. [Figure 7] FIG. 13 is a schematic diagram showing a second domain adaptation method for third-stage training (re-training after Intermediate CTC) of a speech recognition model corresponding to “Related Art 1” of the embodiment. [Figure 8] 13 is a flowchart showing a processing procedure for generating a pseudo-CTC symbol string based on a text character string in the "Related Art 1" of the embodiment. [Figure 9] FIG. 13 is a schematic diagram showing a configuration for adapting a transducer voice recognition model using text data in “Related Art 2” of the embodiment. [Figure 10] 1 is a schematic diagram showing the configuration of a transducer speech recognition model in which a CTC loss calculation means is provided in an intermediate layer (first step learning) according to this embodiment. FIG. [Figure 11] FIG. 11 is a schematic diagram for explaining the second step of learning (learning of a text encoder) in this embodiment. [Figure 12]FIG. 11 is a schematic diagram for explaining the third step of learning (learning of a transducer speech recognition model using only text) in this embodiment. [Figure 13] 1 is a block diagram showing an example of an internal configuration for realizing a voice recognition device 10 of the present embodiment, and each device (model) in the base techniques 1 and 2, or the related techniques 1 and 2. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Next, an embodiment of the present invention will be described with reference to the drawings. This embodiment improves a model called a transducer (see Non-Patent Document 1) suitable for streaming speech recognition. In this embodiment, the transducer is provided with a means for improving the speech recognition accuracy of utterances related to the topic of the content of the text by training the speech recognition model only with the text.

[0014] In this embodiment, a speech recognition model adaptation method using text data ("Related Technology 1" described later) is applied to the transducer speech recognition model. Specifically, a means for calculating a CTC loss function is provided in the intermediate layer of the encoder of the transducer speech recognition model, and the transducer speech recognition model is pre-trained based on the value of this CTC loss function, thereby solving the problem that the correspondence between the output from the text encoder and the output from the intermediate layer of the encoder is unclear.

[0015] In addition, in the above-mentioned Related Art 1 (a method for adapting a speech recognition model using text data), pseudo blank symbols are inserted into a text string, and a neural network is trained using the symbol string containing the blank symbols as an input. In the method of Related Art 1, multiple CTC loss function calculation means are set in the encoder, and the network structure is complicated. To eliminate this complexity, this embodiment introduces a new neural network training method in which the intermediate layer CTC losses of the encoder are combined into one, and a symbol string containing pseudo blank symbols is input. In this embodiment, the neural network corresponding to the function called Adapter in Related Art 1 is called a Text Encoder.

[0016] The reference [Hiroaki Sato, Tomoyasu Komori, Takeshi Mishima, Yoshihiko Kawai1, Takahiro Mochizuki, Shoei Sato1, Tetsuji Ogawa, Text-Only Domain Adaptation Based on Intermediate CTC, Interspeech 2022, 18-22 September 2022, Incheon, Korea] describes the above "Related Technology 1". In other words, a method for end-to-end speech recognition model adaptation using text data is proposed by adding a neural network adapter that inputs text to a speech recognition model trained using Connectionist Temporal Classification (CTC) loss. In this proposed method, CTC loss is also provided in the intermediate layer of the Encoder for training, and alignment text of speech and text is output from the intermediate layer. Then, the alignment text is input to a neural network called the Adapter, and the Adapter is trained so that the obtained output vector matches the output vector of the Encoder. By outputting alignment text from the intermediate layer of the Encoder using CTC, it is possible to obtain correspondence between the Encoder output and the Adapter output, and by making the output of the intermediate layer of the Encoder contain more linguistic information, it is possible to effectively train the Adapter. In this embodiment, the transducer is improved by applying the concept of the above method.

[0017] 1 is a block diagram showing a schematic functional configuration of a speech recognition device according to this embodiment. As shown in the figure, the speech recognition device 10 includes a speech encoder 21, a pseudo alignment generator 31, a text encoder 32, a loss calculator 41, a joint encoder 51, a prediction network 52, a combination network 53, a softmax function processor 54, a loss calculator 55, a softmax function processor 61, a loss calculator 62, and a controller 70. The functions of each unit will be described below.

[0018] Each of these functional units can be realized, for example, by using a computer and a program. Each functional unit has a storage means as necessary. The storage means is, for example, a variable in the program, or a memory allocated by the execution of the program. A non-volatile storage means such as a magnetic hard disk drive or a solid state drive (SSD) may also be used as necessary. At least a part of the functions of each functional unit may also be realized as a dedicated electronic circuit rather than a program. The functions of each unit are as follows.

[0019] The speech encoder 21 calculates and outputs intermediate layer features (vectors) based on the input acoustic features. The output from the speech encoder 21 may be called an "intermediate layer feature." The speech encoder 21 is realized using a neural network, and the values ​​of its internal parameters are configured to be learnable.

[0020] The pseudo alignment generating unit 31 generates a symbol string having a pseudo alignment based on the input text by inserting blank symbols as appropriate or repeating symbols a predetermined number of times. c times(n c ≧1) A repeat operation is performed so that the symbols in the text string are consecutively repeated, and n is added between the symbols in the text string. b times(n bThe pseudo alignment generating unit 31 may determine the number of times (which may be called the number of consecutive times or the number of consecutive times) to repeat a symbol (a symbol such as an ordinary character, or a blank symbol) as follows, for example. That is, the pseudo alignment generating unit 31 may determine the number of times (which may be called the number of consecutive times n c Probability density function P determined from given statistics for c (n) based on the above n c The value of is determined, and the number of consecutive blank symbols is n b Probability density function P determined from given statistics for b (n) based on the above n b The details of the process performed by the pseudo alignment generating unit 31 will be described later.

[0021] The text encoder 32 calculates and outputs features (vectors) for the input text based on the input text. The output from the text encoder 32 may be called text features. The text encoder 32 is realized using a neural network, and is configured so that the values ​​of its internal parameters can be learned. As will be described later, the text input to the text encoder 32 is a CTC symbol string passed from the softmax function processor 61 side, a transducer symbol string passed from the softmax function processor 54 side, or a symbol string having a pseudo alignment generated by the pseudo alignment generator 31.

[0022] The loss calculation unit 41 calculates the loss between the feature amount output from the speech encoder 21 and the feature amount output from the text encoder 32. The loss calculated by the loss calculation unit 41 is used for training the neural network. The loss calculation unit 41 is also called a "third loss calculation unit."

[0023] The shared encoder 51 calculates and outputs another feature quantity (vector) based on the feature quantity (vector) output from the speech encoder 21 and the feature quantity (vector) output from the text encoder 32. The shared encoder 51 passes the output feature quantity to the combination network 53. The shared encoder 51 is realized using a neural network, and is configured so that the values ​​of its internal parameters can be learned.

[0024] The prediction network 52 receives text (correct text) and calculates and outputs features (vectors) based on the text. The prediction network 52 passes the output features to the combination network 53. The prediction network 52 is realized using a neural network, and is configured so that the values ​​of its internal parameters can be learned.

[0025] The combination network 53 integrates the feature quantity (vector) output from the shared encoder 51 and the feature quantity (vector) output from the prediction network 52 into one vector by linearly transforming them and then adding them. The combination network 53 may further apply an activation function to the vector or perform linear transformation. The output from the combination network 53 may be called a final layer feature quantity. The number of dimensions of the vector output from the combination network 53 matches the number of types of output symbols. The combination network 53 is realized using a neural network, and is configured so that the values ​​of its internal parameters are learnable.

[0026] The softmax function processor 54 performs processing to apply a softmax function to elements of each dimension of the vector output from the connection network 53. The value of each dimension output by the softmax function processor 54 can be regarded as a probability value of the corresponding symbol.

[0027] The loss calculation unit 55 calculates a loss based on the output from the softmax function processing unit 54 (the output inference value as a transducer) and the correct answer text. The loss calculated by the loss calculation unit 55 is used for learning the neural network. The loss calculation unit 55 is also called a "first loss calculation unit."

[0028] The softmax function processing unit 61 performs processing to apply a softmax function to the feature amount (vector) output from the speech encoder 21. By regarding the output from the softmax function processing unit 61 as a probability value, a symbol that is an inference result is output.

[0029] The loss calculation unit 62 calculates a loss based on the output (CTC symbol) from the softmax function processing unit 61 and the correct text. The loss calculated by the loss calculation unit 62 is used for training the neural network. The loss calculation unit 62 is also called a "second loss calculation unit."

[0030] The control unit 70 controls the entire voice recognition device 10. As a result, the control unit 70 controls the operation of each of the above-mentioned components constituting the voice recognition device 10. The voice recognition device 10 operates in two modes: a learning mode in which the device learns an internal model, and an inference mode in which the device performs recognition processing on unknown voices. The control unit 70 controls each of the components according to the operating mode. In the learning mode, the voice recognition device 10 performs learning in three stages, from a first step to a third step, which will be described later. Therefore, when the voice recognition device 10 operates in the learning mode, the control unit 70 controls each of the components according to which step of learning is being performed. Among the components of the speech recognition device 10, the speech encoder 21 may be called the "first encoder". In this case, the shared encoder 51 and the combined network 53 may be collectively called the "second encoder". Furthermore, the "second encoder" may include the prediction network 52. When the first encoder and the second encoder are distinguished in this manner, the feature of the intermediate layer output from the first encoder is passed to the second encoder and also to the loss calculation unit 62. The loss calculation unit 62 calculates the loss between the feature of the intermediate layer and the correct text. Since the first encoder is trained based on the loss calculated by the loss calculation unit 62, it is expected that the output from the first encoder contains information about the correct text. In addition, the second encoder is driven by the feature output from the text encoder depending on the situation (the third step of training described later).

[0031] That is, in the configuration of the speech recognition device 10 of this embodiment, the first encoder calculates and outputs intermediate layer features based on the input acoustic features. The second encoder calculates and outputs final layer features based on the intermediate layer features output from the first encoder. The first loss calculation unit (loss calculation unit 55) calculates a first loss based on the final layer features output from the second encoder and a correct answer text corresponding to the input acoustic features. The second loss calculation unit (loss calculation unit 62) calculates a second loss based on the intermediate layer features output from the first encoder and a correct answer text corresponding to the input acoustic features. The text encoder 32 calculates and outputs a text feature based on an input text (which may be a CTC symbol string, a transducer symbol string, or a pseudo symbol string thereof). The third loss calculation unit (loss calculation unit 41) calculates a third loss based on the intermediate layer features output from the first encoder and the text features output from the text encoder.

[0032] The speech recognition device 10 is capable of adjusting the parameters of the network of the second encoder and the parameters of the network of the first encoder based on the first loss. The speech recognition device 10 is also capable of adjusting the parameters of the network of the first encoder based on the second loss. The speech recognition device 10 is also capable of adjusting the parameters of the network of the text encoder 32 based on the third loss.

[0033] When re-learning the trained second encoder based on a text string (e.g., a text string in a target domain), the text encoder 32 calculates and outputs text features based on a pseudo symbol string generated by the pseudo alignment generating unit 31 based on the text string as the text. In this case, the second encoder calculates and outputs the final layer features based on the text features output from the text encoder, instead of the intermediate layer features output from the first encoder.

[0034] Next, the technologies on which this embodiment is based and the technologies related to this embodiment will be described in order. Premise technology 1 is a CTC speech recognition model. Premise technology 2 is a speech recognition model. Related technology 1 is a speech recognition model adaptation method using text data. Related technology 2 is a transducer speech recognition model adaptation method (Maestro) using text data.

[0035] [Base technology 1: Connectionist Temporal Classification (CTC) speech recognition model] The CTC speech recognition model is one of the end-to-end speech recognition model methods that directly learns the relationship between speech and character strings without using phonetic symbols. The CTC speech recognition model can be realized using a neural network. The CTC speech recognition model outputs one symbol for each frame of acoustic features. However, in CTC speech recognition, blank symbols are not included in the output symbol strings of characters, words, etc.<blank>By inserting a blank symbol, and allowing the same symbol to be output consecutively, it is possible to make the acoustic feature frame correspond to the output symbol string. <blank>By inserting and allowing the output symbols to be consecutive, the learning is performed.

[0036] As an example, the type of output symbol is a character, and the alignment of the symbol string "weather" corresponding to the acoustic features of four frames is as follows: <blank>Represents. 01: _ _ Weather 02: Weather 03: Weather 04: Weather 05: Weather 06: Weather _ _ 07: _ Ten Tenki 08: Weather 09: Heavenly Heavenly Spirit 10: Good weather 11: _ Weather 12: Weather 13: Weather 14: Weather 15: Weather 16: Weather 17: Heaven, heaven, heaven

[0037] When making inference using a trained CTC speech recognition model, consecutive identical symbols in the output symbol string are compressed into one (for example, "tententen" is compressed into "ten"), and blank symbols are inserted. <blank>However, if the correct answer contains consecutive identical symbols (for example, the correct answer contains consecutive "i" characters, such as "ii tenki"), a blank symbol must be inserted between the consecutive symbols. <blank>For example, if the correct answer contains a string "ii", a blank symbol must be inserted between the "ii" and "ii". <blank>must be inserted.

[0038] The training of the CTC speech recognition model is performed as follows: That is, the acoustic feature vector sequence input to the CTC speech recognition model is x (in bold) shown in formula (1).

[0039]

number

[0040] In formula (1), t is an index that indicates the order of the sequence. For example, a log-mel spectrogram can be used as the acoustic feature, but is not limited to this. When this x (bold) is input to the CTC speech recognition model, the output vector sequence output from the final layer of the CTC speech recognition model is h (bold) shown in formula (2).

[0041]

number

[0042] The alignment a (bold type) is the alignment corresponding to the output string h (bold type). The alignment a (bold type) is expressed by the following formula (3).

[0043]

number

[0044] In formula (3), V is the vocabulary set (the set of possible output symbols), <blank>} is a set whose elements are only blank symbols. The probability of alignment a (bold) given the output string h (bold) is expressed by equation (4).

[0045]

number

[0046] The correct symbol string y (in bold) is expressed by the following formula (5). In formula (5), u is an index that indicates the order of the symbols. u ∈V,

[0047]

number

[0048] Given the output vector sequence h (bold), the sum of the probabilities for all possible alignments of the correct symbol string y (bold) is expressed by the following equation (6).

[0049]

number

[0050] However, in formula (6), β -1 (y(bold)) is the set of all possible alignments for the correct answer symbol y(bold). The CTC loss function takes the logarithm of the sum of the above probabilities. That is, the loss L is expressed by the following equation (7).

[0051]

number

[0052] To train the CTC speech recognition model, the parameter values ​​in the neural network that constitutes the model are adjusted (optimized) using training data so as to minimize the loss value L.

[0053] [Base technology 2: Transducer speech recognition model] 2 is a block diagram showing an outline of the configuration of a transducer speech recognition model, which is a prerequisite technology. As shown in the figure, the transducer speech recognition model 110 includes an encoder 121, a prediction network 152, a combination network 153, and a softmax function processing unit 154. The transducer speech recognition model 110 may also be realized, for example, by a computer and a program. At least a part of the transducer speech recognition model 110 may be realized by a dedicated electronic circuit.

[0054] In the transducer speech recognition model 10, the output from the combination network 153, z t,u Based on this, the softmax function processor 154 calculates the recognition probability using the softmax function. Then, the symbol having the maximum probability is output as the recognition result by the transducer speech recognition model 110.

[0055] The encoder 121 is also referred to as “Encoder”. The encoder 121 calculates an output vector based on an acoustic feature vector. The encoder 121 receives an acoustic feature vector x t (t=1, 2, . . . , T) is input. The acoustic feature vector may be, for example, a log-mel spectrogram. The encoder 121 converts the acoustic feature vector x t Based on the output vector h t enc Calculate and output (t=1,2,...,T).

[0056] The prediction network 152 is also referred to as a "Prediction Network" or a "Pred. Network." The prediction network 152 inputs the symbol predicted by the transducer speech recognition model 110 and calculates an output vector. Specifically, the prediction network 152 inputs the symbol y predicted by the transducer speech recognition model 110. u-1 A symbol y (letter, word, etc.) is input. The prediction network 152 calculates the symbol y u-1 Based on the output vector h u dec Calculate and output (u=1,2,...,U).

[0057] The joint network 153 is also referred to as a "Joint Network." The joint network 153 is a vector h t enc and the vector h output from the prediction network 152 u dec Then, we linearly transform and add them together to create a single vector, and then apply an activation function and linearly transform them to create the vector z t,u Find and output the vector z t,u is a vector having a dimension that matches the number of types of output symbols.

[0058] The softmax function processor 154 processes the vector z t,u A Softmax function is applied to each element of a to calculate the recognition probability P(a|t,u) of each symbol. Here, a is a symbol that can be output by the transducer speech recognition model 110. The symbol string output by the transducer speech recognition model 110 is a symbol string including blank symbols, similar to the output from the CTC speech recognition model. By including blank symbols in the candidates for output symbols, the difference between the acoustic feature sequence length and the correct symbol string length is filled. The final recognition result is obtained by deleting blank symbols from the symbol string including blank symbols.

[0059] The encoder 121, the prediction network 152, and the joint network 153 may be implemented using neural networks, that is, the parameters of these networks may be adjusted based on training data.

[0060] The difference between the symbol string output from the CTC speech recognition model and the symbol string output from the transducer speech recognition model 110 is as follows: In other words, when symbols other than blank symbols appear consecutively in the symbol string output from the CTC speech recognition model, the consecutive symbols are compressed into one symbol to obtain the final recognition result. On the other hand, when symbols other than blank symbols appear consecutively in the symbol string output from the transducer speech recognition model 110, the consecutive symbols are not compressed into one symbol.

[0061] Next, the recognition operation of the transducer voice recognition model 110 will be described. <sos>is input to the prediction network 152, and the acoustic feature vector x1 is input to the encoder 121. <sos>is a symbol that represents the "start-of-sentence." The transducer speech recognition model 110 calculates the recognition probability P(a|1,1) for all types of symbol a for these inputs (y0 and x1) and outputs the symbol with the highest recognition probability.

[0062] When a blank symbol is output from the transducer speech recognition model 110, t is incremented, and the next input to the transducer speech recognition model 110 is y0= <sos>and the acoustic feature vector x2 is obtained. When a symbol other than a blank symbol is output, u is incremented, y1=a (the symbol with the highest probability, i.e., the recognition result), and the input to the next transducer speech recognition model 110 becomes y1 and x1.

[0063] In the following, the transducer speech recognition model 110 is trained as follows. That is, in training the transducer speech recognition model 110, a symbol string output for a pattern of increments of t and u is taken as a correct symbol string, and training is performed so as to maximize the output probability of the correct symbol string a (bold type). The output probability of the correct symbol string a (bold type) is expressed by the following formula (8).

[0064]

number

[0065] However, the correct symbol string a (bold) is as shown in the following formula (9). In other words, the correct symbol string a (bold) may include blank symbols.

[0066]

number

[0067] 3 is a schematic diagram showing an example of an increment pattern of t and u in the transducer speech recognition model 110. In the illustrated increment pattern, the horizontal arrow line (from left to right) corresponds to an increase in t, and the vertical arrow line (from top to bottom) corresponds to an increase in u. The illustrated increment pattern represents a case where the correct symbol string for the acoustic features of four frames is "weather".

[0068] Among the increment patterns shown in Figure 3, the pattern indicated by the thick arrow is " <blank>,sky, <blank> , <blank>,air, <blank>For example, for this string, the Transducer speech recognition model is trained to maximize the probability that P(a1= <blank>|1,1) P(a2=day|2,1) P(a3= <blank>|2,2) P(a4= <blank>|3,2) P(a5=ki|4,2) P(a6= <blank>|4,3)

[0069] The transducer speech recognition model is trained to maximize the sum of the output probabilities of correct symbol strings in all increment patterns.

[0070] The correct symbol string y (in bold) is expressed by the following equation (10).

[0071]

number

[0072] The sum of the probabilities of all possible increment patterns (i.e., alignments) of the correct symbol string y (bold type) and the acoustic feature vector sequence x (bold type) is expressed by the following equation (11).

[0073]

number

[0074] However, β in Equation (11) -1 (y(bold)) represents the set of all possible alignments between the correct symbol string y(bold) and the acoustic feature vector sequence x(bold). The loss function for training the transducer speech recognition model takes a logarithm and is expressed by the following equation (12).

[0075]

number

[0076] In other words, the transducer speech recognition model is trained by adjusting (updating) the parameters of the neural network to minimize the loss L mentioned above.

[0077] Next, related arts 1 and 2 of this embodiment will be described.

[0078] [Related technology 1: Speech recognition model adaptation method using text data] Related technique 1 relates to a method of learning a model using only text in CTC speech recognition technology.

[0079] As shown in FIGS. 4, 5, 6, and 7, the speech recognition model 210 according to the related art 1 includes a convolution 221 and an encoder 222. l 222 (l-layer (low-layer) encoder) and Encoder m 223 (m-layer (middle layer) encoder) and Encoder h The speech recognition model 210 includes an h-layer (high layer) encoder 224, a CTC loss calculator 225, a pseudo-CTC symbol string generator 230, a CTC symbol string generator 231, an Adapter 232, a CTC loss calculator 233, a mean square error loss calculator 234, a CTC loss calculator 261, a CTC loss calculator 262, and a CTC loss calculator 263. The speech recognition model 210 may also be realized by, for example, a computer and a program. At least a part of the speech recognition model 210 may be realized by a dedicated electronic circuit.

[0080] In the method of Related Art 1, the adapter is trained so that the output of the adapter when text is input matches the output of the intermediate layer of the pre-trained encoder when speech is input. In order to obtain a better correspondence between the output from the encoder and the output from the adapter, CTC loss calculation units 261, 262, and 263 are also provided in the intermediate layer of the encoder during pre-training. Each of the CTC loss calculation units 261, 262, and 263 calculates a CTC loss function. Then, during pre-training, training is performed based on the values ​​of these CTC loss functions. This allows the output from the intermediate layer of the encoder to include more linguistic information.

[0081] The training of the speech recognition model 210 in the related art 1 is carried out in three stages. The first stage is pre-training of the Intermediate CTC speech recognition model. The second stage is training of the adapter. The third stage is post-stage training of the Intermediate CTC speech recognition model. The training of each of these stages will be explained below.

[0082] [Related technology 1: First stage of training in a speech recognition model adaptation method using text data: Pre-training of the Intermediate CTC speech recognition model] As shown in Figure 4, the Intermediate CTC speech recognition model uses Convolution221 and Encoder. l 222 and Encoder m 223 and Encoder h It is composed of four neural network modules with 224.

[0083] Convolution 221 calculates vector h0 (bold) based on the acoustic feature vector x. Convolution 221 is realized using a convolutional neural network. l 222 is the vector h based on the above vector h0 (bold). l (bold) Encoder m 223 is the above vector h l (bold) based on the vector h m (bold) Encoder h 224 is the above vector h m (bold) based on the vector h h (bold) Encoder l 222, Encoder m 223, and Encoder h Each of the four neural network modules uses a self-attention neural network. These four neural network modules are composed of multiple layers of networks. l 222, Encoder m 223, and Encoder h Calculate the value of the CTC loss function from each of the 224 outputs.

[0084] Specifically, the CTC loss calculation unit 261 calculates the vector h l Based on the CTC loss L l The CTC loss calculation unit 262 calculates the vector h m Based on the CTC loss L m The CTC loss calculation unit 263 calculates the vector h h Based on the CTC loss L l In other words, in this first stage of learning, the CTC loss function is also calculated for the intermediate layers of the neural network of the intermediate CTC speech recognition model, and the weighted sum of these losses is used as the final loss for learning. This will be explained in more detail below.

[0085] The output h0 (bold type) from the convolution 221 has the number of dimensions shown in equation (13) and is calculated by equation (14).

[0086]

number

[0087]

number

[0088] In addition, in formula (13), D h is the dimension of the vector, and D h is the sequence length.

[0089] Encoder l h, which is the output from 222 l (bold) and Encoder m h, which is the output from 223 m (bold) and Encoder h h, which is the output from 224 h (Bold) has the number of dimensions shown in equation (15) and is calculated by equations (16), (17), and (18), respectively.

[0090]

number

[0091]

number

[0092]

number

[0093]

number

[0094] Encoder l 222, Encoder m 223, and Encoder h Loss function L for each output from each of 224 l , L m , and L h are as shown in equations (19), (20), and (21) below.

[0095]

number

[0096]

number

[0097]

number

[0098] The objective function of the base model is, for example, the loss L l , L m , and L h In this case, the loss Loss1 is calculated by the following equation (22).

[0099]

number

[0100] In other words, in the first stage of learning here, the parameters of the model are adjusted to minimize the loss Loss1.

[0101] [Related technology 1: Second stage of learning in speech recognition model adaptation method using text data: Adapter learning] In the related art 1 described above, the input text string is converted into a pseudo CTC symbol string, and this CTC symbol string is converted into a feature frame by the neural network Adapter described later. h By retraining the neural network, we adapt it to the topic of the genre to which the text string belongs.

[0102] Fig. 5 is a block diagram for explaining the training of the Adapter. In addition to the configuration described with reference to Fig. 4, the speech recognition model 210 has a CTC symbol string generation unit 231, an Adapter 232, a CTC loss calculation unit 233, and a mean square error loss calculation unit 234, as described above.

[0103] The method of training Adapter 232 is as follows. The acoustic feature x (in bold) is input to Convolution 221 of the pre-trained Intermediate CTC speech recognition model, and h is calculated using the following equations (23) and (24). m (bold) is calculated.

[0104]

number

[0105]

number

[0106] The CTC symbol string generator 231 uses h calculated by the above formula (23). l (bold) corresponds to the most probable alignment a l (bold, hatched) is calculated using equation (25).

[0107]

number

[0108] In addition, a obtained from equation (25) l Based on the (bold, hat) CTC alignment sequence was generated by Encoder m Output feature frame h m (bold) to approach the Adapter232. However, the output from Adapter232 is h m (bold, hatched) is calculated by the following equation (26).

[0109]

number

[0110] h m (bold, hatched) to h m In order to learn to get closer to (bold), the loss L is calculated using the following equations (27) and (28). m and L mse Specifically, the CTC loss calculation unit 233 calculates the loss L m In addition, the mean square error loss calculation unit 234 calculates the loss L mse Calculate.

[0111]

number

[0112]

number

[0113] The objective function Loss2 in the learning of the Adapter 232 is calculated by the following equation (29).

[0114]

number

[0115] In the formula (29), α is a hyperparameter and is determined appropriately.

[0116] In other words, the parameters of Adapter 232 are adjusted so as to minimize the value of Loss2 in equation (29). Here, the learning of the parameters of Adapter 232 at this stage is performed for layers other than the final layer. After learning is completed, Adapter 232 outputs the input CTC symbol string to Encoder 232. m 223 output feature frame h m (bold).

[0117] [Related technology 1: Third stage learning in a speech recognition model adaptation method using text data: learning after Intermediate CTC] As the third stage of learning, the latter stage of Intermediate CTC (i.e., Encoder h The training data used in this third stage of training is text strings that contain the topic (target domain) of the speech to be recognized (inferred). This training data is used for the Encoder, which is the latter stage of Intermediate CTC. h The third stage of training is domain adaptation, which adapts the learning to a specific topic (target domain). h In order to perform re-learning, the method described in either Fig. 6 or Fig. 7 may be selected arbitrarily. In the following, the first domain adaptation method (Fig. 6) and the second domain adaptation method (Fig. 7) will be described.

[0118] [Related technology 1: First domain adaptation method in speech recognition model adaptation method using text data: Domain adaptation using only text from the target domain] FIG. 6 is a block diagram for explaining domain adaptation using only the text of the target domain. As shown in the figure, a speech recognition model 210 includes a pseudo-CTC symbol string generator 230. In this first domain adaptation method, h In this method, only the text of the target domain is used for retraining H.224. Although this domain adaptation method degrades the speech recognition accuracy of the source domain, it is expected to improve the speech recognition accuracy of the target domain more than the second domain adaptation method described below.

[0119] In this method, the pseudo CTC symbol string generator 230 generates N pseudo CTC symbol strings a (N is a positive integer) based on a text string of a target domain. i (bold, hatched) are generated, where i = 1, 2, . . . , N. The method for generating pseudo-CTC symbol strings will be described later. i (Bold, hatched) for Adapter232 and Encoder h 224 is output h by the following equation (30): h,i (bold, hatched). In addition, the CTC loss calculation unit 235 calculates the loss function value L h,i Calculate.

[0120]

number

[0121]

number

[0122] Then, the N L values ​​obtained by Eq. (31) are h,i Based on this, the objective function value Loss3 is calculated.

[0123]

number

[0124] In other words, Loss3 is a function of N L h,i In this first domain adaptation method, the parameters of the neural network that constitutes the model are adjusted to minimize Loss3. However, in this method, the parameters of Adapter232 are not trained, and the parameter values ​​are fixed. When training by this method is completed, h 224 is expected to be a neural network that is adapted to the topics of text strings in the target domain.

[0125] [Related technology 1: Second domain adaptation method in speech recognition model adaptation method using text data: Domain adaptation using target domain text and source domain speech-text pairs] FIG. 7 is a block diagram for explaining domain adaptation using a target domain text and a source domain speech-text pair. In this method, the Encoder h In the second domain adaptation method, not only the target domain text but also the source domain speech text data is used for re-training the speech recognition algorithm in accordance with the first domain adaptation method. In the second domain adaptation method, the improvement in speech recognition accuracy in the target domain is smaller than that in the first domain adaptation method, but the deterioration in speech recognition accuracy in the source domain can be reduced.

[0126] In this method, the pseudo-CTC symbol string generator generates N t pieces(N t is a positive integer) pseudo CTC string a t,i (bold, hat) for i=1,2,...,N t And, N t As expected t,i For (bold, hat), the loss function value L t,i Calculate.

[0127]

number

[0128]

number

[0129] In addition, we applied N s pieces(N s is a positive integer) acoustic feature x j (bold) and calculate the loss function value L using the following equations (35), (36), (37), and (38). s,j where j=1,2,...,N s It is.

[0130]

number

[0131]

number

[0132]

number

[0133]

number

[0134] And, N calculated by equation (34) t As expected t,i and N calculated by equation (38) s As expected s、j Based on this, the objective function value Loss st In other words, in this method, N t Loss function values ​​L t,i For the average of N s Loss function values ​​L s,j By adding the average of the Loss st Calculate the loss. st is calculated by the following equation (39).

[0135]

number

[0136] In other words, in this method, the Loss of Equation (39) st The parameter values ​​of the neural network that constitutes the model are adjusted so as to minimize . In this learning, the parameters of the adapter are not learned. In other words, the values ​​of the parameters of the adapter are fixed. h It is expected that this will result in a neural network that can adapt to the topics of text strings belonging to the target domain without compromising its processing ability for the speech of the source domain.

[0137] [Related technology 1: Pseudo-CTC symbol string generation method for speech recognition model adaptation using text data] Here, a pseudo-CTC symbol string generation method used in each of the above first and second domain adaptation methods will be described.

[0138] Consider converting a given text string into a pseudo-CTC symbol string. For example, if the base text string is "nice weather" and the length of the CTC symbol string is limited to 6 (characters), the following 9 CTC symbol strings can be generated. Note that "_" is a blank symbol. <blank>Represents. 1: Good weather 2: Good weather 3: Good weather 4: Good weather 5: Good weather 6: Good weather 7: Good weather 8: Good weather 9: Good weather

[0139] It should be noted that in CTC, when the same symbol appears consecutively, such as in the case of "ii" in "ii tenki," one or more blank symbols must be inserted between them. The blank symbols correspond to the parts of the feature frame where no pronunciation exists.

[0140] There are many CTC symbol strings that correspond to a certain character string (for example, "good weather"). Therefore, if all patterns of CTC symbol strings are generated, it is considered that it will take a huge amount of time to train a neural network. However, among CTC symbol strings, there are symbol strings that are theoretically possible but are difficult to generate in reality. For example, among CTC symbol strings of length 10 that correspond to the text "good weather", it is considered that a feature frame in which there is a long silent interval between "i" and "i", such as "ii _ _ _ _ _ _ ii weather", is difficult to generate in reality. Also, it is considered that a feature frame in which the pronunciation interval of a specific symbol is extremely long, such as "ii _ ii weather iki iki iki iki".

[0141] In other words, when generating a CTC symbol string based on a text string, it is considered acceptable to limit the pattern of the CTC symbol string to a pattern that can actually occur. l Output feature frame h from l The CTC symbol string a calculated from l (Bold, Hat) The number of consecutive identical characters (same symbols) and blank symbols <blank>Based on the statistics of the consecutive numbers of , a pseudo-CTC symbol string is generated according to a predetermined probability density function (conversion from a text string to a pseudo-CTC symbol string).

[0142] Here, the use of the probability density function will be explained. For example, the voice data used for training the Intermediate CTC may be assumed as the voice used for taking statistics. l From a l (bold, hatched) and output the a l (bold, hatched) The number of times the same character (same symbol) appears consecutively is N c (n) and then n blank symbols <blank>The number of consecutive occurrences of N b (n).

[0143] These N c (n) and N b Using (n), the probability density function P c (n) and blank symbols <blank>Probability density function P of consecutive numbers b (n) is defined as equations (40) and (41), respectively.

[0144]

number

[0145]

number

[0146] In other words, the probability density function P c (n) represents the probability that the number of consecutive identical symbols (e.g., letters) in a block in a CTC string is exactly n. Also, the probability density function P b (n) is a blank symbol in the CTC string. <blank>represents the probability that the consecutive number in a contiguous block is exactly n.

[0147] For example, for two pieces of audio data (one for "nice weather" and the other for "tomorrow"), l The CTC symbol string a calculated by l Let us consider the cases where (bold, hat) are "_ _ good _ good _ weather _ _ _" and "_ clear _ _ day _". However, the "_" in the CTC symbol string is a blank symbol. <blank>Represents.

[0148] In this case, for each n of 0≦n≦3, the number of consecutive characters N c (n) is as follows: N c (0)=0 (not applicable) N c (1)=3 (each of "i", "ten", and "ki" applies) N c (2)=2 (each of "Morning Morning" and "Day Day" applies) N c (3)=1 ("Good, good" applies) For n ≧ 4, N c (n)=0 Therefore, for each n in the range 0≦n≦3, P c (0)=0 / 6=0 P c (1)=3 / 6=1 / 2 P c (2)=2 / 6=1 / 3 P c (3)=1 / 6 For n ≧ 4, P c (n)=0 / 6=0 It is.

[0149] Blank symbol <blank>The number of consecutive digits is as follows: For each n in the range 0≦n≦3, a blank symbol <blank>The number of consecutive b (n) is as follows: N b (0)=1 (There is no "_" between "heaven" and "ki", so this is correct) N b (1)=4 (between "i" and "i", between "i" and "ten", before "min" and after "hi") N b (2)=1 (the part before "i" applies) N b (3)=2 (after "ki" and between "min" and "hi") For n ≧ 4, N b (n)=0 Therefore, for each n in the range 0≦n≦3, P b (0)=1 / 8 P b (1)=4 / 8=1 / 2 P b (2)=1 / 8 P b (3)=2 / 8=1 / 4 For n ≥ 4, P b (n)=0 / 8=0 It is.

[0150] The example explained above takes statistics for two speech data ("nice weather" and "tomorrow") and calculates P c (n) and P b When taking actual statistics, for example, all of the speech data used in training the Intermediate CTC may be used.

[0151] Next, a process for generating a pseudo CTC symbol string based on a text string will be described.

[0152] 8 is a flowchart showing a process for generating a pseudo CTC symbol string based on a given text string. A method for generating a pseudo CTC symbol string will be described below with reference to this flowchart.

[0153] When executing the process of this flowchart, the base text string is passed in the form of being stored in the memory area c[]. c[] can be thought of as an array of symbols (characters). The size of the memory area c[] is J. In other words, the length of the text string (number of symbols, number of characters) is J. In other words, the given text string is a string of symbols starting with c[1], followed by c[1], c[2], . . ., c[J].

[0154] In step S01, the pseudo-CTC symbol string generator 230 calculates a i (bold, hatched). That is, the pseudo-CTC symbol string generator 230 initializes a i Set the value of (bold, hatched) to the null string (null symbol string). Here, a i (Bold, hatched) is a working memory area for storing the pseudo-CTC symbol string generated by the pseudo-CTC symbol string generating unit 230. In other words, i (Bold, hatched) are variables. In the following steps, the pseudo-CTC symbol string generator 230 i A pseudo-CTC symbol string is generated by sequentially concatenating strings (symbol strings) with (bold, hatched).

[0155] Step S02 is the beginning of the first loop process (for convenience, called loop 1). Loop 1 uses an integer-valued variable j as an index and repeats the process in the loop until a predetermined condition is met. The variable j indexes the symbols stored in the array c[]. Note that "j:1,+1,J" is a description that represents the control of the loop process. That is, the initial value of the variable j is 1. Furthermore, the action performed on the variable j each time the loop is executed (that is, each time the process returns from step S10 to step S04) is an increment of the value. In other words, the value of the variable j is incremented by "1" each time the loop is executed. Furthermore, the end condition of loop 1 is when the value of the variable j reaches J.

[0156] In step S03, the pseudo-CTC symbol string generator 230 calculates a probability density function P b According to (n), probabilistically, <blank>The number of consecutive b In this step, the pseudo-CTC symbol string generation unit 230 appropriately uses a random number generation means or a pseudo-random number generation means to select the consecutive number N b When performing this step, the consecutive number N b The value of is not definitive, but the probability density function P b It will be determined according to the probability distribution determined by (n).

[0157] In step S04, the pseudo-CTC symbol string generation unit 230 judges whether the condition "(j ≠ 1) AND (c [j-1] == c [j])" is true or false. In other words, the pseudo-CTC symbol string generation unit 230 judges whether the symbol indexed by the variable j is the same as the previous symbol (the symbol indexed by j-1). That is, the pseudo-CTC symbol string generation unit 230 judges whether the same symbol appears consecutively at the current position and the previous position. This condition is true (TRUE) when j is 2 or more and the element (symbol, character) indexed by the value j in the array c [] is the same as the element (symbol, character) indexed by the value (j-1). This condition is false (FALSE) in other cases. That is, in this step, it is determined whether the symbol c[j] currently pointed to by j in the given text string is the same as the symbol c[j-1] pointed to by j one time ago. If the above condition is true (i.e., the same symbols are consecutively arranged in c[j] and c[j-1] in the given text string) (step S04: YES), the process proceeds to the next step S05. If the above condition is false (step S04: NO), the process jumps to step S08.

[0158] When the process proceeds to step S05, the pseudo-CTC symbol string generating unit 230 performs the process from step S05 to step S07 by inserting a predetermined number (one or more) of blank symbols between consecutive identical symbols (characters). <blank>To insert, select 1 or more N b is selected.

[0159] Specifically, step S05 is the start of "loop 2." The condition for repeating "loop 2" is b The value of N is 0. b The value of N is selected probabilistically in step S03 or step S06. b If the value of is not 0 (i.e., is 1 or more), the process in "Loop 2" is not executed even once, and "Loop 2" is exited. Conversely, if the value of N selected in step S03 is b If the value of is 0, in step S06, b The process of "loop 2" is repeated until a value other than 0 (i.e., 1 or more) is selected as the value of the probability density function P b According to (n), N b The value of the probability density function P b According to (n), N b The process of selecting the value of is the same as the process in step S03 above. Step S07 is the end point of "loop 2." After leaving "loop 2" in step S07, the process proceeds to step S08.

[0160] As described above, if the value of j is 2 or more and "c[j-1] == c[j]" is true (if the same symbol appears consecutively), N b In other cases, that is, when the value of j is 1 or when "c[j-1] ≠ c[j]" is true, a value of N equal to or greater than 0 is selected before the process of step S08. b The value of is chosen probabilistically.

[0161] In step S08, the pseudo-CTC symbol string generator 230 selects the already selected N b According to the value of N b Blank symbols <blank>to variable a in the work area. i Concatenate to the end of the string (symbol string) stored in (bold, hatched).

[0162] In step S09, the pseudo-CTC symbol string generator 230 concatenates a predetermined number of consecutive symbols c[j] to a character string (symbol string) in the work area. c According to (n), the probability of the consecutive number of symbols (characters) N c In this step, the pseudo-CTC symbol string generation unit 230 appropriately uses a random number generation means or a pseudo-random number generation means to select the consecutive number N c When performing this step, the consecutive number N c The value of is not definitive, but the probability density function P c The pseudo CTC symbol string generator 230 determines the probability distribution of the determined N c According to the value of c c[j] is stored in the variable a of the work area. i Concatenate to the end of the string (symbol string) stored in (bold, hatched).

[0163] Step S10 is the end point of "loop 1." That is, when step S10 is reached, as explained in step S02, j is incremented (+1), the loop repetition condition (j≦J) is checked, and if the loop repetition condition is true, the process inside "loop 1" for the next round is performed. When "loop 1" is exited, the process proceeds to the next step S11.

[0164] In step S11, the pseudo-CTC symbol string generator 230 generates a predetermined number (0 or more) of blank symbols. <blank>Specifically, the pseudo-CTC symbol string generator 230 concatenates the probability density function P b According to (n), probabilistically, <blank>The number of consecutive b Select N b The process of selecting N is the same as the process in step S03. Then, the pseudo-CTC symbol string generator 230 selects N b According to the value of b Blank symbols <blank>to variable a in the work area. i (bold, hatched) is concatenated to the end of the string (symbol string). i (Bold, hatched) is the pseudo-CTC symbol string generated by pseudo-CTC symbol string generation unit 230.

[0165] According to the above-described processing procedure, the pseudo CTC symbol string generator 230 generates a probability density function P c (n) and P b According to (n), letters (symbols) and blank symbols <blank>In other words, the pseudo-CTC symbol string generator 230 generates a pseudo-CTC symbol string that is statistically plausible based on a given text character string.

[0166] Next, a procedure for generating a pseudo CTC symbol string by the pseudo CTC symbol string generator 230 will be described according to an example of an actual text string. Here, an example of a base text string is "Tomorrow will be a nice day."

[0167] Step 0: The string in the work area that stores the generated CTC symbol string is " " (null symbol string). Step 1: Probability density variable P b According to (n), probabilistically, <blank>Select consecutive numbers of . b If =3 is selected, then three blank symbols <blank>Connect it to the rear side of the column of the working area. As a result, the column of the working area is "_ _ _". Step 2: The first symbol of the given string is "明". According to the probability density variable P c (n), probabilistically select the number of consecutive symbols (characters). Probabilistically, N c Consider the case where N = 2 is selected. Connect two "明" to the rear side of the column of the working area. As a result, the column of the working area is "_ _ _ 明 明". Step 3: According to the probability density variable P b (n), probabilistically, the blank symbol <blank>Select consecutive numbers of . b If =1 is selected, one blank symbol <blank>Connect it to the rear side of the column in the working area. As a result, the column in the working area is "_ _ _ Ming Ming _". Step 4: The next symbol of the given string is "Ri". Probability density variable P c (n), probabilistically select the number of consecutive symbols (characters). Probabilistically, N c Consider the case where N = 3 is selected. Connect two "Ming" to the rear side of the column in the working area. As a result, the column in the working area is "_ _ _ Ming Ming _ Ri Ri Ri".

[0168] Similarly, for each of the symbols "Ming", "Ri", "Ha", "I", "I", "Tian", "Qi", and the blank symbol <blank>and alternately, the probability density variable P c (n) and P b The sequence number is determined according to (n), and the blank symbol between the symbols "i" and "i" is added to the end of the sequence in the work area. <blank>The number of consecutive N is set to 1 or more (i.e., not 0). b Be sure to select a value for .

[0169] The pseudo-CTC symbol string generator 230 repeats the above pseudo-CTC symbol string generation process N times for a given text string, thereby generating N CTC symbol strings a i (bold, hat) for i = 1, 2, , N.

[0170] [Related Technology 1: Inference in speech recognition model adaptation methods using text data] The procedure for inference using a trained model is as follows: Based on the input acoustic feature x (bold), the most likely CTC symbol string a (bold, hat) is calculated using the following formulas (42) and (43).

[0171]

number

[0172]

number

[0173] Then, as in normal CTC speech recognition, consecutive characters (symbols) contained in the CTC symbol string a (bold, hatched) calculated by equation (43) are combined into one character, and then blank symbols are added. <blank>The recognition result string (symbol string) is obtained by deleting

[0174] [Related technology 2: Adaptation method for transducer speech recognition models using text data (Maestro)] 9 is a block diagram showing the configuration of a transducer speech recognition model according to Related Art 2. As shown in the figure, the transducer speech recognition model 310 includes a speech encoder 321 (a speech encoder) and a text encoder 331 (a text encoder). The speech encoder 321 derives a feature quantity c s The text encoder 331 calculates and outputs the feature quantity c t Calculate and output (hat).

[0175] The "method of adapting a speech recognition model using text data" of Related Technology 1 described above is a method of training a CTC speech recognition model using text-only training data. On the other hand, Related Technology 2 described here is a method of training a transducer speech recognition model, which is excellent for streaming applications, using text-only training data, rather than a CTC speech recognition model. Related Technology 2 is also called "Maestro." In the method of Related Technology 2, in the transducer, training is performed on a text encoder so that the output from the intermediate layer (speech encoder 321) of the pre-trained encoder when speech is input matches the output from the text encoder (text encoder 331) when text is input.

[0176] In the above-mentioned related technique 1, the CTC loss function is also calculated for the intermediate layer of the Encoder during pre-learning, so that the output of the Encoder intermediate layer contains more linguistic information and the output from the Encoder intermediate layer corresponds to the output from the Adapter. Meanwhile, in related technique 2 (Maestro), learning is performed simply so that the output from the Text Encoder (text encoder 331) matches the output from the intermediate layer of the Encoder (audio encoder 321).

[0177] However, it is expected that the output from the intermediate layer of the Encoder (audio encoder 321) will have insufficient or unclear alignment information with the output from the text encoder 331. For this reason, as in the case of Related Art 1, if alignment information is clearly added to the output from the intermediate layer of the Encoder, it is expected that the performance improvement effect by learning using text data will be even greater.

[0178] [Learning of the voice recognition device in this embodiment] The training of the speech recognition device 10 in this embodiment is performed in the following three steps. The first step is training of a transducer speech recognition model with a CTC loss in the intermediate layer. The second step is training of a text encoder. The third step is training of a transducer speech recognition model using only text.

[0179] [First step: Training a transducer speech recognition model with CTC loss in the intermediate layer] FIG. 10 is a schematic diagram showing a model structure of a transducer according to this embodiment. The correspondence between the configuration shown in FIG. 10 and the configuration shown in FIG. 1 is as follows. The Speech Encoder 521 corresponds to the speech encoder 21 in FIG. 1. The Shared Encoder 551 corresponds to the shared encoder 51. The Pred. Network 552 corresponds to the prediction network 52. The Joint Network 553 corresponds to the joint network 53. The Softmax 554 corresponds to the softmax function processing unit 54. The Softmax 561 corresponds to the softmax function processing unit 61. The Transducer loss shown in FIG. 10 is the loss calculated by the loss calculation unit 55. The CTC loss shown in FIG. 10 is the loss calculated by the loss calculation unit 62.

[0180] A feature of the transducer of this embodiment is that it calculates CTC loss even in the middle layer of the Encoder by providing a loss calculation unit 62. The network on the input stage side of the middle layer of the Encoder is called a Speech Encoder 521, and the network on the output stage side is called a Shared Encoder 551.

[0181] The acoustic feature x (in bold) input to the Speech Encoder 521 or the acoustic feature x after subsampling by a convolutional neural network or the like has the dimensionality expressed in Equation (44).

[0182]

number

[0183] Also, the output from the Speech Encoder521 s (Bold) has the number of dimensions shown in equation (45).

[0184]

number

[0185] The correct symbol string y (in bold) is expressed by equation (46), where U is the length of the correct symbol string y (in bold) (the number of symbols in the string).

[0186]

number

[0187] CTC Loss L ctc is calculated by the loss calculation unit 62 (FIG. 1) using the following equation (47).

[0188]

number

[0189] Transducer Loss L trans is calculated by the loss calculation unit 55 (FIG. 1) using the following equation (48).

[0190]

number

[0191] The loss L used in this first step of learning base is expressed by the following formula (49). Here, α is a hyperparameter. The value of α may be limited to the range of 0≦α≦1.

[0192]

number

[0193] [Second step: learning the text encoder] 11 is a schematic diagram for explaining the learning of the text encoder in this embodiment. A text encoder 532 shown in FIG. 11 corresponds to the text encoder 32 shown in FIG.

[0194] That is, in this embodiment, a text encoder 532 that receives text as input is used in addition to the transducer speech recognition model. In this second step of learning, the text encoder 532 is learned so that the output from the text encoder 532 when text is input matches the output from the speech encoder 521. For this purpose, the MSE loss (mean squared error loss) between the output vector from the text encoder 532 and the output vector from the speech encoder 521 is used. This MSE loss is calculated by the loss calculation unit 41 (FIG. 1).

[0195] However, the length of an acoustic feature sequence is usually longer than the length of the sequence of the spoken text contained in the speech. Here, the length refers to the number of elements contained in the sequence. This is because the time corresponding to a single acoustic feature vector is usually shorter than the time corresponding to a single symbol (character) contained in the spoken text. Therefore, the sequence input to the Text Encoder 532 is not simply text, but is a CTC symbol string obtained from the intermediate layer when the acoustic features are input, or a transducer symbol string obtained from the transducer output. The CTC symbol string is information output from the Speech Encoder 521 and the Softmax 561. The transducer symbol string is information output from the Joint Network 553 and the Softmax 554.

[0196] We consider the following two types of CTC symbol strings, or transducer symbol strings. The first type of symbol string uses the output when acoustic features are input to the transducer that has undergone the first step of learning as is (called "output alignment"). The second type of symbol string uses a probability distribution calculated by the Softmax function when acoustic features are input to the transducer, and uses forced alignment of ground truth data using the Viterbi algorithm (called "forced alignment").

[0197] The output alignment above is a symbol string that includes errors. On the other hand, the forced alignment above uses the correct symbol and is a symbol string that does not include errors. The data input to the Text Encoder 532 can have the following four patterns depending on the combination of output alignment and forced alignment for each of the CTC symbol string and the transducer symbol string.

[0198] The symbol string of the first pattern is the output alignment of the transducer symbol string. An example of the symbol string of the first pattern shown in FIG. 11 is "_ heaven _ _ _ _". The symbol string of the second pattern is the forced alignment of the transducer symbol string. An example of the symbol string of the second pattern shown in FIG. 11 is "_ _ heaven _ _ air". The symbol string of the third pattern is the output alignment of the CTC symbol string. An example of the symbol string of the third pattern shown in FIG. 11 is "_ _ heaven heaven _ _". The symbol string of the fourth pattern is the forced alignment of the CTC symbol string. An example of the symbol string of the fourth pattern shown in FIG. 11 is "_ heaven heaven _ air _".

[0199] Text Encoder532 takes the alignment sequence a (bold) of one of the four patterns above as input and generates t (bold type) is output. a (bold type) is expressed by equation (50). Also, h t (boldface), has the dimension expressed by equation (51), where T is the length of the alignment sequence a (boldface) (the number of symbols in the sequence).

[0200]

number

[0201]

number

[0202] Using the learning data used in the first step of learning, h s (bold) and h t The loss calculation unit 41 (FIG. 1) calculates the MSE loss L mse Then, calculate this L mse We train Text Encoder532 to minimize

[0203]

number

[0204] However, only the parameters of the network are updated in the second step of learning. In the second step of learning, the parameters of the other networks are fixed. The second step of learning technique differs from Related Art 2 (Maestro) in the following points. That is, Related Art 2 (Maestro) does not add CTC loss to the output from the Speech Encoder. On the other hand, in the second step of learning technique of this embodiment, a CTC loss is calculated for the output from the Speech Encoder 521 so as to achieve better alignment between the Text Encoder 532 and the Speech Encoder 521. Also, in Related Art 2 (Maestro), duplication is performed on the neural network output in the Text Encoder, but in the technique of this embodiment, duplication is performed on the symbols.

[0205] [Third step training: Training the transducer speech recognition model using only text] 12 is a schematic diagram for explaining the learning in the third step. A Pseudo-alignment Generator 531 shown in FIG. 12 corresponds to the pseudo-alignment generating unit 31 shown in FIG.

[0206] In the learning in this step, text is input to the trained Text Encoder 532, and re-learning is performed with the Shared Encoder 551, the Joint Network 553, and the Pred. Network 552, which are networks other than the Text Encoder 532. However, since the Text Encoder 532 is trained to receive a CTC symbol string or a transducer symbol string, it is necessary to pseudo-generate a CTC symbol string or a transducer symbol string from a text string. Therefore, the pseudo-CTC symbol string generation method described in the above-mentioned Related Art 1 is used. In other words, the Pseudo-alignment Generator 531 pseudo-generates a symbol string from an input text string. This generation method is called a pseudo-alignment generation method.

[0207] The pseudo-alignment generator 531 generates four patterns of pseudo symbol strings, as described with reference to Fig. 11. The symbol string of the first pattern is a pseudo output alignment of a transducer symbol string. The symbol string of the second pattern is a pseudo forced alignment of a transducer symbol string. The symbol string of the third pattern is a pseudo output alignment of a CTC symbol string. The symbol string of the fourth pattern is a pseudo forced alignment of a CTC symbol string.

[0208] In the pseudo-alignment generation method described above, the number of times each symbol appears consecutively in the generated symbol string (this number is called the "consecutive number") is appropriately determined. For this purpose, a probability distribution of the consecutive numbers is required. Therefore, statistics are taken of the number of times each symbol v appears consecutively from the alignment sequence used in the learning in the second step. Note that the symbol v is as shown in the following equation (53).

[0209]

number

[0210] where V is the set of vocabulary to be recognized, <blank>is a blank symbol. In the statistics taken, the number of times that the symbol v appears consecutively n times is defined as N v Then, the probability density function P of the consecutive number n for the symbol v is v (n) is expressed by the following equation (54).

[0211]

number

[0212] The probability density function P v The pseudo-symbol string is generated by determining the consecutive number of each symbol included in the correct answer text by the inverse function method using the cumulative distribution function generated from (n), and duplicating each symbol according to the determined consecutive number. When generating the pseudo-symbol string, the Pseudo-alignment Generator 531 inserts blanks at the beginning, end, and between symbols of the given correct answer text, and also duplicating blank symbols. However, since the transducer does not output symbols other than blanks consecutively, the Pseudo-alignment Generator 531 does not perform duplication to make symbols other than blanks appear consecutively in the pseudo-symbol string of the transducer.

[0213] The symbol string generated by the Pseudo-alignment Generator 531 is input to the Text Encoder 532, and the transducer loss L trans The training step is performed by calculating the following. The speech used in the first and second training steps is also input to the Speech Encoder 521 for training, which is expected to prevent deterioration of recognition accuracy for the domain of speech used in the first and second training steps.

[0214] As described above, the parameters of the Shared Encoder 551, the Joint Network 553, and the Pred. Network 552 of the speech recognition device 10 are adjusted by the learning in the third step.

[0215] FIG. 13 is a block diagram showing an example of an internal configuration for realizing the voice recognition device 10 of the present embodiment, and each device (model) in the base technologies 1 and 2 and the related technologies 1 and 2. Each device can be realized by using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be realized by using existing technology. The central processing unit 901 executes instructions included in a program read from the RAM 902 or the like. The central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations according to each instruction. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. Note that RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with an external input / output device or the like. The input / output devices 904 and 905 exchange data with the central processing unit 901 via the input / output port 903. The bus 906 is a common communication path used inside the computer. For example, the central processing unit 901 reads and writes data from and to the RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port 903 via the bus 906.

[0216] At least a part of the functions of the voice recognition device 10 of the above-mentioned embodiment, the functions of the base technologies 1 and 2, and the related technologies 1 and 2 can be realized by a computer and a program. In that case, the functions may be realized by recording a program for realizing the functions in a computer-readable recording medium, reading the program recorded in the recording medium into a computer system, and executing the program. Note that the "computer system" here includes hardware such as an OS and peripheral devices. Furthermore, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, an optical magnetic disk, a ROM, a CD-ROM, a DVD-ROM, a USB memory, and a storage device such as a hard disk built into a computer system. In other words, the "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, the "computer-readable recording medium" may include a medium that temporarily and dynamically holds a program, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. Furthermore, the above program may be for realizing part of the functions described above, or may be capable of realizing the functions described above in combination with a program already recorded in the computer system.

[0217] As described above, according to the present embodiment, the CTC loss is calculated for the output from the intermediate layer of the transducer speech recognition model, and the output from the intermediate layer is trained, so that alignment information is explicitly added to the output from the intermediate layer. This allows the transducer speech recognition model to be trained only with text. In addition, by utilizing the configuration that allows the streaming speech recognition model to be trained with training data consisting of text only, the speech recognition accuracy of topics (target domain) related to the content of the text can be improved.

[0218] That is, according to this embodiment, a pre-trained transducer speech recognition model can be trained using text data on a topic (target domain) for which the recognition accuracy is to be improved. This makes it possible to improve the speech recognition accuracy of the topic to be recognized. The transducer speech recognition model can also operate in real time. Furthermore, by using a transducer speech recognition model adapted to a specific topic to recognize the speech included in video content, it is also possible to add subtitles in real time, for example.

[0219] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included.

[0220] The reference characters in the claims merely indicate the correspondence with the embodiments and do not limit the scope of the claims.

[0221] [Verification of effectiveness] A demonstration experiment was conducted to verify the effect of this embodiment. Specifically, it was confirmed to what extent speech recognition accuracy is improved by learning using only text, in the cases where the CTC loss is calculated and not calculated in the intermediate layer of the transducer encoder. In the demonstration experiment, an English open corpus was used. The experimental specifications are as follows:

[0222] [Specifications] As features, we used an 80-dimensional log-mel filter bank for every 10 milliseconds of speech. In the streaming transducer model used, the sequence length was subsampled to 1 / 4 using two layers of Causal Convolution. The speech encoder has 6 conformer layers, 8 attention heads, and a linear transformation layer dimension of 2048. The output dimension is 512 dimensions, and the look ahead is 0. The text encoder has 6 conformer layers, an attention head of 8, and a linear transformation layer dimension of 2048. The output dimension is 512, and the look ahead is 0. The shared encoder has 6 conformer layers, an attention head of 8, and a linear transformation layer dimension of 2048. The output dimension is 512, and the look ahead is 0. The prediction network had two LSTM layers and the linear transformation layer had dimensions of 512. The joint network had a post-joint dimension of 640. However, the depthwise convolution in the above conformer was causal convolution.

[0223] The following data was used as learning data. In the first and second steps of training, we used approximately 960 hours of speech-text pairs from the LibriSpeech training data. In the third step of learning, we used 37,414 sentences of text from the WSJ si-284 training data.

[0224] The following data was used as validation data. In the first and second steps of training, we used speech-text pairs and 5,567 utterances from the LibriSpeech validation data. In the third step of training, we used 503 sentences of text from the WSJ dev93 validation data.

[0225] The following data was used as evaluation data. · 2620 speech-to-text pairs in the LibriSpeech test clean evaluation data ·LibriSpeech test other evaluation data: 2939 speech-to-text utterances WSJ eval92 evaluation data: 333 speech-to-text utterances 4,500 subwords were used as output words.

[0226] The values ​​of the hyperparameter α in training the transducer speech recognition model with CTC loss in the intermediate layer were 0.0 and 0.2.

[0227] The models used in learning from the first step to the third step are as follows: First step: Best loss on LibriSpeech validation data after 50 epochs of training Second step: Best loss on LibriSpeech validation data after 50 epochs of training Third step: To be described later.

[0228] [result] First, we evaluated the word error rate (WER) of the model after the first and second steps of training when the hyperparameter α was set to various values, for each evaluation dataset of LibriSpeech test clean and LibriSpeech test other. The results are shown in Table 1 below.

[0229] [Table 1]

[0230] As shown in Table 1, when α = 0.2, the word error rate is lower than when α = 0.0. By adjusting α using the speech-text pair data for pre-training the transducer, it is also possible to improve the speech recognition accuracy of the transducer itself before text training.

[0231] Experiments were conducted using WSJ data for the model with α=0.0 and the model with α=0.2, which showed the above-mentioned improvement. In the third step, the transducer model was trained using only text. The WSJ text was trained for 10 epochs using various alignments, and the word error rate of the model after training for one epoch out of the 10 epochs was the lowest for all alignments. The word error rate (WER) after training for one epoch with various alignments is shown in Table 2. Note that the numbers in parentheses in Table 2 are the reduction rates of the word error rate compared to before the alignment training. From the results in Table 2, the CTC output alignment when α=0.2 had the highest reduction effect on the word error rate, with a reduction rate of 0.059. This confirmed the effectiveness of the method of providing a CTC loss calculation means in the intermediate layer of the transducer encoder.

[0232] [Table 2]

[0233] In the third step, learning using only text is based on the assumption that verification data for the target domain cannot be obtained. Therefore, when calculating the loss of the verification data, the loss is calculated using only text data, as in the case of learning. Evaluation was also performed using a model with the best loss on the verification data, and the recognition accuracy was found to be worse than before alignment learning in both methods. This is thought to be due to the fact that the vectors of the text encoder output and the speech encoder output are diverged, and when the subsequent network is optimized using the text encoder output, the processing power for the original speech encoder output decreases. This problem may be prevented by using speech-text pair data and training using both the speech encoder output and the text encoder output to optimize the text encoder output within a range that does not reduce the loss of the verification data for the speech-text pair.

[0234] Table 3 below shows a comparison of the word error rate (WER) in four cases from A) to D). A) is the case before alignment learning. B) is the case of CTC output alignment. C) is the case in which the voice in the training data of WSJ is input to the Speech Encoder, and the CTC symbol string obtained from the intermediate layer (CTC output alignment Oracle) is input to the Text Encoder, thereby learning a network other than the Speech Encoder as in this embodiment. D) is the case in which the voice in the training data of WSJ is input, and a network other than the Speech Encoder is learned as in this embodiment. Table 3 shows the word error rate of the WSJ evaluation set in each of the above cases from A) to D). Case C) shows the upper performance limit of the pseudo-alignment generation method used in this embodiment. Case D) of voice input is considered to show the upper performance limit when the alignment is ideal and the feature vector of the ideal Speech Encoder output is obtained. However, the value of the hyperparameter α was 0.2, and the model was trained for one epoch.

[0235] [Table 3]

[0236] As shown in Table 3, the improvement in word error rate in the case of C) where the CTC output alignment was ideally generated (Oracle) was 2.7 points compared to the CTC output alignment in case B) (word error rate from 12.8 to 10.1). The CTC output alignment generated by the pseudo-alignment generation method in case B) is generated from the correct text. Therefore, unlike the CTC output alignment in the case of C) where it was ideally generated (Oracle), it does not contain any symbol errors. In order to make up for this 2.7 point difference, it is necessary to verify whether symbol alignment is an important factor in generating the pseudo-alignment, or whether the way in which symbol errors are made that simulates actual speech recognition errors is important.

[0237] There was also a difference of 2.5 points between the CTC output alignment (Oracle) in case C) and the speech input in case D). This suggests that there is a difference between the output vector of the Text Encoder and the vector output by the Speech Encoder when actual speech is input. To improve this, it is necessary to select a loss other than the MSE loss and to verify the robustness of the alignment, etc. [Industrial Applicability]

[0238] The present invention can be used, for example, in voice recognition processing in all industries, and in processing for constructing (learning) a model for voice recognition processing, etc. However, the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0239] 10 Voice recognition device 21 Speech Encoder 31 Pseudo Alignment Generator 32 Text Encoder 41 Loss calculation unit (third loss calculation unit) 51 Shared Encoder 52 Prediction Network 53 Bonding Network 54 Softmax function processing section 55 Loss calculation unit (first loss calculation unit) 61 Softmax function processing section 62 Loss calculation unit (second loss calculation unit) 70 Control section 110 transducer speech recognition model 121 Encoder 152 Prediction Network 153 Bonding Network 154 Softmax function processing section 210 Speech Recognition Model 221 Convolution 222 Encoder l 223 Encoder m 224 Encoder h 225 CTC loss calculation section 230 Pseudo CTC symbol string generator 231 CTC symbol string generator 232 Adapter 233 CTC loss calculation section 234 Mean Square Error Loss Calculation Unit 261 CTC loss calculation section 262 CTC loss calculation section 263 CTC loss calculation section 310 transducer speech recognition model 321 Audio Encoder 331 Text Encoder 521 Speech Encoder 532 Text Encoder 551 Shared Encoder 552 Pred. Network 553 Joint Network 554 Softmax 561 Softmax 531 Pseudo-alignment Generator 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus< / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / sos> < / sos> < / sos> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank>

Claims

1. a first encoder that calculates and outputs intermediate layer features based on input acoustic features; a second encoder that calculates and outputs a final layer feature based on the intermediate layer feature output from the first encoder; a first loss calculation unit (55) that calculates a first loss based on the final layer feature output from the second encoder and a correct answer text corresponding to the input acoustic feature; a second loss calculation unit (62) that calculates a second loss based on the intermediate layer features output from the first encoder and a correct answer text corresponding to the input acoustic features; A text encoder that calculates and outputs text features based on an input text; a third loss calculation unit (41) that calculates a third loss based on the intermediate layer feature output by the first encoder and the text feature output by the text encoder; Equipped with A parameter of a network of the second encoder and a parameter of a network of the first encoder are adjustable based on the first loss, A parameter of a network of the first encoder is adjustable based on the second loss, The parameters of the network of the text encoder can be adjusted based on the third loss. Speech recognition device.

2. Based on the input text string, the symbols contained in the text string are ｃ times (n ｃ ≧1) A repeat operation is performed so that the characters in the text string are consecutively repeated, and n is added between the characters in the text string. ｂ times (n ｂ ≧0) a pseudo-alignment generator that generates pseudo-symbol strings by performing an operation of inserting consecutive blank symbols; Further equipped with When retraining the trained second encoder based on the text string, the text encoder calculates and outputs text features based on the pseudo symbol string generated by the pseudo alignment generating unit based on the text string as the text; the second encoder calculates and outputs the final layer feature based on the text feature output from the text encoder, instead of the intermediate layer feature output from the first encoder; 2. The speech recognition device according to claim 1.

3. The pseudo alignment generation unit generates a pseudo alignment by dividing the number of consecutive symbols by n ｃ Probability density function P determined from predetermined statistics for ｃ Based on (n), ｃ The value of n is determined, and the number of consecutive blank symbols is determined. ｂ Probability density function P determined from predetermined statistics for ｂ Based on (n), ｂ Determine the value of 3. The speech recognition device according to claim 2.

4. a first encoder that calculates and outputs intermediate layer features based on input acoustic features; a second encoder that calculates and outputs a final layer feature based on the intermediate layer feature output from the first encoder; a first loss calculation unit (55) that calculates a first loss based on the final layer feature output from the second encoder and a correct answer text corresponding to the input acoustic feature; a second loss calculation unit (62) that calculates a second loss based on the intermediate layer features output from the first encoder and a correct answer text corresponding to the input acoustic features; A text encoder that calculates and outputs text features based on an input text; a third loss calculation unit (41) that calculates a third loss based on the intermediate layer feature output by the first encoder and the text feature output by the text encoder; Equipped with A parameter of a network of the second encoder and a parameter of a network of the first encoder are adjustable based on the first loss, A parameter of a network of the first encoder is adjustable based on the second loss, The parameters of the network of the text encoder can be adjusted based on the third loss. A program that enables a computer to function as a voice recognition device.

Images