Speech recognition device and program

Abstract

To provide a speech recognizer that can train a speech recognition model using only text of a target topic (domain) and output a symbol sequence suitable for the topic (domain) during inference.SOLUTION: An encoder unit includes a model consisting of multiple layers so as to input a sequence of frames of acoustic features and output an output symbol sequence consisting of a sequence of symbols corresponding to frames of the acoustic features. An adapter unit inputs a sequence of symbols belonging to a target region and outputs features for input into the model of the last layer of the encoder unit. After machine learning based on the relationship between the frame sequence of acoustic features and the output symbol sequence, the encoder unit is configured to enable adaptation of the model of the last layer to the target region, by inputting the features output by the adapter unit into the model of the last layer of the encoder unit, and using a first loss that is a loss based on an output from the model of the last layer.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a speech recognition device and a program. [Background technology]

[0002] One of the methods used for speech recognition processing is the CTC speech recognition model. CTC stands for "Connectionist Temporal Classification." The CTC speech recognition model is a model that is configured to directly learn the relationship between speech and character strings without using phonetic symbols. In other words, the CTC speech recognition model is one of the end-to-end speech recognition model methods. The CTC speech recognition model does not include a blank character string in the output symbol string. <blank>and allows the same symbol to appear consecutively in the output symbol string. This allows the CTC speech recognition model to match the feature frame with the output symbol string.

[0003] Non-Patent Document 1 describes a method of intermediate loss regularization. Intermediate loss regularization is a learning method in which a means for calculating a CTC loss function is provided in the intermediate layer of a CTC speech recognition model, and the values ​​of the CTC loss functions of multiple layers are weighted and added together.

[0004] Non-Patent Document 2 describes the Shallow Fusion method. Shallow Fusion is a method for utilizing a language model trained only with text data for speech recognition. In the Shallow Fusion method, recognition results are output by linearly interpolating the inference probabilities of the speech recognition model and the language model in the logarithmic domain during inference. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Jaesong Lee, Shinji Watanabe, "Intermediate Loss Regularization for CTC-based Speech Recognition," In Proc. 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6224-6228, 2021. [Non-patent document 2] Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, Yoshua Bengio, "On Using Monolingual Corpora in Neural Machine Translation," in arXiv:1503.03535v2, 2015. Summary of the Invention [Problem to be solved by the invention]

[0006] For example, in Japanese, there are expressions such as "hashi" (bridge) and "hashi" (edge), which are different symbol sequences but have the same or similar pronunciation. The CTC speech recognition model assumes that the probabilities between output symbols (characters) are independent. In other words, the probability of an output symbol does not depend on the probabilities of other adjacent or nearby symbols. Therefore, for a pronunciation with multiple possible output symbol sequences, the CTC speech recognition model outputs a symbol sequence based on the correspondence between the pronunciation and the symbol sequence obtained as a training result. When such a CTC speech recognition model is trained to recognize speech on a topic different from that of the text used during training, it tends to produce a symbol sequence based on the training text. For example, if the speech being recognized is about the weather, the CTC speech recognition model may output "optimal sound" when it should output "minimum temperature" as the recognition result. In other words, the CTC speech recognition model has the problem of outputting a symbol sequence that is inappropriate for the topic of the text being inferred, resulting in recognition errors.

[0007] If data pairs of speech and text (correct speech recognition results) on a target topic are available, it is possible to train a CTC speech recognition model in advance using that data. However, such speech data is not always available in advance. Even if such data is available, the task of annotating the speech is costly. In other words, it is impossible or difficult to adapt the output symbol string of a CTC speech recognition model in conventional technology to a desired topic.

[0008] The present invention has been made based on the above-mentioned problem recognition, and aims to provide a speech recognition device and program that can train a speech recognition model using only the text data and output a symbol string appropriate for the topic (area) during inference, even when paired data of speech and text is not available for the topic (area) to be inferred, as long as only the text is available. [Means for solving the problem]

[0009] [1] In order to solve the above problem, a speech recognition device according to one aspect of the present invention includes an encoder unit having a model composed of multiple layers configured to input a sequence of frames of acoustic features and output an output symbol sequence consisting of a sequence of symbols corresponding to the frames of the acoustic features, each of the multiple layer models being configured to be capable of machine learning, and an adapter unit that inputs a symbol sequence belonging to a target domain and outputs features to be input to a model in a final layer of the encoder unit, wherein intermediate features are passed from a model in a certain layer of the multiple layer model of the encoder unit to a model in a next layer, and the encoder unit is configured to perform machine learning based on a relationship between the frame sequence of the acoustic features and the output symbol sequence, and then inputs the features output by the adapter unit to the model in the final layer of the encoder unit and adapt the model in the final layer to the target domain using a first loss that is a loss based on the output from the model in the final layer.

[0010] [2] In one aspect of the present invention, the speech recognition device further includes a pseudo-symbol string generation unit that generates the symbol string to be input to the adapter unit by performing at least one of repeating characters in the text or inserting zero or more blank symbols between characters in the text, based on the text belonging to the target region.

[0011] [3] In one aspect of the present invention, the speech recognition device further comprises a substitution processing unit that performs a process of replacing characters included in the text belonging to the target region with other characters in accordance with an output character probability distribution obtained based on predetermined statistical data, and the pseudo-symbol string generation unit generates the symbol string based on the text belonging to the target region after the substitution process output by the substitution processing unit.

[0012] [4] In another aspect of the present invention, in the above-mentioned speech recognition device, the pseudo-symbol string generation unit determines the number of repetitions of the same character when repeating characters in the text based on statistics regarding the number of repetitions of the same character in an intermediate symbol string, which is a symbol string calculated based on features output from the model of a predetermined layer in the encoder unit that has been trained based on the relationship between the frame string of the acoustic features and the output symbol string, and determines the number of consecutive blank symbols to be inserted between characters in the text based on statistics regarding the number of consecutive blank symbols in the intermediate symbol string.

[0013] [5] In addition, one aspect of the present invention is the above-mentioned speech recognition device, further comprising: a symbol string generation unit that generates a symbol string to be input to the adapter unit for training the adapter unit based on the intermediate features output from a model of a predetermined layer of the encoder unit based on the acoustic features; and an adapter unit loss calculation unit that calculates a second loss, which is a loss for the features output by the adapter unit based on the symbol string generated by the symbol string generation unit, and is configured to perform machine learning of the adapter unit using the second loss calculated by the adapter unit loss calculation unit.

[0014] [6] In another aspect of the present invention, in the speech recognition device described above, the adapter loss calculation unit calculates the second loss based on a loss based on the likelihood of a correct output symbol string given the features output from the adapter unit, and a loss representing the difference between the features output from the adapter unit and the features output from a model of a corresponding layer in the encoder unit.

[0015] [7] In accordance with another aspect of the present invention, the speech recognition device outputs a character string obtained by summarizing repeated characters in the output symbol string and deleting the blank symbols included in the output symbol string as a speech recognition result.

[0016] [8] In another aspect of the present invention, in the speech recognition device described above, when the encoder unit adapts the model of the final layer to the target domain using the first loss, the encoder unit also adapts the model based on a third loss obtained from a symbol string output by the multi-layer model of the encoder unit based on the frame sequence of acoustic features and the correct output symbol string corresponding to the frame sequence of acoustic features. This aspect uses a second re-training technique described later. According to this aspect, while adapting to the target domain (called the target domain), it is possible to prevent or at least suppress degradation of recognition performance in the domain where the encoder unit was originally trained (called the source domain).

[0017] [9] Also, one aspect of the present invention is a program for causing a computer to function as a speech recognition device, the program comprising: an encoder unit having models composed of multiple layers configured to input a sequence of frames of acoustic features and output an output symbol sequence consisting of a sequence of symbols corresponding to the frames of the acoustic features, each of the multiple layer models being configured to be capable of machine learning; and an adapter unit that inputs a symbol sequence belonging to a target domain and outputs features to be input to a model in a final layer of the encoder unit, wherein intermediate features are passed from a model in a certain layer of the multiple layer models of the encoder unit to a model in a next layer, and the encoder unit is configured to perform machine learning based on the relationship between the frame sequence of the acoustic features and the output symbol sequence, and then inputs the features output by the adapter unit to the model in the final layer of the encoder unit and adapt the model in the final layer to the target domain using a first loss that is a loss based on the output from the model in the final layer. [Effects of the Invention]

[0018] According to the present invention, it is possible to perform domain adaptation of a speech recognition model using only text data in a target domain. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a block diagram showing a schematic functional configuration of a speech recognition device according to an embodiment of the present invention; [Figure 2] 2 is a block diagram showing the functional configuration of the main part of a voice recognition function of the voice recognition device according to the embodiment. FIG. [Figure 3] FIG. 2 is a block diagram showing a functional configuration for learning an adapter unit of the speech recognition device according to the embodiment. [Figure 4] FIG. 2 is a block diagram showing a functional configuration for relearning an Encoder (third-layer encoder unit) using a first relearning technique in the speech recognition device according to the embodiment. [Figure 5] FIG. 10 is a block diagram showing a functional configuration for relearning an Encoder (third-layer encoder unit) using a second relearning method in the speech recognition device according to the embodiment. [Figure 6] 10 is a flowchart (1 / 2) showing the procedure of a process for converting a character string into a pseudo CTC symbol string in the embodiment. [Figure 7] 10 is a flowchart (2 / 2) showing the procedure of a process for converting a character string into a pseudo CTC symbol string in the embodiment. [Figure 8] FIG. 2 is a block diagram showing an example of the internal configuration of the voice recognition device according to the embodiment. [Figure 9] FIG. 13 is a block diagram showing the functional configuration of a part for performing replacement processing according to a fifth modified example in the speech recognition device of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] Next, an embodiment of the present invention will be described with reference to the drawings. In the following, technical matters on which the present embodiment is based will be described first, and then configurations specific to the present embodiment will be described in order.

[0021] An end-to-end speech recognition model that directly learns the relationship between speech and character strings without using phonetic symbols or other means is a model that converts an input acoustic feature sequence, such as a log-mel spectrogram of length T, into an output symbol sequence (character string) of length L. This input acoustic feature sequence x is expressed by the following equation (1). Also, the output symbol sequence y is expressed by the following equation (2).

[0022]

number

[0023]

number

[0024] In the above equation (1), D is the dimension of the acoustic feature x[t] for one frame. In the above equation (2), V is the set of output symbols (vocabulary).

[0025] Shallow Fusion (Non-Patent Document 2), a conventional technique, linearly interpolates the inference probability of a speech recognition model and the inference probability of a language model in the logarithmic domain during inference. The recognition result y (hat) obtained by the Shallow Fusion technique is expressed by the following equation (3):

[0026]

number

[0027] P in equation (3) AM (y|x) is the inference probability of the output symbol sequence of the speech recognition model (the probability of the output symbol sequence y given the input acoustic feature sequence x). LM (y) is the inference probability of the output symbol string of the language model (the probability of the output symbol string y). LM (y) can be trained using only text data. On the other hand, the speech recognition model P AM (y|x) is trained using speech-text pair data.

[0028] The feature of this embodiment is that a language model P LM (y) but speech recognition model P AM The speech recognition device of this embodiment has a configuration for learning (y|x). In other words, the speech recognition device of this embodiment adapts a speech recognition model to the topics contained in the learning text data.

[0029] Next, we will explain the outline of the CTC speech recognition model. The CTC speech recognition model itself is a method that belongs to existing technology. The CTC speech recognition model outputs one symbol for each frame of input features. The difference between the frame length of the features and the sequence length between output symbols is the blank symbol. <blank>This is absorbed by the fact that it is possible to insert a blank symbol and to make the output symbol (character) continuous. As an example, when associating the symbol string "weather" with the acoustic features of four frames, there are 17 possible alignments as shown below. <blank>is replaced with "_". Also, the separator between symbols is represented by a slash. 01: _ / _ / Weather / Ki 02: _ / Heaven / _ / Mind 03: _ / Weather / Ki / _ 04: Heaven / _ / _ / Mind 05: Heaven / _ / Air / _ 06: Weather / Air / _ / _ 07: _ / Heaven / Heaven / Ki 08: Heaven / _ / Heaven / Ki 09: Heaven / Heaven / _ / Ki 10: Heaven / Heaven / Ki / _ 11: _ / Heaven / Ki / Ki 12: Heaven / _ / Ki / Ki 13: Heaven / Ki / _ / Ki 14: Heaven / Ki / Ki / _ 15: Heaven / Ki / Ki / Ki 16: Heaven / Heaven / Ki / Ki 17: Heaven / Heaven / Heaven / Ki

[0030] The symbol string output by the CTC speech recognition model as an inference result is a blank symbol. <blank>In other words, each of the 17 alignments above is converted into the recognition result "weather". However, when the correct character is "ii" in "ii tenki" (good weather), for example, the correct recognition result contains a series of identical characters, and the alignment must have a blank symbol between the "ii" and "ii". <blank>must be inserted.

[0031] The training of the CTC speech recognition model is as follows. In the recognition process using the CTC speech recognition model, the probability of alignment a of the output symbol string is calculated for the feature frame string h extracted from the neural network. Here, the feature frame string h is expressed by the following formula (4). Also, the alignment a is expressed by the following formula (5). Note that T in formulas (4) and (5) h is the length of the sequence.

[0032]

number

[0033]

number

[0034] The alignment a of the output symbol string given the input feature frame string h is expressed by the following equation (6).

[0035]

number

[0036] The sum of the probabilities of the output symbol string y for all possible alignments is given by the following equation (7):

[0037]

number

[0038] β in Equation (7) -1 (y) is the set of all possible alignments for the output symbol sequence y and feature sequence h. The CTC loss function L is calculated by taking the logarithm of the above P(y|h) using the following equation (8).

[0039]

number

[0040] The CTC speech recognition model is trained by updating the values ​​of the internal parameters of the neural network so as to minimize the loss function value L mentioned above.

[0041] Each of the functions of the speech recognition device of this embodiment can be realized, for example, by a computer and a program. Furthermore, each of these functions has a storage means as necessary. The storage means is, for example, a variable in the program or a memory allocated by the execution of the program. Furthermore, non-volatile storage means such as a magnetic hard disk drive or a solid-state drive (SSD) may be used as necessary. Furthermore, at least some of the functions of each functional unit may be realized as a dedicated electronic circuit rather than a program.

[0042] The speech recognition device 1 of this embodiment is configured based on Intermediate CTC (Non-Patent Document 1). The outline of the speech recognition model in this embodiment is as follows. The speech recognition model of this embodiment is an improvement on the open source Espnet (https: / / github.com / espnet / espnet) provided on Github. In Intermediate CTC, the value of the CTC loss function is calculated even in the intermediate layers of the neural network, and the sum of the weighted values ​​for each layer is used as the final loss function to train the model.

[0043] 1 is a block diagram showing a schematic functional configuration of a speech recognition device 1 according to this embodiment. As shown in the figure, the speech recognition device 1 includes a speech supply unit 10, a convolutional neural network unit 20, and a first layer encoder unit 30. l 30) and the second layer encoder unit 40 (Encoder m 40) and the third layer encoder unit 50 (Encoder h 50), a target region text supply unit 110, a pseudo-CTC symbol string generation unit 120, an adapter unit 130, a CTC symbol string generation unit 210, a mean square error loss calculation unit 220, and a CTC loss calculation unit 230.

[0044] The speech supply unit 10 supplies speech data to be subjected to recognition processing by the speech recognition device 1. The speech supply unit 10 passes the speech data to the convolutional neural network unit 20. The speech supply unit 10 supplies, as speech data, for example, a sequence of acoustic features represented by a vector.

[0045] The convolutional neural network unit 20 performs convolution processing on the audio data passed from the audio supply unit 10. The convolutional neural network unit 20 passes the feature amount (h0 described later) obtained as a result of the convolution processing to the first-layer encoder unit 30.

[0046] The first-layer encoder unit 30, the second-layer encoder unit 40, and the third-layer encoder unit 50 form a three-layer encoder. Each of the first-layer encoder unit 30, the second-layer encoder unit 40, and the third-layer encoder unit 50 is implemented using a self-attention neural network. The speech recognition device 1 has a function (CTC loss calculation unit) for calculating losses for the outputs of each of the first-layer encoder unit 30, the second-layer encoder unit 40, and the third-layer encoder unit 50, but this function is not shown in this figure. The loss calculation function will be described in FIG. 2.

[0047] The first layer encoder unit 30, the second layer encoder unit 40, and the third layer encoder unit 50, which are configured with models in multiple layers, may be collectively referred to as the "encoder unit." The encoder unit has models in multiple layers so that it inputs a sequence of frames of acoustic features and outputs an output symbol sequence consisting of a sequence of symbols corresponding to the frames of the acoustic features, and each of the models in the multiple layers is configured to be capable of machine learning. Intermediate features (h l ya h m ) is passed to the encoder unit. After performing machine learning based on the relationship between the frame sequence of the acoustic features and the output symbol sequence, the encoder unit inputs the features output by the adapter unit to the final layer model (third-layer encoder unit 50) of the encoder unit, and adapts the final layer model (third-layer encoder unit 50) to the target domain using a first loss, which is a loss based on the output from the final layer model (third-layer encoder unit 50).

[0048] The first layer encoder unit 30 is an Encoder l 30, which is also called the feature value h based on the feature value h0 received from the convolutional neural network unit 20. l and passes it to the second layer encoder unit 40. The second layer encoder unit 40 m 40, which is also called the feature value h received from the first-layer encoder unit 30. l Based on this, the feature value h m and passes it to the third layer encoder unit 50. The third layer encoder unit 50 h 50, which is also called the feature value h received from the second-layer encoder unit 40. m Based on this, the feature value h h is calculated and output.

[0049] In addition, the features h0 and h l , h m , h h Each of is a sequence of vectors.

[0050] The target region text supply unit 110 is an Encoder h The target region text supply unit 110 supplies text data for the adaptation of the target region 50. h The target domain text supply unit 110 supplies the text of the target domain to the pseudo-CTC symbol string generation unit 120 so that the target domain text supply unit 110 outputs text that matches the target domain.

[0051] The pseudo-CTC symbol string generator 120 generates a pseudo-CTC symbol string based on the text string passed from the target region text supply unit 110. The pseudo-CTC symbol string generator 120 is also simply referred to as a "pseudo-symbol string generator." A method for generating a pseudo-CTC symbol string will be described in detail later with reference to a flowchart, etc. Multiple CTC symbol strings may correspond to one text string. That is, the pseudo-CTC symbol string generator 120 outputs multiple pseudo-CTC symbol strings corresponding to one text string. In other words, the pseudo-CTC symbol string generator 120 converts one text string into multiple pseudo-CTC symbol strings. The pseudo-CTC symbol string generator 120 can generate multiple pseudo-CTC symbol strings corresponding to each text string included in the text data supplied by the target region text supply unit 110.

[0052] The pseudo-CTC symbol string generation unit 120 generates a symbol string to be input to the adapter unit 130 by at least one of repeating characters in the text or inserting zero or more blank symbols between characters in the text based on text belonging to a target domain. The pseudo-CTC symbol string generation unit 120 may determine the number of times the same character is repeated when repeating characters in the text based on statistics regarding the number of times the same character is repeated in an intermediate symbol string, which is a symbol string calculated based on features output from the model of a predetermined layer in the encoder unit that has been trained based on the relationship between the frame sequence of the acoustic features and the output symbol string. The pseudo-CTC symbol string generation unit 120 may also determine the number of consecutive blank symbols to be inserted between characters in the text based on statistics regarding the number of consecutive blank symbols in the intermediate symbol string.

[0053] The adapter unit 130 is an Encoder h When adapting the pseudo-CTC symbol string 50, each pseudo-CTC symbol string passed from the pseudo-CTC symbol string generator 120 is h In other words, the adapter unit 130 inputs a symbol string (pseudo CTC symbol string) belonging to the target region, and generates a model (Encoder h It outputs the features to be input to the

[0054] The adaptor unit 130 is realized using a self-attention neural network and is configured to be able to learn. In other words, the adaptor unit 130 learns. When the adaptor unit 130 learns, the adaptor unit 130 uses the Encoder m The adaptor unit 130 generates features similar to those output by the encoder 40. m It is expected that the feature values ​​will approach those output by the adaptor unit 130. When the adaptor unit 130 trains, the parameters of the neural network inside the adaptor unit 130 are adjusted based on the losses calculated by the mean squared error loss calculation unit 220 and the CTC loss calculation unit 230, which will be described later. In other words, the adaptor unit 130 is configured to be able to perform machine learning using the "second loss," which will be described later.

[0055] When training the adaptor unit 130, the CTC symbol string generator 210 l Feature value h output from 30 l Based on this, alignment a l The CTC symbol string generation unit 210 calculates the calculated a l (hat) to the adaptor unit 130. In other words, the CTC symbol string generation unit 210 outputs a symbol string to be input to the adaptor unit 130 to train the adaptor unit 130 to a predetermined layer (Encoder l The CTC symbol string generator 210 is also simply called a "symbol string generator."

[0056] The mean square error loss calculation unit 220 and the CTC loss calculation unit 230 calculate the feature quantity h output from the adaptor unit 130 when training the adaptor unit 130. m (hat). In other words, mean squared error loss calculation unit 220 and CTC loss calculation unit 230 are each part of the function of the adapter unit loss calculation unit that calculates losses for training of adapter unit 130. Details of the losses calculated by mean squared error loss calculation unit 220 and CTC loss calculation unit 230 will be explained later. The losses calculated by mean squared error loss calculation unit 220 and CTC loss calculation unit 230 are used to update the parameters of the neural network inside adapter unit 130.

[0057] [Voice recognition function] 2 is a block diagram showing the functional configuration of the main part of the speech recognition function of the speech recognition device 1 according to this embodiment. As shown in the figure, the speech recognition device 1 includes a convolutional neural network unit 20, an Encoder l 30, a CTC loss calculation unit 32, and an Encoder m 40, a CTC loss calculation unit 42, and an Encoder h The CTC loss calculation unit 52 is configured to include a CTC loss calculation unit 50.

[0058] As shown in the figure, the speech recognition model consists of a convolutional neural network (convolutional neural network) 20, an encoder, l 30 (low level encoder), Encoder m 40 (Mid-level encoder), Encoder h It is configured to have four neural networks of 50 (high-level encoder). Each of these four neural networks consists of multiple layers. l 30 and Encoder m 40 and Encoder h 50 are self-attention networks. This speech recognition model uses the Encoder l 30, Encoder m 40, Encoder h From the 50 outputs, the CTC loss function value L l , L m , L h The feature value h0, which is the output of the convolution, is expressed by the following equations (9) and (10).

[0059]

number

[0060]

number

[0061] In addition, in equation (9), T h is the sequence length of the feature. h is the number of dimensions of each vector included in h0. Also, as shown in equation (10), h0 is the output of convolution when an acoustic feature x is input.

[0062] Encoder l Output from 30h l , Encoder m Output from 40h m , and Encoder h Output from 50h h are as shown in the following equation (11), and are calculated by equations (12), (13), and (14), respectively.

[0063]

number

[0064]

number

[0065]

number

[0066]

number

[0067] That is, the feature value h l takes feature h0 as input and l 30 is the output quantity. Also, the feature quantity h m is the feature value h l Encoder with m 40 is the output quantity. Also, the feature quantity h h is the feature value h m Encoder with h 50 is the output amount.

[0068] Encoder l 30, Encoder m 40, and Encoder h ·50 is the loss function value L for each output l , L m , and L h are calculated by the following equations (15), (16), and (17), respectively. l Feature value h output from 30 l Based on this, the loss L l The CTC loss calculation unit 42 calculates m Feature value h output from 40 m Based on this, the loss L m The CTC loss calculation unit 52 calculates the h Feature value h output from 50 h Based on this, the loss L h Calculate.

[0069]

number

[0070]

number

[0071]

number

[0072] The objective function in the base model is, for example, the loss function value L l and L m and L h It is calculated as the average of the following equation (18):

[0073]

number

[0074] That is, in the Intermediate CTC that is the premise of this embodiment, learning is performed by updating the internal parameters of the neural network so as to minimize Loss1 calculated by equation (18).

[0075] Next, a configuration for relearning a speech recognition model using text data on a target topic in the speech recognition device 1 of this embodiment will be described. In this embodiment, a text string is converted into a pseudo CTC symbol string, and the resulting CTC symbol string is converted into a feature frame by a neural network adapter (to be described later). In this embodiment, this feature frame is used to generate an Endocer of the intermediate CTC speech recognition model (the model on which this embodiment is based) described above. h The neural network is adapted to the target topic by retraining the

[0076] [Adapter learning] 3 is a block diagram showing a configuration for training the adapter unit 130 (also simply called "Adapter") of the speech recognition device 1 according to this embodiment. The Adapter is a device for training the pseudo-CTC symbol strings in the Encoder during re-training of the speech recognition model, which will be described later. h As shown in the figure, the speech recognition device 1 includes a convolutional neural network unit 20 and an encoder 50. l 30 and Encoder m 40 and Encoder h The system includes a processor 50, an adapter unit 130, a CTC symbol string generator 210, a mean square error loss calculator 220, and a CTC loss calculator 230.

[0077] The functions of the components shown in FIG. 3 have already been described. l Feature value h output from 30 l Based on the alignment a l (Hat). The adapter is aligned a l (hat) is input, and the estimated value of the feature value h m (hat). The adaptor unit 130 is trained based on the losses calculated by the mean square error loss calculation unit 220 and the CTC loss calculation unit 230. Through training, the adaptor unit 130 m Feature value h output by 40 m It is expected that the feature values ​​output will be close to

[0078] Here, the details of the training of the neural network Adapter will be explained. The speech recognition device 1 inputs the acoustic feature x to the trained Intermediate CTC speech recognition model (Fig. 3), and calculates the feature h by the following equations (19) and (20). m Calculate.

[0079]

number

[0080]

number

[0081] This h m The calculation process is the same as the calculations already explained in equations (10), (12), and (13). l The most likely alignment for a l (hat) is calculated using the following equation (21).

[0082]

number

[0083] The speech recognition device 1 detects that the CTC alignment sequence is m Output feature frame h m In other words, the speech recognition device 1 updates the internal parameters of the neural network of the adapter. l (hat) is input to the adapter, and the output is h m (hat) is calculated using the following equation (22).

[0084]

number

[0085] The speech recognition device 1 calculates h in equation (22). m (Hat) to h m Specifically, the speech recognition device 1 performs training of the adapter so that the loss function value L calculated by the following equation (23) m and the loss function value L calculated by equation (24) mse The internal parameters of the neural network of the Adapter are updated based on the above equation. Note that the CTC loss calculation unit 230 updates the internal parameters of the neural network of the Adapter based on the above equation (23). m Loss L based on the likelihood of the correct output sequence y given (hat) m Furthermore, the mean square error loss calculation unit 220 calculates h m (Hat) and Encoder m Loss L, which represents the difference between the feature value output from 40 (the model of the layer corresponding to the Adapter) and mse Calculate.

[0086]

number

[0087]

number

[0088] The loss function value L shown in Equation (24) mse is the mean square error loss. An example of the objective function used by the speech recognition device 1 when training the adapter is calculated by the following equation (25).

[0089]

number

[0090] In equation (25), α is a hyperparameter. The speech recognition device 1 updates the internal parameters of the neural network of the Adapter so as to minimize the loss Loss2 expressed in equation (25). This loss Loss2 may be referred to as the "second loss" for convenience. The speech recognition device 1 trains the Adapter so as to update the parameters of layers other than the final layer of the multi-layer neural network in the Adapter. When training the Adapter, the speech recognition device 1 uses the trained Encoder m The parameter values ​​of the final layer of 40 are directly used as the parameters of the final layer of the Adapter. m The parameters of the layers other than the final layer of the adapter are adjusted (updated) so that the output of 40 can be simulated. By training the adapter in this way, the adapter converts the CTC symbol string into the m Output feature frame h m It is expected to become a model for converting

[0091] Encoder h 50 Relearn] Next, Encoder h Explains relearning of 50. Encoder h In the retraining of 50, text strings containing the speech topic to be inferred are used as training data to improve the recognition accuracy of the speech topic to be inferred. In the following, the domain to which the speech topic trained for the Intermediate CTC speech recognition model and Adapter belongs is called the source domain, and the domain to which the speech topic to be inferred belongs is called the target domain. Encoder h As a method for relearning the parameter 50, either a first relearning method or a second relearning method, which will be described below, may be used. In this embodiment, either the first relearning method or the second relearning method can be selected as desired depending on the application, etc.

[0092] Encoder h 50 Retraining: First Retraining Method FIG. 4 shows a process in which the speech recognition device 1 according to the present embodiment uses the first re-learning method to re-train the Encoder h This is a block diagram showing the functional configuration for retraining the 50. In the first retraining method, domain adaptation is performed using only text from the target domain. h The re-learning of 50 is performed on the premise that the above-mentioned Adapter learning has been completed. h 4 includes a convolutional neural network unit 20, an encoder unit 130, a CTC loss calculation unit 50, a pseudo-CTC symbol string generation unit 120, and an adapter unit 130. l 30 and Encoder m 40 is also listed.

[0093] As shown in the figure, the pseudo CTC symbol string generator 120, the adapter 130, and the encoder h The configuration including the convolutional neural network unit 20 and the CTC loss calculation unit 52 may be called a model adaptation device 2. The model adaptation device 2 includes a convolutional neural network unit 20 and an Encoder l 30 and Encoder m 40 and Encoder h 50, an encoder (in this embodiment, Encoder h The feature of this embodiment is that the model adaptation device 2 does not require speech data of the target region, but operates based only on text data of the region.

[0094] 4 has already been described. That is, the trained adaptor unit 130 generates the encoder based on the pseudo-CTC symbol string output by the pseudo-CTC symbol string generator 120. m Feature value h that 40 will output m Outputs features close to h 50 calculates the feature quantity h based on the output from the adapter unit 130. h,i (Hat) is output. Encoder h The encoder 50 adjusts the parameters of the internal neural network based on the loss calculated by the CTC loss calculation unit 52. h The loss calculated by the CTC loss calculation unit 52 for the re-learning of 50 may be referred to as the "first loss" for convenience. h Encoder obtained by inputting 50 models h Losses based on output from 50.

[0095] Referring to Figure 4, the first retraining method is used to h We will explain the details of retraining 50. We explained how to train the adapter above, but here we will use the trained adapter to train the encoder. h Retrain 50 Encoder h In the retraining of 50, only text strings containing the speech topic to be recognized are used as training data. h ·50 Retraining uses training data of text strings on the target topic (domain).

[0096] Encoder using the first retraining method h In the re-training of 50, the speech recognition device 1 first generates N pseudo CTC symbol strings based on the text strings of the target domain, which are the training data, using a pseudo CTC symbol string conversion method described later. These N pseudo CTC symbol strings are generated by a i (Hat).

[0097]

number

[0098] The CTC loss calculation unit 52 of the speech recognition device 1 calculates the above i (hat), the loss function value L is calculated by the following equations (27) and (28). h,i Calculate.

[0099]

number

[0100]

number

[0101] That is, the i-th loss function value (i=1, ,N) is L h,i is the i-th CTC symbol string a i The feature output by the Adapter is input as a h ·50 obtained by inputting h h,i (Hat). The first retraining method is used to h The objective function for adaptive learning of 50 is calculated as the average of N loss function values ​​using equation (29) below.

[0102]

number

[0103] Encoder h In the re-training process, the speech recognition device 1 updates the internal parameters of the neural network so as to minimize the value of Loss3 expressed by equation (29). h When re-learning 50, the speech recognition device 1 fixes the parameters of the learned adapter. h During the 50 retraining steps, the internal parameters of the Adapter are not updated, and the Encoder h -Update 50 internal parameters.

[0104] When you have completed the above retraining, the Encoder h The 50 neural networks are expected to adapt to the topics of the text strings used for retraining.

[0105] Encoder h 50 Retraining: Second Retraining Method FIG. 5 shows a process in which the speech recognition device 1 according to the present embodiment uses the first re-learning method to re-train the Encoder h 5 is a block diagram showing a functional configuration for re-learning the speech recognition device 1. As shown in the figure, the speech recognition device 1 includes an Encoder h 50, a CTC loss calculation unit 52, a pseudo-CTC symbol string generation unit 120, an adapter unit 130, a convolutional neural network unit 20, and an Encoder l 30 and Encoder m 40. Also, here, Encoder h 50, a CTC loss calculation unit 52, a pseudo-CTC symbol string generation unit 120, an adapter unit 130, a convolutional neural network unit 20, and an Encoder l 30 and Encoder m The configuration including the encoder 40 functions as a model adaptation device 2 that executes processing by the second re-learning method. h ·50 retraining is performed using domain adaptation, which uses not only the target domain text but also the source domain speech-text pair. When using the second retraining method, it is expected to be effective in reducing the degradation of speech recognition accuracy in the source domain.

[0106] Encoder using the second retraining method h In the re-training of 50, the speech recognition device 1 uses a pseudo-CTC symbol string conversion method described later based on the text string of the target domain, which is the training data, to convert N t Generate pseudo CTC symbol strings. t The pseudo CTC symbol strings are a t,i (Hat).

[0107]

number

[0108] The CTC loss calculation unit 52 of the speech recognition device 1 calculates the above t,i (hat), the loss function value L is calculated by the following equations (31) and (32). t,i Calculate.

[0109]

number

[0110]

number

[0111] In addition, we have applied N to the trained Intermediate CTC speech recognition model. s By inputting a sequence of acoustic features, the loss function value L s,j The sequence of acoustic features is expressed by the following equation (33). Also, the loss function value L s,j is calculated by equations (34), (35), (36), and (37).

[0112]

number

[0113]

number

[0114]

number

[0115]

number

[0116]

number

[0117] That is, the convolutional neural network unit 20 generates a sequence of acoustic features x j Convolution(x j ) is output. l 30 is Convolution(x j ) based on h sl,j Outputs the Encoder m 40 is h sl,j Based on this, h sm,j Outputs the Encoder h 50 is h sm,j Based on this, h sh,j Output.

[0118] The objective function in the second relearning method is the Loss function expressed in the following equation (38): st is.

[0119]

number

[0120] In other words, Loss st is N t Loss function values ​​L t,i For the average of N s Loss function values ​​L s,j The average of Loss is added as a regularization term to suppress the deterioration of recognition accuracy in the source domain. st By updating the model parameters to minimize h Re-learning is performed for 50. However, at this time, the adapter parameters are not learned and are fixed. Encoder after re-learning is completed h ·50 is a neural network that adapts to the topic of text strings without compromising its processing power on the source domain speech.

[0121] The loss calculated by the above formula (37) is conveniently called the "third loss." In other words, the third loss is a loss obtained (calculated) from the symbol string output by the model consisting of multiple layers (in this embodiment, the first-layer encoder unit 30, the second-layer encoder unit 40, and the third-layer encoder unit 50) of the encoder unit based on the frame sequence of acoustic features, and the correct output symbol string corresponding to the frame sequence of acoustic features. In other words, in the second re-training method described here, when the encoder unit uses the first loss to adapt the model of the final layer to the target domain, it also performs adaptation based on the above third loss. That is, the loss Loss expressed in formula (38) st The second re-training method makes it possible to prevent or at least suppress degradation of recognition performance in the domain where the encoder was originally trained (the source domain) while adapting to the target domain.

[0122] When comparing the first and second relearning methods described above, the first relearning method is expected to improve speech recognition accuracy in the target domain more than the second relearning method. However, when the first relearning method is used, speech recognition accuracy in the source domain deteriorates. When the second relearning method is used, the improvement in speech recognition accuracy in the target domain is smaller, but the effect of reducing the deterioration in speech recognition accuracy in the source domain can be expected.

[0123] [Generation of pseudo-CTC symbol strings] Next, we will explain how to convert a text string into a pseudo-CTC symbol string. The pseudo-CTC symbol string is generated by the Encoder h This was used for re-learning 50. The speech recognition device 1 generates a set of pseudo CTC symbol strings based on character strings included in the text data for training.

[0124] Consider the case where a text string is converted into a pseudo-CTC symbol string. For example, if the length of the CTC symbol string is limited to 6, there are nine possible CTC symbol strings that correspond to the string "nice weather": 1: Good weather 2: Good weather 3: Good weather 4: Good weather 5: Good weather 6: Good weather 7: Good weather 8: Good / _ / Good / Heaven / Heaven / Ki 9: Good / _ / Good / Weather / Mind / Mind

[0125] In the above pseudo-CTC string, "_" is a blank symbol. <blank>In a CTC string, a blank symbol is always inserted where the same symbol appears consecutively, such as "ii". <blank>is inserted between them. Note that the separator between symbols is represented by a slash.

[0126] blank symbol <blank>The position of can be considered to be a position (time period) in the feature frame where no pronunciation is present. As shown in the example above, there are many CTC symbol strings corresponding to a certain string (here, "good weather"). In other words, training a neural network using all CTC symbol string patterns corresponding to a certain string can require an enormous amount of time. On the other hand, there are also symbol strings that are possible but unlikely to occur in reality among CTC symbol strings. For example, assuming that a CTC symbol string of length 10 is generated for the string "good weather," it is unlikely that a feature frame with a long period without pronunciation between "i" and "i," such as "i / _ / _ / _ / _ / _ / _ / i / ten / ki," or a feature frame with a repetition of a specific symbol (pronunciation interval) longer than a predetermined length, such as "i / _ / i / ten / ki / ki / ki / ki / ki / ki / ki," will occur in reality. Therefore, when converting a text string into a CTC symbol string, it is possible to limit the symbol strings to only those with patterns that are considered to occur in reality.

[0127] Therefore, the speech recognition device 1 of this embodiment uses the trained Intermediate CTC Encoder l The feature frame h output from l a is a CTC symbol string calculated based on i In the (hat) part, the speech recognition device 1 converts the CTC symbol string into a pseudo-CTC symbol string that is likely to occur based on statistics. <blank>The data is converted into a pseudo-CTC symbol string according to a probability density function generated from the statistics of consecutive numbers.

[0128] To generate pseudo-CTC symbol strings, we use the speech data used for training the Intermediate CTC. We input the speech data to the trained Intermediate CTC and run the Encoder. l a output from l (Hat) based statistics. Specifically, a l The number of times the same symbol (excluding blank symbols) appears consecutively in (hat) is N c (n) and a blank symbol <blank>The number of times that appears consecutively is N b Based on the data of these times, the probability density function P c (n), and a blank symbol <blank>Probability density function P of consecutive numbers b (n) are calculated using the following equations (39) and (40), respectively. That is, in this embodiment, a probability density function is generated based on statistics regarding the symbol strings actually output from the encoder based on speech.

[0129]

number

[0130]

number

[0131] An example of calculation for finding the probability density function is as follows. Assume that the speech data contains two linguistic expressions, "nice weather" and "tomorrow." For the speeches "nice weather" and "tomorrow," Encoder l are respectively "_ / _ / good / _ / good / _ / good / _ / weather / _ / _ / _" and "_ / light / light / _ / _ / _ / day / day / _", the number of consecutive symbols is as follows. <blank>is represented as "_".

[0132] The number of occurrences of each consecutive symbol (character) other than the blank symbol is as follows: N c (0)=0 N c (1)=3: (applies to "i", "ten", and "ki") N c (2)=2: (Morning / Morning, Sunday / Sunday) N c (3)=1: (This corresponds to "i / i / i")

[0133] Therefore, the probability P c (n) is as follows: P c (0)=0 / 6=0 P c (1)=3 / 6=1 / 2 P c (2)=2 / 6=1 / 3 P c (3)=1 / 6

[0134] On the other hand, a blank symbol <blank>The number of occurrences for each consecutive number is as follows: N b (0)=1: (There is no "_" between "heaven" and "ki", so N b (Count 0) N b (1)=4: (applies between "i" and "i", between "i" and "ten", before "akari" (light), after "hi" (day)) N b (2)=1: (The character before "i" applies) N b (3)=2: (After "ki" (spirit)), between "myo" (light) and "hi" (day)

[0135] Therefore, the probability P b (n) is as follows: P b (0)=1 / 8 P b (1)=4 / 8=1 / 2 P b (2)=1 / 8 P b (3)=2 / 8=1 / 4

[0136] The above example is a probability value calculated based on two pieces of speech data, "nice weather" and "tomorrow." However, the actual statistics are created based on all the speech data used in training Intermediate CTC. This allows the probability of consecutively appearing symbols (other than blank symbols, or blank symbols) in the CTC symbol string corresponding to the speech data used for training to be calculated. <blank>The probability (trend) regarding consecutive numbers of

[0137] 6 and 7 are flowcharts showing the procedure for the process of conversion into a pseudo-CTC symbol string. Figures 6 and 7 are connected by a connector, and these two figures form a single flowchart. The process of this flowchart is executed by the pseudo-CTC symbol string generator 120. Below, the procedure for the process of generating a CTC symbol string will be explained with reference to this flowchart.

[0138] In the process of this flowchart, the text string c is converted into the CTC symbol string a i By repeating the process of this flowchart, multiple (N) CTC symbol strings a i (hat) (where i = 1, , N), the text string c is expressed as in the following formula (41). In formula (41), V is the set of output symbols (i.e., characters) that do not contain blank symbols, and J is the length (number of characters) of the string c.

[0139]

number

[0140] First, in step S1 of FIG. 6, the pseudo-CTC symbol string generator 120 i (Hat) is set (initialized) to the null string.

[0141] The next step S2 is the start point of loop 1. Here, the pseudo-CTC symbol string generation unit 120 controls the conditions for loop 1. In this flowchart, j is a variable that serves as an index for the repetition in loop 1. The initial condition for loop 1 is j=1. The condition when loop 1 is repeated is "+1" (i.e., j=j+1). The condition for determining the end of loop 1 is "J" (the loop is exited when j is such that the condition j≦J is no longer satisfied). In other words, the processing of loop 1 is repeated while incrementing j by 1 from when j=1 to when j=J. The end point of loop 1 is step S10.

[0142] Next, in step S3, the pseudo CTC symbol string generator 120 b According to (n), the blank symbol <blank>The number of consecutive b That is, the pseudo CTC symbol string generator 120 selects the consecutive number N based on the previously obtained statistics and according to random elements. b In the process of step S3, unless the probability is zero, N b = 0, 1, 2,...

[0143] Next, in step S4, the pseudo-CTC symbol string generation unit 120 determines whether the following conditions are true or false for the current character position j: (j ≠ 1) and (c[j-1] == c[j]). Note that if j = 1, the condition is confirmed to be false, so c[j-1] may not be referenced. Alternatively, it may be assumed that c[0] is any arbitrary symbol. If the above condition is true (step S4: YES), the process proceeds to step S5. If the above condition is false (step S4: NO), the process jumps to step S8 in FIG. 7. The branching based on the condition determination in step S4 is intended to execute the processes from steps S5 to S7 only for locations where the same character appears consecutively in the input string.

[0144] Next, when the process proceeds to step S5, the following occurs. That is, step S5 is the starting point of loop 2. Here, the pseudo-CTC symbol string generation unit 120 controls the conditions related to loop 2. The condition for executing the processing inside loop 2 is (N b =0). N b The initial value of N determined in step S3 b If N is not 0, the process inside loop 2 is never executed. b If the initial value of is 0, then N b While the value of is 0, repeat the process of loop 2. b When the value of is no longer 0, the process exits from Loop 2. The end point of Loop 2 is step S7.

[0145] Step S6 is the only step inside loop 2. In step S6, the pseudo-CTC symbol string generator 120 b According to (n), the blank symbol <blank>The number of consecutive b Select N b Each time we make a choice, we make a choice based on a different probability factor. b While repeating the selection of N b It is expected that a value other than 0 will be selected as

[0146] Step S7 is the end point of Loop 2. b If the condition (N = 0) is true, the process of loop 2 is repeated. b If the condition (N = 0) is false, the process exits from Loop 2 and proceeds to step S8 in FIG. 7. In other words, when the process exits from Loop 2, N b A value other than 0 is selected as

[0147] In step S8 of FIG. 7, the pseudo-CTC symbol string generator 120 uses the already determined N b Based on the value of N b blank symbols <blank>the symbol string a at that time i It is connected to the back (usually the right side) of the hat.

[0148] Next, in step S9, the pseudo CTC symbol string generator 120 calculates the P c According to (n), the number of consecutive characters (symbols other than blank symbols) is probabilistically N c That is, the pseudo CTC symbol string generator 120 selects the consecutive number N based on the previously obtained statistics and according to random elements. c In other words, unless the probability is zero, choose N c = 0, 1, 2, .... Next, the pseudo CTC symbol string generator 120 generates a pseudo CTC symbol string c pieces(N c is the value selected in this step) to the character c[j] of the current symbol string a i It is connected to the back (usually the right side) of the hat.

[0149] Step S10 is the end point of Loop 1. As per the above condition, the value of j is incremented by 1 (incremented by 1), and the processing of Loop 1 is repeated as long as the resulting value of j is 1, 2, . . . , J. When the condition for repetition is no longer met, that is, when incrementing J causes the value of j to reach J+1, Loop 1 is exited and the process proceeds to the next step S11. In other words, when Loop 1 is exited, processing of characters c[1] to c[J] in the source string is completed.

[0150] Next, in step S11, the pseudo CTC symbol string generator 120 generates a pseudo CTC symbol string P b According to (n), the blank symbol <blank>The number of consecutive b That is, the pseudo CTC symbol string generator 120 selects the consecutive number N based on the previously obtained statistics and according to random elements. b In other words, unless the probability is zero, choose N b = 0, 1, 2, .... Next, the pseudo CTC symbol string generator 120 generates a pseudo CTC symbol string b pieces(N b is the value selected in this step) <blank>the symbol string a at that time i Connect to the end of (hat).

[0151] Among the processes shown in this flowchart, the process of step S1 can be regarded as an initialization process. The processes from steps S3 to S8 can be regarded as the first phase of the process. The first phase of the process is <blank>Determine the number of consecutive blank symbols <blank>The processing in step S9 can be considered as the second phase processing. The second phase processing determines the number of consecutive characters (other than blank symbols) and concatenates that number of characters as part of the string to be generated. The processing in the first and second phases is repeated for each character (starting from the beginning) contained in the source text string (loop 1 processing controlled by steps S2 and S10). The processing in step S11 can be considered as the third phase processing. The third phase processing is performed after the above loop 1 processing, i.e., for concatenating the blank symbols following the last character of the input string. <blank>Determine the number of consecutive blank symbols <blank>as part of the string to be generated.

[0152] Here, the processing of the above flowchart will be explained using an actual example. As an example, assume that the source text string is "Tomorrow will be a nice day."

[0153] <First phase processing> In the process of step S3 of the above-mentioned flowchart, the calculated P b According to (n), the blank symbol <blank>N is the number of consecutive b Here, the selected N b Assume that N is 3. b The probability that =3 is selected is P b (3)=1 / 4. In step S8 of the above-mentioned flowchart, i (Hat) with three blank symbols <blank>When concatenated, a i (hat) = "_ / _ / _". Here, the slashes are separators between the symbols inserted for convenience.

[0154] <Second Phase Processing> As the processing in step S9 of the above flowchart, according to the calculated P c (n), probabilistically select N which is the number of consecutive non-blank symbols (i.e., characters). c Select N. Assume that the selected N c is 2. Here, the probability that N c = 2 is selected is P c (2) = 1 / 3 in the above example. Then, concatenate N c pieces (i.e., 2 pieces here) of the consecutive characters "明" to a i (hat). Then, a i (hat) = "_ / _ / _ / 明 / 明". The slashes are separators between the symbols inserted for convenience.

[0155] <Repeating the First Phase and the Second Phase> For each of the remaining characters "日", "は", "い", "い", "天", "気" in the input string, the above processing of the first phase and the second phase is repeated.

[0156] <Third Phase Processing> After the above repeating process ends, by performing the third phase processing, the last consecutive blank symbols are added to the CTC symbol sequence. <blank>is added.

[0157] If the same character is repeated in the input string (such as "ii" in "tomorrow is a good weather"), a blank symbol is inserted between the characters ("ii" and "ii"). <blank>The number N b does not become 0. This is guaranteed by the processing from steps S5 to S7 in the flowchart.

[0158] In this way, based on the character string "Tomorrow will be a good day", a CTC symbol string such as "_ / _ / _ / Morning / Morning / _ / Day / Day / _ / _ / Yes / _ / Yes / _ / Heaven / Heaven / _ / _" is output. i Each of the (hat) is a random element (a consecutive number of letters or a blank symbol) <blank>Since the symbol strings are generated based on the number of consecutive symbols, it is entirely possible to expect that the symbol strings will be different from each other, except for pairs that happen to match by chance.

[0159] [Processing during inference] Next, the inference process of the speech recognition device 1 after the model learning is completed will be described. Note that, as a prerequisite for executing the inference, the Encoder 1 uses the text data of the target region. h Re-learning has also been completed.

[0160] For the input speech x to be recognized, the speech recognition device 1 calculates the output feature h h Furthermore, the speech recognition device 1 calculates the feature quantity h h Based on this, the CTC symbol string a (hat) is calculated.

[0161]

number

[0162]

number

[0163] Then, the speech recognition device 1, like the conventional CTC speech recognition method, combines the succession of identical characters contained in the obtained CTC symbol string a (hat) into one character, and also removes blank symbols. <blank>In other words, the speech recognition device 1 aggregates repeated characters in the output symbol string (a (hat)) and deletes blank symbols contained in the output symbol string, thereby outputting the character string obtained as the speech recognition result.

[0164] FIG. 8 is a block diagram showing an example of the internal configuration of the speech recognition device 1 of this embodiment. The speech recognition device 1 can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions contained in a program read from the RAM 902 or the like. In accordance with each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices. The input / output devices 904 and 905 are input / output devices. Input / output devices 904 and 905 exchange data with the central processing unit 901 via an input / output port 903. A bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from and to RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906.

[0165] At least some of the functions of the speech recognition device 1 in the above-described embodiment can be implemented by a computer. In this case, a program for implementing the functions may be recorded on a computer-readable recording medium and then loaded and executed by a computer system. The term "computer system" as used herein includes hardware such as an operating system and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memory, as well as storage devices such as hard disks built into computer systems. In other words, a "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, the term "computer-readable recording medium" may also include media that temporarily and dynamically store programs, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or media that store programs for a certain period of time, such as volatile memory within a computer system that serves as a server or client. The program may be a program that implements some of the above-described functions, or it may be a program that can be implemented in combination with a program already stored in the computer system.

[0166] In addition, in training the neural network (updating the parameters), the backpropagation method based on the calculated loss can be used.

[0167] Although the embodiments have been described above, the present invention can also be implemented in the following modified examples. Note that multiple embodiments and modified examples may be implemented in combination to the extent that they can be combined.

[0168] [Variation 1] In the above embodiment, the encoder is divided into three layers (first layer encoder unit 30). l 30) and the second layer encoder unit 40 (Encoder m 40) and the third layer encoder unit 50 (Encoder h This was achieved using the 50) method. The number of encoder layers may be set to a different number. For example, the number of encoder layers may be set to four or more. Even in such cases, the encoder should be able to adapt based on the text in the target domain at least for the layers including the final layer (highest level).

[0169] [Variation 2] Instead of implementing the entire speech recognition device 1 shown in Fig. 1, only the model adaptation device 2 shown in Fig. 4 may be implemented. In this case, the model adaptation device 2 can still adapt the speech recognition model. In other words, the model adaptation device 2 can adapt the speech recognition model based on the text in the target domain so that it outputs a character string as a speech recognition result that matches the target domain.

[0170] [Variation 3] Different Encoders for each topic (field, area) h You can also retrain the 50 beforehand. In that case, you can select an Encoder that matches the topic of the speech recognition target. h By selecting 50 each time, the speech recognition result string that matches the topic will be output. Conversely, by using the text of the target topic each time, h You may also study the following.

[0171] [Variation 4] In the above embodiment, the Encoder h In order to re-learn the 50, the pseudo CTC symbol string generator 120 generates a pseudo CTC symbol string. As a modified example, instead of providing the pseudo CTC symbol string generator 120, a symbol string belonging to the target region (corresponding to a pseudo CTC symbol string) may be provided from outside. In this case as well, the Adapter performs the Encoder h Calculate the features to be input to 50 and output them.

[0172] [Variation 5] As a fifth modification, before converting the text string into a pseudo-CTC symbol string, a character substitution process may be performed in the text string. By performing such a substitution process, the text string is converted into a pseudo-CTC symbol string by the Encoder. l This has the effect of resembling the error pattern of the string output by the Encoder 30. Then, based on the text string after such replacement processing, a pseudo CTC symbol string is generated. In other words, based on the CTC symbol string corresponding to the string that may contain such an error, the Adapter h 50 retraining can be performed, which is expected to further improve recognition performance.

[0173] 9 is a block diagram showing the functional configuration of a portion for the replacement processing according to Modification 5 in the speech recognition device 1. As shown in the figure, the speech recognition device 1 (model adaptation device 2) according to Modification 5 includes a replacement processing unit 115. The replacement processing unit 115 performs processing for replacing characters in text data (character strings) supplied by the target area text supply unit 110. The replacement processing unit 115 passes the text data (character strings) after replacement to the pseudo-CTC symbol string generation unit 120. In Modification 5, the pseudo-CTC symbol string generation unit 120 generates a pseudo-CTC symbol string based on the text string passed from the target area text supply unit 110, using the character string after the replacement processing unit 115 has performed character replacement.

[0174] The method of calculating the output character probability distribution used in the fifth modification is as follows: The speech data used for training the Intermediate CTC is input to the trained Intermediate CTC. l 30 outputs the highest probability alignment a l (hat) is expressed by the following equation (44).

[0175]

number

[0176] This a l Consider the case where the following equation (45) is satisfied at the position t=t' (hat).

[0177]

number

[0178] In the case of the above equation (45), at the position t=t', Encoder l The probability that 30 outputs each symbol v (v is an element of V) is as shown in equation (46) below. In other words, the probability expressed by equation (46) is the probability that symbol s is replaced by symbol v, and does not depend on t.

[0179]

number

[0180] For each t in each training data, if the symbol at that position is s (s is an element of V as described above), calculate the probability of the above equation (46), and average the calculated probabilities to obtain the output character probability distribution P s Once this output character probability distribution is found, the value expressed by the following equation (47) is the value of the Encoder l 30 can be considered to be the probability of accidentally outputting another character s'.

[0181]

number

[0182] For every s (where s is an element of V), the output character probability distribution P s (v) can be calculated and used in the text string replacement process.

[0183] The text string replacement process in this variation 5 is performed as follows: For each character c[j] included in the text string c, character replacement is performed according to the output character probability distribution (obtained above) expressed by the following equation (48).

[0184]

number

[0185] For example, if the text string c is "weather forecast," then c[1] = "Ten." Furthermore, the character substitution is as shown in the following formulas (49) and (50).

[0186]

number

number

[0187] In this example, the substitution process from "ten" to "ten" is performed with a probability of 0.8. In other words, the character "ten" does not change, and the original string c does not change as a result of this substitution. On the other hand, the substitution process from "ten" to "dot" is performed with a probability of 0.1. In other words, since "ten" is replaced with "dot", the original string c changes to "Tenki Yoho Desu" (This is a weather forecast). By performing the above substitution process on all c[j] contained in the string c, the Encoder l A string intended to resemble the error in the string output by Encoder 30 can be obtained. Then, using the CTC symbol string obtained based on this substitution process, Encoder h ·50 re-learnings can be performed.

[0188] That is, in the fifth modification, the replacement processing unit 115 performs a process of replacing characters included in the text belonging to the target area (the text supplied from the target area text supply unit 110) with other characters according to an output character probability distribution obtained based on predetermined statistical data. Also, the pseudo-CTC symbol string generation unit 120 performs a process of generating a CTC symbol string based on the text belonging to the target area after the replacement process output by the replacement processing unit 115.

[0189] The above has described in detail an embodiment of the present invention (including modified examples) with reference to the drawings, but the specific configuration is not limited to this embodiment, and also includes designs within the scope that do not deviate from the gist of the present invention.

[0190] As described above, in this embodiment (including the modified examples), an end-to-end speech recognition model can be adapted using only text data. In other words, it is possible to domain-adapt a speech recognition model using only text data (without speech data) and improve recognition accuracy in a target domain.

[0191] [Verifying the effect of the configuration] A demonstration experiment was conducted to verify the effectiveness of the speech recognition device 1 according to the embodiment described above. The first speech used in the demonstration experiment was the CSJ corpus, which records speeches presented at conferences on phonetics. Using this CSJ ​​corpus, training of the Intermediate CTC and training of the Adapter were conducted. After that, the Encoder of the Intermediate CTC that had been trained above was used. h Using the weather forecast text data, we performed relearning using the first and second relearning methods. h Using the above, we calculated the character error rate (CER) for the weather forecast evaluation set (the second speech) and the CSJ corpus evaluation set. We also calculated the character error rate with and without the text string replacement process (the process described in Variation 5).

[0192] The experimental specifications common to both the first and second relearning methods are as follows: Feature quantity: 80 dimensions of log-mel spectrogram + 3 dimensions of pitch + Δ + ΔΔ, total 249 dimensions Character types: 3260 types

[0193] The model used is as follows: Convolution: Two-layer convolutional neural network Encoder l : Conformer 6 layers Encoder m : Conformer 3 layers Encoder h : Conformer 3 layers Adapter: Conformer 6 layers Number of pseudo-CTC symbol strings to generate: N: 64

[0194] Below is the Encoder h These are the individual experimental specifications and results for the first and second relearning methods for relearning 50.

[0195] The training data for the first retraining method (domain adaptation using only text from the target domain) is as follows: 1) Intermediate CTC Study: CSJ Corpus 2) Adapter learning: CSJ corpus 3) Encoder h Relearning: 1000 Weather Forecast Texts

[0196] The evaluation data for the first relearning method is as follows. 1) eval1, eval2, and eval3 from the CSJ evaluation set 2) Weather forecast evaluation audio for 2 hours, but Encoder h It does not include audio of the 1,000 weather forecast text sentences used for relearning.

[0197] The learning parameters for the first relearning method are as follows: 1) Intermediate CTC training: Use the model trained for the epoch with the lowest validation loss out of 100 epochs. 2) Adapter training: Use the model trained for the epoch with the lowest validation loss out of 50 epochs. 3) Encoder h Retraining: Use the model trained on the epoch that showed the greatest improvement in CER (Character Error Rate) out of 20 epochs. The number of pseudo-CTC character strings N is 64.

[0198] Table 1 below shows the Encoder h 1 shows the comparison results of CER (character error rate) before and after relearning (in the case of the first relearning method).

[0199] [Table 1]

[0200] As shown in Table 1 above, for weather forecast evaluation audio, if there is no text string replacement process, CER uses the Encoder h The CER was improved by 3.2% from 18.7% before re-learning to 15.5%. When text string replacement processing is performed (in the case of Variation 5), the CER was h The retraining rate for each of the eval1, eval2, and eval3 speeches in the CSJ corpus improved by 5.0%, from 18.7% before retraining to 13.7%. h The CER worsened due to the retraining of the Encoder. h The reason for the deterioration of recognition accuracy after retraining is the Encoder h However, this is because the topics were re-adapted from the original CSJ corpus topics to the topic of weather forecasting.

[0201] A specific example of an improvement to the weather forecast evaluation voice (provided that a text string replacement process is performed) is as follows. Evaluation voice utterance: "This is the predicted minimum temperature." Encoder h Recognition result before relearning: "This is the predicted optimal fundamental tone." Encoder h Recognition result after retraining: "This is the predicted minimum temperature."

[0202] As described above, in the demonstration experiment described here, we used only the weather forecast text data. h By retraining the system, we were able to demonstrate that it is now possible to output symbol strings appropriate for the topic of weather forecasts during inference.

[0203] The training data for the second retraining method (domain adaptation using text in the target domain and a speech-text pair in the source domain) is as follows: 1) Intermediate CTC Study: CSJ Corpus 2) Adapter learning: CSJ corpus 3) Encoder h Retraining: 1000 weather forecast texts and the CSJ corpus

[0204] The evaluation data for the second relearning method is as follows. 1) eval2 of the CSJ evaluation set 2) Weather forecast evaluation audio for 2 hours, but Encoder h It does not include audio of the 1,000 weather forecast text sentences used for relearning.

[0205] The learning parameters for the second relearning method are as follows: 1) Intermediate CTC training: Use the model trained for the epoch with the lowest validation loss out of 100 epochs. 2) Adapter training: Use the model trained for the epoch with the lowest validation loss out of 50 epochs. 3) Encoder h Retraining: Use the model trained for the epoch that showed the greatest improvement in CER among 20 epochs (Encoder in the second retraining method in Table 2 below). h (after relearning) However, regarding epochs, one epoch was counted when the weather forecast text had been learned once. Number of pseudo CTC strings N t is 64. The number of CSJ sounds to be learned simultaneously with the pseudo-CTC strings, N s is 64.

[0206] Table 2 below shows the Encoder h The following shows the comparison results of CER before and after relearning (in the case of the second relearning method). Note that the results after relearning shown in Table 2 are for the case where the text string replacement process was performed (transformation method 5).

[0207] [Table 2]

[0208] In the results shown in Table 2, Encoder h Encoder before retraining and with the second retraining method h The CER for the source domain, the CSJ evaluation set (CSJ eval2), is 5.2% in both cases, before and after retraining with the second retraining method. In other words, the recognition accuracy in the source domain is not degraded by retraining with the second retraining method. On the other hand, the CER for the target domain, the weather forecast evaluation set (weather forecast), is improved from 18.7% before retraining to 15.3% after retraining with the second retraining method.

[0209] In Table 2, the Encoder in the first retraining method h The CERs after relearning (4 epochs and 5 epochs) are used for comparison. This is because stopping relearning using the first relearning method, which uses only text data as training data, midway is a possible way to improve speech recognition accuracy in the target domain without degrading speech recognition accuracy in the source domain. In other words, when using the first relearning method, a CER equivalent to the CER (15.3%) for the weather forecast evaluation set achieved using the second relearning method is achieved after 4 epochs (CER is 15.5%) or 5 epochs (CER is 15.1%). After 4 epochs of relearning using the first relearning method, the CER for the CSJ evaluation set (CSJ eval2) is 5.8%. Furthermore, after 5 epochs of relearning using the first relearning method, the CER for the CSJ evaluation set (CSJ eval2) is 6.1%. In other words, for both 4 and 5 epochs, the CER on the CSJ evaluation set (CSJ eval2) deteriorated from 5.2% before relearning. In other words, we were able to confirm that the second relearning method is more effective than the first relearning method for comparison in terms of not degrading source domain speech recognition accuracy.

[0210] As described above, the first re-learning method and the second re-learning method can be used appropriately. Furthermore, it has been shown that the re-learning method according to this embodiment is effective when either the first re-learning method or the second re-learning method is used. In other words, assume a case where it is desired to improve the recognition accuracy of a pre-trained general-purpose speech recognition model for speech on a topic that is to be recognized. For example, assume that for a topic about a sport that is just starting to become popular, no speech-text pairs are available, and only the text is available. In such a situation, the adaptive learning (Encoder) of this embodiment can be performed using text on the topic of the target sport. h Recognition performance can be improved by retraining the model. [Industrial Applicability]

[0211] The present invention can be used, for example, for speech recognition processing and model adaptation processing for speech recognition. Such an invention can be used in a wide range of industries. However, the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0212] 1. Voice recognition device 2. Model Adaptation Device 10 Audio supply unit 20 Convolutional Neural Networks 30 1st layer encoder l ) 32 CTC loss calculation section 40 Second layer encoder m ) 42 CTC loss calculation section 50 Third layer encoder h ) 52 CTC loss calculation section 110 Target area text supply unit 115 Replacement processing section 120 Pseudo CTC symbol string generator (pseudo symbol string generator) 130 Adapter part 210 CTC symbol string generator (symbol string generator) 220 Mean Square Error Loss Calculation Unit (Loss Calculation Unit for Adapter Unit) 230 CTC loss calculation unit (loss calculation unit for adapter unit) 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus< / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank> < / blank>

Claims

1. an encoder unit having a model consisting of multiple layers, each of which is configured to be capable of machine learning, so as to input a sequence of frames of acoustic features and output an output symbol sequence consisting of a sequence of symbols corresponding to the frames of the acoustic features; an adapter unit that receives a symbol string belonging to a target region and outputs a feature quantity to be input to a model in the final layer of the encoder unit; a pseudo-symbol string generator that generates the symbol string to be input to the adapter unit by performing at least one of repeating characters in the text or inserting zero or more blank symbols between characters in the text, based on the text belonging to the target region; Equipped with an intermediate feature is passed from a model in a certain layer to a model in a next layer in the multiple layer models of the encoder unit; the encoder unit is configured to perform machine learning based on a relationship between a frame sequence of the acoustic features and the output symbol sequence, and then input the features output by the adapter unit to a model of the final layer of the encoder unit, and adapt the model of the final layer to the target domain using a first loss that is a loss based on an output from the model of the final layer. Voice recognition device.

2. a replacement processing unit that performs processing to replace characters included in the text belonging to the target region with other characters in accordance with an output character probability distribution obtained based on predetermined statistical data; Furthermore, the pseudo-symbol string generation unit generates the symbol string based on the text belonging to the target region after the replacement process output by the replacement processing unit.

2. The speech recognition device according to claim 1.

3. The pseudo-symbol string generation unit determining the number of times the same character is repeated when repeating characters in the text based on statistics regarding the number of times the same character is repeated in an intermediate symbol string, which is a symbol string calculated based on features output from the model of a predetermined layer in the encoder unit that has been trained based on the relationship between the frame string of the acoustic features and the output symbol string; determining the number of consecutive blank symbols to be inserted between characters in the text based on statistics regarding the number of consecutive blank symbols in the intermediate symbol string; 3. The speech recognition device according to claim 1.

4. a symbol string generation unit that generates a symbol string to be input to the adapter unit for training the adapter unit based on the intermediate features output from a model of a predetermined layer of the encoder unit based on the acoustic features; an adapter unit loss calculation unit that calculates a second loss, which is a loss for a feature output by the adapter unit, based on the symbol string generated by the symbol string generation unit; Furthermore, The adapter section is configured to perform machine learning on the adapter section using the second loss calculated by the adapter section loss calculation unit.

4. A speech recognition device according to claim 1.

5. The adapter loss calculation unit a loss based on the likelihood of a correct output symbol string given the feature values ​​output from the adapter unit; a loss representing the difference between a feature output from the adapter unit and a feature output from a model of a corresponding layer in the encoder unit; Calculating the second loss based on 5. The speech recognition device according to claim 4.

6. The speech recognition device aggregating repeated characters in the output symbol string and deleting blank symbols contained in the output symbol string, and outputting the resulting character string as a speech recognition result; 3. The speech recognition device according to claim 1.

7. When adapting the model of the final layer to the target region using the first loss, the encoder unit also performs adaptation based on a third loss obtained from a symbol string output by the model consisting of multiple layers included in the encoder unit based on the frame sequence of the acoustic features and the correct output symbol string corresponding to the frame sequence of the acoustic features. A speech recognition device according to any one of claims 1 to 6.

8. an encoder unit having a model consisting of multiple layers, each of which is configured to be capable of machine learning, so as to input a sequence of frames of acoustic features and output an output symbol sequence consisting of a sequence of symbols corresponding to the frames of the acoustic features; an adapter unit that receives a symbol string belonging to a target region and outputs a feature quantity to be input to a model in the final layer of the encoder unit; a pseudo-symbol string generator that generates the symbol string to be input to the adapter unit by performing at least one of repeating characters in the text or inserting zero or more blank symbols between characters in the text, based on the text belonging to the target region; Equipped with an intermediate feature is passed from a model in a certain layer to a model in a next layer in the multiple layer models of the encoder unit; the encoder unit is configured to perform machine learning based on a relationship between a frame sequence of the acoustic features and the output symbol sequence, and then input the features output by the adapter unit to a model of the final layer of the encoder unit, and adapt the model of the final layer to the target domain using a first loss that is a loss based on an output from the model of the final layer. A program that enables a computer to function as a speech recognition device.

Images