Speech recognition device, machine learning method, speech recognition method, and program

Abstract

A speech recognition device according to the present disclosure comprises: a speech distributed representation sequence conversion unit that generates an intermediate acoustic feature sequence through conversion by means of a first neural network; a symbol distributed representation sequence conversion unit that generates an intermediate character feature sequence through conversion by means of a second neural network; a blank estimation unit that accepts the intermediate acoustic feature sequence and the intermediate character feature sequence as input, and outputs a first output probability distribution by means of a third neural network; a label estimation unit that accepts the intermediate acoustic feature sequence and the intermediate character feature sequence as input, and outputs a second output probability distribution by means of a fourth neural network; an HAT loss calculation unit that accepts a third output probability distribution, which is obtained by integration of the first output probability distribution and the second output probability distribution, and a ground-truth symbol sequence as input, and calculates a first loss for HAT; and an internal acoustic model loss calculation unit that accepts the third output probability distribution and the ground-truth symbol sequence as input, and uses the same model parameters as the HAT loss calculation unit.

Description

Speech recognition device, machine learning method, speech recognition method, and program

[0001] The present disclosure relates to a speech recognition technology, and more particularly to a technology for improving recognition performance by newly defining an internal acoustic model in an end-to-end speech recognition model.

[0002] Recent speech recognition methods using neural networks are capable of directly outputting word sequences (characters, subwords, or words) from speech (acoustic) features. Non-Patent Document 1 discloses a method for training a neural network for speech recognition using a training method based on a Recurrent Neural Network Transducer (RNNT) (see the section "Recurrent Neural Network Transducer" in Non-Patent Document 1).

[0003] In training an RNNT model, by introducing a "blank" symbol (referred to as "null output" or "null output" in Non-Patent Document 1) that represents redundancy, it is possible to dynamically learn the correspondence between speech and output sequences from training data as long as only character strings (e.g., phonemes, characters, subwords, word sequences (≠ frame-by-frame)) that correspond to the content of the speech are prepared. In other words, it is possible to train using features and labels in an incompatible relationship between input length T and output length U (generally, T>>U). In addition, since it can operate frame-by-frame during inference, it has attracted a lot of attention as a technology that can operate in streaming mode.

[0004] Alex Graves, "Sequence transduction with recurrent neural networks," in Proc. of ICML, 2012.

[0005] However, RNNT models blank symbols and output symbols (phonemes, characters, subwords, or word sequences) using a single probability distribution, and requires prediction of output symbols (phonemes, characters, subwords, or word sequences) (calculation of output probability sequences) for each frame, which leads to the problem of reduced recognition processing time.

[0006] The present invention has been made in view of the above points, and has an object to suppress a decrease in recognition processing time.

[0007] In order to solve the above problem, the present invention according to claim 1 is a speech recognition device for training a neural network for speech recognition, comprising: an audio densified representation sequence conversion unit that receives an acoustic feature sequence and converts an intermediate acoustic feature sequence using a first neural network; a symbol densified representation sequence conversion unit that receives a symbol sequence and converts it into an intermediate character feature sequence using a second neural network; a blank estimation unit that receives the intermediate acoustic feature sequence and the intermediate character feature sequence and performs label estimation of blank symbols in the speech using a third neural network, thereby outputting a first output probability distribution; a label estimation unit that receives the intermediate acoustic feature sequence and the intermediate character feature sequence and performs label estimation of symbols in an output sequence using a fourth neural network, thereby outputting a second output probability distribution; an output probability integration unit that integrates the first output probability distribution and the second output probability distribution, thereby outputting a third output probability distribution; and a hybrid autoregressive (HAT) unit that receives the third output probability distribution and a correct symbol sequence and performs label estimation of symbols in an output sequence using a fourth neural network. an internal acoustic model loss calculation unit that receives the third output probability distribution and a correct symbol sequence and calculates a second loss for an internal acoustic model by using the same model parameters as those of the HAT loss calculation unit; a loss integration unit that outputs a third loss by integrating the first loss and the second loss; and a parameter update unit that updates model parameters in the first to fourth neural networks by using the third loss.

[0008] As described above, the present invention has the effect of suppressing a decrease in recognition processing time.

[0009] FIG. 1 is a schematic diagram of the premise of this embodiment and a communication system of this embodiment. FIG. 2 is an electrical hardware configuration diagram of a speech recognition device. FIG. 3 is an electrical hardware configuration diagram of a communication terminal. FIG. 4 is a functional configuration diagram of a speech recognition device at the time of learning using a first technology (Non-Patent Document 1) that is the premise of this embodiment. FIG. 5 is a functional configuration diagram of a speech recognition device at the time of learning using a second technology that is the premise of this embodiment. FIG. 6 is a functional configuration diagram of a speech recognition device at the time of learning using a third technology that is the premise of this embodiment. FIG. 7 is a functional configuration diagram of a speech recognition device at the time of inference using a third technology that is the premise of this embodiment. FIG. 8 is a functional configuration diagram of a speech recognition device according to this embodiment at the time of learning. FIG. 9 is a functional configuration diagram of a speech recognition device according to this embodiment at the time of inference. FIG. 10 is a flowchart showing processing during learning of a speech recognition device according to this embodiment. FIG. 11 is a flowchart showing processing during inference of a speech recognition device according to this embodiment.

[0010] The speech recognition device that is the premise of this embodiment and the speech recognition device of this embodiment will be described below with reference to the drawings. Note that the present invention is not limited to the embodiments shown below, and various modifications are possible within the scope of the technical concept of the present invention. Since the drawings are intended to conceptually explain the present invention, dimensions, ratios, and numbers may be exaggerated or simplified as necessary to facilitate understanding.

[0011] [System Configuration] First, the premise of this embodiment and an outline of the configuration of a communication system of this embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic diagram of the premise of this embodiment and the communication system of this embodiment.

[0012] 1, the premise of this embodiment and the communication system 1 of this embodiment are constructed by a voice recognition device 3 and a communication terminal 5. The communication terminal 5 is managed and used by a user.

[0013] Furthermore, the voice recognition device 3 and the communication terminal 5 can communicate with each other via a communication network 100 such as the Internet. The communication network 100 may be connected wirelessly or by wire.

[0014] The speech recognition device 3 is composed of one or more computers. When the speech recognition device 3 is composed of multiple computers, it may be referred to as a "speech recognition device" or a "speech recognition system." The speech recognition device 3 is a device that performs speech recognition using machine learning and / or a neural network for speech recognition.

[0015] It should be noted that voice recognition devices 3 a , 3 b , 3 c 1 , 3 c 2 , 3 d 1 , and 3 d 2 described below are all examples of the voice recognition device 3 .

[0016] The communication terminal 5 is a computer, and a notebook computer is shown as an example in Fig. 1. In Fig. 1, a user operates the communication terminal 5. Note that the speech recognition device 3 may perform processing independently without using the communication terminal 5.

[0017] [Hardware Configuration] <Hardware Configuration of Speech Recognition Device> Next, the electrical hardware configuration of the speech recognition device 3 will be described with reference to Fig. 2. Fig. 2 is a diagram showing the electrical hardware configuration of the speech recognition device.

[0018] As shown in FIG. 2 , the speech recognition device 3 is a computer and includes a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, an SSD (Solid State Drive) 304, an external device connection I / F (Interface) 305, a network I / F 306, a media I / F 309, and a bus line 310.

[0019] Of these, the CPU 301 controls the overall operation of the voice recognition device 3. The ROM 302 stores programs such as an IPL (Initial Program Loader) used to drive the CPU 301. The RAM 303 is used as a work area for the CPU 301.

[0020] The SSD 304 reads or writes various data under the control of the CPU 301. Note that instead of the SSD 304, a hard disk drive (HDD) may be used.

[0021] The external device connection I / F 305 is an interface for connecting various external devices, such as a display, a speaker, a keyboard, a mouse, a USB (Universal Serial Bus) memory, and a printer.

[0022] The network I / F 306 is an interface for performing data communication via the communication network 100 .

[0023] The media I / F 309 controls reading and writing (storing) of data from and to a recording medium 309m such as a flash memory, etc. The recording medium 309m includes a DVD (Digital Versatile Disc) and a Blu-ray Disc (registered trademark).

[0024] The bus line 310 is an address bus, a data bus, etc. for electrically connecting the components such as the CPU 301 shown in FIG.

[0025] <Hardware Configuration of Communication Terminal> Next, the electrical hardware configuration of the communication terminal 5 will be described with reference to Fig. 3. Fig. 3 is a diagram showing the electrical hardware configuration of the communication terminal.

[0026] As shown in FIG. 3, the communication terminal 5 is a computer and includes a CPU 501, a ROM 502, a RAM 503, an SSD 504, an external device connection I / F (Interface) 505, a network I / F 506, a display 507, a pointing device 508, a media I / F 509, and a bus line 510.

[0027] Of these, the CPU 501 controls the overall operation of the communication terminal 5. The ROM 502 stores programs such as IPL used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501.

[0028] The SSD 504 reads or writes various data under the control of the CPU 501. Note that instead of the SSD 504, a hard disk drive (HDD) may be used.

[0029] The external device connection I / F 505 is an interface for connecting various external devices, such as a display, a speaker, a keyboard, a mouse, a USB memory, and a printer.

[0030] The network I / F 506 is an interface for performing data communication via the communication network 100 .

[0031] The display 507 is a type of display means such as a liquid crystal display or organic electroluminescence (EL) display that displays various images.

[0032] The pointing device 508 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. When user Y uses a keyboard, the function of the pointing device 508 may be turned off.

[0033] The media I / F 509 controls reading and writing (storing) of data from and to a recording medium 509m such as a flash memory, etc. The recording medium 509m includes DVDs and Blu-ray Discs (registered trademarks).

[0034] The bus line 510 is an address bus, a data bus, etc. for electrically connecting the components such as the CPU 501 shown in FIG.

[0035] First Technology on Which the Present Embodiment is Based First, the first technology on which the present embodiment is based will be described with reference to FIG.

[0036] [Functional Configuration of Speech Recognition Apparatus Using First Technology] FIG. 4 is a diagram showing the functional configuration of a speech recognition apparatus using the first technology (Non-Patent Document 1) during learning, which is the premise of this embodiment.

[0037] 4, the speech recognition device 3 a includes an acquisition unit 10 , a speech distributed representation sequence conversion unit 11 , a symbol distributed representation sequence conversion unit 12 , a blank label estimation unit 13 , an RNNT loss calculation unit 14 , and a parameter update unit 15 .

[0038] The input (data), output (data), and processing (to output based on the input) of each functional unit will be described below. Note that the input, output, and processing are similar to those in the second and third technologies described below, as well as in this embodiment. In addition, in the first to third technologies and this embodiment, similar functional units are assigned the same reference numerals, and a second description will be omitted.

[0039] <Acquisition unit 10> Input: acoustic feature sequence X (length T), symbol sequence c (length U) Output: acoustic feature sequence X (length T), symbol sequence c (length U) <Audio densified representation sequence conversion unit 11> Input: acoustic feature sequence X Output: intermediate acoustic feature sequence H Processing: The audio densified representation sequence conversion unit 11 acts as an encoder that converts input acoustic features into intermediate acoustic features using a multi-stage neural network.

[0040] <Symbol condensed representation sequence converter 12> Input: Symbol sequence c (length U) Output: Intermediate character feature sequence C (length U) Processing: The symbol condensed representation sequence converter 12 converts the input symbols into corresponding continuous value features. In this case, the symbol condensed representation sequence converter 12 functions as an encoder, first converting the input data into a one-hot vector and then converting it into intermediate character features using a multi-stage neural network.

[0041] <Blank / Label Estimation Unit 13> Input: Intermediate acoustic feature sequence H, intermediate character feature sequence C Output: Output probability distribution Y Processing: The blank / label estimation unit 13 estimates blanks and labels from the intermediate acoustic feature sequence H and the intermediate character feature sequence C (length U) using a neural network, using the following (Equation 1).

[0042] In this case, since the lengths of t and u are different, the number of elements in the neural network is also included in addition to the dimensions of t and u, resulting in a three-dimensional result. Specifically, when adding W1H and W2C, the blank label estimation unit 13 copies and expands the same value for W1H in the U dimension direction, and similarly copies and expands the same value for W2C in the T dimension direction. This adjusts the dimensions of W1H and W2C before adding the three-dimensional tensors, so the output is also a three-dimensional tensor. Generally, during RNNT training, learning is performed using RNNT loss, assuming that the result will be a three-dimensional tensor. During inference, there is no expansion operation, so the output is a two-dimensional matrix.

[0043] <RNNT loss calculation unit 14> Input: output probability distribution Y (three-dimensional tensor), correct symbol sequence C (length U) Output: loss L RNNT Processing: The RNNT loss calculation unit 14 inputs the output probability distribution Y obtained by the blank label estimation unit 13 and the correct symbol sequence c, and calculates the loss L RNNT In this case, the RNNT loss calculation unit 14 creates a tensor with a vertical axis U (symbol sequence length), a horizontal axis T (input sequence length), and a depth K (number of classes: number of symbol entries), and calculates the path of the optimal transition probability on the U×T plane based on a forward-backward algorithm. A more detailed calculation process is described in section "2. Recurrent Neural Network Transducer" of Non-Patent Document 1.

[0044] <Parameter update unit 15> Input: Loss L RNNT Processing: The parameter update unit 15 updates the loss L RNNT The model parameters of the speech distributed representation sequence converter 11, the symbol distributed representation sequence converter 12, and the blank label estimator 13 are updated using the above.

[0045] Second Technology on Which the Present Embodiment is Based Next, the second technology on which the present embodiment is based will be described with reference to FIG.

[0046] [Functional Configuration of a Speech Recognition Device Using the Second Technology] Fig. 5 is a functional configuration diagram of a speech recognition device during training using the second technology (Reference 1), which is the premise of this embodiment. Here, in order to introduce IAM, which is a technology in this embodiment described later, CTC (Connectionist Temporal Classification) in the second technology (Reference 1) will be explained. (Reference 1) Alex Graves et al., "Connectionist Temporal Classification: Labeling Unsegmented Sequence Data with Recurrent Neural Networks," in Proc. of ICML, 2006. In Fig. 5, a speech recognition device 3b includes an acquisition unit 10, a speech distributed representation sequence conversion unit 11, a blank label estimation unit 21, a CTC loss calculation unit 22, and a parameter update unit 23.

[0047] When simultaneously learning CTC, as shown in Figure 5, a blank label estimation unit 21 that holds a neural network for calculating the output probability of CTC is separately prepared, and the blank and the label of the output probability sequence are learned using a single probability distribution, similar to RNNT.

[0048] <Blank Label Estimation Unit 21> Input: Intermediate acoustic feature sequence H Output: Output probability distribution Y Processing: The blank label estimation unit 21 performs label estimation from the intermediate acoustic feature sequence H using the following (Equation 2) with a neural network.

[0049] In this case, unlike RNNT, it depends only on the time frame t, so the number of dimensions of the output is two, which is T plus the number of output classes (blank + number of labels in the output sequence).

[0050] <CTC loss calculation unit 22> Input: output probability distribution Y (two-dimensional matrix), correct symbol sequence C (length U) Output: loss L ctc Processing: As shown in FIG. 2, the CTC loss calculation unit 22 inputs the output probability distribution Y and the correct symbol sequence c obtained by the blank label estimation unit 21 and calculates the loss L ctcThis is a task similar to RNNT, and the CTC loss calculation unit 22 creates a matrix of T (input sequence length) and K (number of classes: number of symbol entries) excluding the vertical axis U (symbol sequence length), and calculates the path of the optimal transition probability that indicates in which frame the label of the output sequence will be output based on a forward-backward algorithm. A more detailed calculation process is described in section "4.1. The CTC Forward-Backward Algorithm" in Reference 1.

[0051] The parameter update unit 15 calculates the loss L CTC The model parameters of the speech distributed representation sequence converter 11 and the blank label estimator 21 are updated using the above.

[0052] Third and Fourth Technologies on Which the Present Embodiment is Based (Functional Configuration of a Speech Recognition Device Using the Third Technology During Training) Next, the third technology on which the present embodiment is based will be described with reference to FIG.

[0053] FIG. 6 is a functional configuration diagram of a speech recognition device using the third technology (Reference 2) that is the premise of this embodiment, during training.

[0054] Here, we will explain the HAT, which is a key technology in this embodiment, which will be described later. The HAT (Hybrid Autoregressive Transducer) is a technology proposed to more effectively integrate with a language model by separately learning the probability distributions of blanks and symbols representing output sequences. (Reference 2) Ehsan Variani, et al., "HYBRID AUTOREGRESSIVE TRANSDUCER (HAT)," in Proc. of ICASSP, 2020. In FIG. 6, the speech recognition device 3c1 during training includes an acquisition unit 10, a speech densified representation sequence conversion unit 11, a symbol densified representation sequence conversion unit 12, a blank estimation unit 31, a label estimation unit 32, an output probability integration unit 33, a HAT loss calculation unit 34, and a parameter update unit 15.

[0055] <Blank estimation unit 31> Input: intermediate acoustic feature sequence H, intermediate character feature sequence C Output: output probability distribution Y' Processing: The blank estimation unit 31 uses the following (Equation 3) to estimate labels of only blank symbols from the intermediate acoustic feature sequence H and the intermediate character feature sequence C (length U) using a neural network.

[0056] The output of the blank estimation unit 31 is a three-dimensional tensor, similar to RNNT. The difference from RNNT is that the class direction dimension is a single class, consisting of only blank. Note that the function used here is a Sigmoid function (range: 0.0 to 1.0), not a Softmax function.

[0057] <Label Estimation Unit 32> Input: intermediate acoustic feature sequence H, intermediate character feature sequence C Output: output probability distribution Y" Processing: The label estimation unit 32 uses the following (Equation 4) to estimate labels of only the symbols in the output sequence from the intermediate acoustic feature sequence H and the intermediate character feature sequence C (length U) using a neural network.

[0058] The output of the label estimation unit 32 is the same as that of the RNNT, and is a three-dimensional tensor. The difference from the RNNT is that the dimension in the class direction is the number of classes (K-1) excluding the blank output.

[0059] <Output Probability Integration Unit 33> Input: output probability distribution Y', output probability distribution Y" Output: output probability distribution Y (three-dimensional tensor) Processing: The output probability integration unit 33 integrates (combines) the probability distributions obtained from the blank estimation unit 31 and the label estimation unit 32, and outputs a three-dimensional tensor of T×U×K, which has the same number of dimensions as the RNNT. In this case, the output probability integration unit 33 combines the above probability distributions after scaling the probabilities of the symbols in the output sequence in the dimension direction of class K, using the following (Equation 5).

[0060] where y' t,u is the probability of blank, y" t,u is the probability of the label, which follows the Bernoulli distribution. The combined probability distribution Y has a sum of 1 in the dimension of the number of classes K, and can be learned using the RNNT loss.

[0061] <HAT loss calculation unit 34> Input: output probability distribution Y (3D tensor), correct symbol sequence C (length U) Output: loss L HAT Processing: The HAT loss calculation unit 34 inputs the output probability distribution Y obtained by the output probability integration unit 33 and the correct symbol sequence c, and calculates the loss L HAT In this case, the HAT loss calculation unit 34, like the RNNT loss calculation unit 14, creates a three-dimensional tensor with a vertical axis U (symbol sequence length), a horizontal axis T (input sequence length), and a depth K (number of classes: number of symbol entries), and calculates the path of the optimal transition probability on the U×T plane based on a forward-backward algorithm (see Non-Patent Document 1).

[0062] [Functional Configuration of a Speech Recognition Apparatus Using the Fourth Technology During Inference] Next, the fourth technology that is the premise of this embodiment will be described with reference to FIG.

[0063] FIG. 7 is a functional configuration diagram of a speech recognition device using the fourth technology (Reference 3) that is the premise of this embodiment, during inference.

[0064] In Non-Patent Document 1, RNNT is trained using a single probability distribution for the output probability of blank and the label of the output sequence. In speech recognition using RNNT, more than half of the frames are blank, which creates a bottleneck in computational processing speed, but the probability of calculating the output probability of the symbol in the output sequence is low. On the other hand, as mentioned above, HAT is capable of predicting blank and the label of the output sequence separately. This section describes a technology (see Reference 3) that is an inference method that takes advantage of this characteristic of HAT. (Reference 3) Duc Le et al., "FACTORIZED BLANK THRESHOLDING FOR IMPROVED RUNTIME EFFICIENCY OF NEURAL TRANSDUCERS," in Proc. of ICASSP 2023 In Figure 7, during inference corresponding to the learning time in Figure 6, the speech recognition device 3c2 has an acquisition unit 10, a speech distributed representation sequence conversion unit 11, a symbol distributed representation sequence conversion unit 12, a blank estimation unit 31, a label estimation unit 32, a calculation judgment unit 41, an output probability integration unit 42, and an output unit 43.

[0065] <Calculation determination unit 41> Input: Blank output probability distribution y', threshold λ HAT Output: True / False value. Processing: The calculation determination unit 41 calculates the output probability y′ of the blank estimation unit 31 and a preset threshold λ HAT Compared with the threshold λ HAT If above, False, threshold λ HAT If the result is True, the calculation determination unit 41 determines that the label estimation unit 302 of the frame should perform the calculation. On the other hand, if the result is False, the calculation determination unit 41 determines that the label estimation unit 302 of the frame should not perform the process (omit the process). Note that the probability P blank (scalar) is the threshold λ HAT In the above cases, the label estimation unit 32 estimates the probability P label (vector) is output as a vector with all values ​​of "0". HAT is an example of the second threshold value.

[0066] <Output probability integration unit 42> Input: output probability distribution P blank (scalar), output probability distribution P label (vector) Output: Integrated probability (vector) Processing: The output probability integration unit 42 integrates the output probability distribution P blank (scalar), and the output probability distribution P label (vector) to obtain the integrated output probability distribution (P blank ;(1-P blank )*P label ) is output. At this time, the integrated output probability distribution is sampled according to the output probability distribution (for example, the symbol corresponding to the index with the maximum probability) to uniquely determine a symbol, which is then recursively input to the symbol distributed representation sequence converter 12 as the output of the model itself.

[0067] In addition, the output probability distribution P blank is an example of the first output probability distribution, output probability distribution P label is an example of the second output probability distribution, the output probability distribution (P blank ;(1-P blank )*P label) is an example of the second output probability distribution.

[0068] <Output unit 43> Input: Integrated output probability distribution Output: Symbol sequence (speech recognition result) Processing: The output unit 43 outputs a symbol sequence obtained by sampling (for example, the symbol corresponding to the index with the maximum probability) according to the integrated output probability distribution obtained from the output probability integration unit 42 to an external device such as the outside of the speech recognition device 3c2.

[0069] Speech recognition device of this embodiment (Functional configuration of the speech recognition device according to this embodiment during training) Next, the functional configuration of the speech recognition device according to this embodiment during training will be described with reference to Fig. 8. Fig. 8 is a functional configuration diagram of the speech recognition device according to this embodiment during training. In this embodiment, an internal acoustic model (IAM) is used.

[0070] In addition, the speech distributed representation sequence conversion unit 11 holds a first neural network, the symbol distributed representation sequence conversion unit 12 holds a second neural network, the blank estimation unit 31 holds a third neural network, and the label estimation unit holds a fourth neural network.

[0071] In Figure 8, the speech recognition device 3d1 during training has an IAM loss calculation unit 51 and a loss integration unit 52 in addition to the configuration of the speech recognition device 3c1 shown in Figure 6 (acquisition unit 10, speech distributed representation sequence conversion unit 11, symbol distributed representation sequence conversion unit 12, blank estimation unit 31, label estimation unit 32, output probability integrating unit 33, HAT loss calculation unit 34, and parameter update unit 15). Note that in Figure 8 and Figure 9 described below, the route indicated by the solid arrow is a route that uses both speech and language. Also, the route indicated by the dashed arrow is a route that uses only speech, as in Figure 5.

[0072] <IAM (Internal Acoustic Model) Loss Calculation Unit 51> As shown by the dashed arrow route in Fig. 8, in the IAM, blanks and labels are estimated without using linguistic information using the speech embedding sequence conversion unit 11, the HAT blank estimation unit 31, and the label estimation unit 32. Input: Output probability distribution Y (two-dimensional matrix), correct symbol sequence C (length U) Output: Loss L IAM Processing: In general multitask learning, each model parameter is learned separately, as shown in FIG. 5, for example. However, the IAM (internal acoustic model) loss calculation unit 51 performs learning using the same model parameters as the HAT loss calculation unit 34.

[0073] The IAM loss calculation unit 51 calculates the loss L of the internal acoustic model (IAM) by using the output probability distribution Y integrated by the output probability integration unit 33 and the correct symbol sequence C as the same loss as that of the CTC loss calculation unit 22 shown in FIG. IAM Calculate.

[0074] The IAM loss calculation unit 51 receives the input output probability distribution Y and the correct symbol sequence C and calculates the loss L in the same way as the blank label estimation unit 21 shown in FIG. IAM As a result, the output probability integrating unit 33 and the IAM loss calculating unit 51 constitute a role unit 21′ similar to the blank label estimating unit 21 shown in FIG.

[0075] <Loss integration unit 53> Input: Loss L from IAM loss calculation unit 51 IAM , loss L from the HAT loss calculation unit 34 HAT , and weight α Output: loss L HAT+IAM Processing: The loss integration unit 53 calculates the loss L using the following (Equation 6). HAT and the loss L multiplied by the weight α IAM Adding these together, the integrated loss L HAT+IAM Output.

[0076] Here, α is a hyperparameter that represents the weight of each loss.

[0077] By simultaneously training the IAM, a task similar to the HAT, the training of the HAT progresses efficiently, and it is expected that the recognition performance will improve. The HAT and the IAM are combined using the loss L HAT+IAM is learned by

[0078] [Functional Configuration of the Speech Recognition Apparatus According to the Present Embodiment During Inference] Next, the functional configuration of the speech recognition apparatus according to the present embodiment during inference will be described with reference to Fig. 9. Fig. 9 is a functional configuration diagram of the speech recognition apparatus according to the present embodiment during inference. As shown in Fig. 7, the calculation determination unit 41 uses symbols estimated by the immediately preceding model itself in the calculation of the blank estimation unit 31, so it is necessary to perform calculation and determination recursively for each frame. Therefore, in this embodiment, a method is proposed in which unnecessary frames are determined all at once using IAM.

[0079] In Figure 9, the speech recognition device 3d2 during inference has the same configuration as the speech recognition device 3c2 shown in Figure 7 (acquisition unit 10, speech distributed representation sequence conversion unit 11, symbol distributed representation sequence conversion unit 12, blank estimation unit 31, label estimation unit 32, calculation judgment unit 41, and output probability integration unit 42) as well as a speech distributed representation sequence rejection unit 61.

[0080] <Speech distributed expression sequence rejection unit 61> Input: Blank output probability distribution y' of IAM, threshold λ IAM Output: True / False value. Processing: The speech embedding sequence rejection unit 61, during inference, determines whether the blank output probability distribution y′ acquired from the blank estimation unit 31 of the IAM exceeds a preset threshold λ IAM If above, False, threshold λ IAM If the threshold λ is less than , the output is True. IAM is an example of the first threshold value.

[0081] The difference between the IAM and the calculation determination unit 41 is that, unlike HAT or RNNT, the blank estimation unit 31 and label estimation unit 32 of the IAM do not depend on their own outputs, and therefore the speech distributed representation sequence rejection unit 61 can simultaneously calculate the blank output probability distribution y' for all frames 1 to T. Another advantage of the IAM is that, unlike CTC, blank and output sequence symbols are not learned using a single distribution, but are learned using separate distributions. Therefore, if calculations are performed in the blank estimation unit 31, calculations do not need to be performed in the label estimation unit 32, and therefore the calculation processing is lighter than that of CTC.

[0082] Furthermore, since the IAM shares model parameters with the HAT and is trained together, the timing of the blank output is synchronized, which is expected to enable more efficient omission of calculations in the label estimation unit 32. In other words, it is expected that the recognition speed will be improved while maintaining performance.

[0083] Specifically, the speech embedding sequence rejection unit 61 rejects the speech embedding sequence h t is input to the inference unit (blank estimation unit 31, label estimation unit 32) of the HAT, and the speech embedding sequence for the corresponding frame t that is False is rejected without being input to the inference unit (blank estimation unit 31, label estimation unit 32) of the HAT. As described above, the speech embedding sequence rejection unit 61 can be used in conjunction with the calculation determination unit 41, and using them in conjunction further improves the recognition speed.

[0084] For example, since the input speech data contains mostly blanks (e.g., 90%), the portion where symbols appear as recognition results (e.g., 10%) is small. In this case, even if the speech embedded representation sequence rejection unit 61 acquires T-dimensional speech intermediate features from the speech embedded representation sequence conversion unit 11, the estimation result by the blank estimation unit 31 contains 90% blanks. Therefore, the speech embedded representation sequence rejection unit 61 only needs to handle the remaining 10% of the T-dimensional speech intermediate features.

[0085] [Processing During Learning of the Speech Recognition Apparatus According to the Present Embodiment] Next, processing during learning of the speech recognition apparatus according to the present embodiment will be described with reference to Fig. 10. Fig. 10 is a flowchart showing processing during learning of the speech recognition apparatus according to the present embodiment.

[0086] S11: The acquisition unit 10 acquires an acoustic feature sequence X and a symbol sequence c (length U) from the outside.

[0087] S12: The speech distributed representation sequence conversion unit 11 converts the acoustic feature sequence X acquired from the acquisition unit 10 into an intermediate acoustic feature sequence H.

[0088] S13: The symbol distributed representation sequence conversion unit 12 converts the symbol sequence c (length U) acquired from the acquisition unit 10 into an intermediate character feature sequence C (length U). Note that if no language information is input and only audio information is received, the symbol distributed representation sequence conversion unit 12 does not execute any processing.

[0089] S14: The blank estimation unit 31 performs label estimation of only blank symbols based on the intermediate acoustic feature sequence H obtained from the speech distributed representation sequence conversion unit 11 and the intermediate character feature sequence C obtained from the symbol distributed representation sequence conversion unit 12, and outputs an output probability distribution Y' (first output probability distribution), which is a three-dimensional tensor whose class-direction dimension is one class consisting only of blank.

[0090] S15: The label estimation unit 32 performs label estimation of only the symbols in the output sequence based on the intermediate acoustic feature sequence H acquired from the speech distributed representation sequence conversion unit 11 and the intermediate character feature sequence C acquired from the symbol distributed representation sequence conversion unit 12, and outputs an output probability distribution Y″ (second output probability distribution), which is a three-dimensional tensor whose class-direction dimension is the number of classes, K−1, excluding the blank output.

[0091] S16: The output probability integration unit 33 integrates the output probability distribution Y′ acquired from the blank estimation unit 31 and the output probability distribution Y″ acquired from the label estimation unit 32, and outputs the output probability distribution Y (third output probability distribution), which is a three-dimensional tensor having the same number of dimensions as RNNT.

[0092] S17: The HAT loss calculation unit 34 inputs the output probability distribution Y obtained by inputting the speech and language obtained by the output probability integration unit 33 and the correct symbol sequence c, and calculates the loss L for the HAT. HAT And / or, the IAM loss calculation unit 51 inputs the output probability distribution Y obtained by inputting only the speech obtained by the output probability integration unit 33 and the correct symbol sequence c, and calculates the loss L for the IAM using the same model parameters as the HAT loss calculation unit 34. IAM Calculate.

[0093] S18: The loss integration unit 52 calculates the loss L obtained from the IAM loss calculation unit 51. HAT and the loss L obtained from the HAT loss calculation unit 34 IAM , and the weight α, the integrated loss L HAT+IAM Output.

[0094] S19: The parameter update unit 15 updates the loss L HAT+IAM The model parameters of the speech distributed representation sequence converter 11, the symbol distributed representation sequence converter 12, the blank estimator 31, and the label estimator 32 are updated using the above.

[0095] [Processing During Inference in the Speech Recognition Apparatus According to the Present Embodiment] Next, processing during inference in the speech recognition apparatus according to the present embodiment will be described with reference to Fig. 11. Fig. 11 is a flowchart showing processing during inference in the speech recognition apparatus according to the present embodiment.

[0096] S31: The acquisition unit 10 acquires an acoustic feature sequence X and a symbol sequence c (length U) from the outside.

[0097] S32: The speech distributed representation sequence conversion unit 11 converts the acoustic feature sequence X acquired from the acquisition unit 10 into an intermediate acoustic feature sequence H.

[0098] S33: The symbol distributed representation sequence conversion unit 12 converts the symbol sequence c (length U) acquired from the acquisition unit 10 into an intermediate character feature sequence C (length U). Note that if no language information is input and only audio information is received, the symbol distributed representation sequence conversion unit 12 does not execute any processing.

[0099] S34: The blank estimation unit 31 performs label estimation of only blank symbols based on the intermediate acoustic feature sequence H obtained from the speech distributed representation sequence conversion unit 11 and the intermediate character feature sequence C obtained from the symbol distributed representation sequence conversion unit 12, and outputs an output probability distribution Y', which is a three-dimensional tensor whose class direction dimension is one class consisting only of blank.

[0100] S35: The speech distributed expression sequence rejection unit 61 determines whether the blank output probability distribution y′ acquired from the blank estimation unit 31 of the IAM exceeds a preset threshold λ IAM If above, False, threshold λ IAM If the speech embedding sequence rejection unit 61 determines that the speech embedding sequence h of the frame t is True, the speech embedding sequence rejection unit 61 determines that the speech embedding sequence h of the frame t is True. t is input to the inference unit of the HAT (blank estimation unit 31, label estimation unit 32), and the speech embedding sequence for the corresponding frame t that is False is rejected without being input to the inference unit of the HAT (blank estimation unit 31, label estimation unit 32).

[0101] S36: The blank estimation unit 31 determines whether the speech embedding sequence h of the frame t is True, which is output by the speech embedding sequence rejection unit 61. t and outputs the blank output probability distribution y′ to the calculation determination unit 41 and the output probability integration unit 42.

[0102] S37: The calculation determination unit 41 determines whether the blank output probability distribution y′ acquired from the blank estimation unit 31 is greater than or equal to the threshold λ HAT If above, False, threshold λ HAT If the result is True, the calculation determination unit 41 determines that the label estimation unit 302 of the frame should execute the processing. On the other hand, if the result is False, the calculation determination unit 41 determines that the label estimation unit 302 of the frame should not execute the processing (omit the processing).

[0103] S38: The label estimation unit 32 performs label estimation of only the symbols in the output sequence based on the intermediate acoustic feature sequence H obtained from the speech distributed representation sequence conversion unit 11 via the speech distributed representation sequence rejection unit 61 and the intermediate character feature sequence C obtained from the symbol distributed representation sequence conversion unit 12, and outputs an output probability distribution Y'', which is a three-dimensional tensor whose class-direction dimension is the number of classes, K-1, excluding the blank output.

[0104] S39: The output probability integration unit 42 calculates the output probability distribution P blank (scalar), and the output probability distribution P label (vector) to obtain the integrated output probability distribution (P blank ;(1-P blank )*P label ) is output. At this time, the integrated output probability distribution is sampled according to the output probability distribution (for example, the symbol corresponding to the index with the maximum probability) to uniquely determine a symbol, which is then recursively input to the symbol distributed representation sequence converter 12 as the output of the model itself.

[0105] S40: The output unit 43 outputs a symbol sequence obtained by sampling (for example, the symbol corresponding to the index with the maximum probability) according to the integrated output probability distribution obtained from the output probability integration unit 42 as a speech recognition result to an external device such as the outside of the speech recognition device 3c2.

[0106] [Major Effects of the Embodiment] As described above, according to this embodiment, an internal acoustic model (IAM) that improves recognition performance is proposed by further developing the third technology (HAT in Reference 2) that models blank symbols and output sequence symbols (phonemes, characters, subwords, or word sequences) using separate distributions. This IAM has the same parameters as the HAT (completely shares them) and is trained simultaneously with the HAT using the second technology (CTC in Reference 1) specifically designed for speech. Because the HAT and IAM perform similar tasks, training them together is expected to improve the performance of the HAT. Furthermore, because the model parameters are completely shared during inference, it is expected that the timing of blank symbol output between the HAT and the IAM can be synchronized. For frames where the probability of a blank symbol is below a certain threshold, the label estimation unit 37 can omit calculation of the output sequence symbols (phonemes, characters, subwords, or word sequences) from the label estimation unit 32. This is expected to improve recognition speed while maintaining recognition performance.

[0107] [Supplementary Information] The present invention is not limited to the above-described embodiment, and may have the following configurations or processes (operations). (1) The speech recognition device 3 can be realized by a computer and a program, and this program can be recorded on a (non-temporary) recording medium or provided via the communication network 100. (2) In the above-described embodiment, a notebook computer is shown as an example of the communication terminal 5, but this is not limited to this, and the communication terminal 5 may also be, for example, a desktop computer, a tablet terminal, a smartphone, a smartwatch, a car navigation device, a refrigerator, a microwave oven, etc. (3) Each CPU 301, 501 as a processor may be a single processor or multiple processors.

[0108] REFERENCE SIGNS LIST 1 Communication system 3 Speech recognition device 5 Communication terminal 10 Acquisition unit 11 Speech distributed representation sequence conversion unit 12 Symbol distributed representation sequence conversion unit 15 Parameter update unit 31 Blank estimation unit 32 Label estimation unit 33 Output probability integration unit 34 HAT loss calculation unit 41 Calculation determination unit 42 Output probability integration unit 43 Output unit 51 IAM loss calculation unit (internal acoustic model loss calculation unit) 52 Loss integration unit 61 Speech distributed representation sequence rejection unit

Claims

1. A speech recognition device that performs machine learning on a neural network for speech recognition, comprising: an audio distributed representation sequence conversion unit that receives an acoustic feature sequence and converts an intermediate acoustic feature sequence using a first neural network; a symbol distributed representation sequence conversion unit that receives a symbol sequence and converts it into an intermediate character feature sequence using a second neural network; a blank estimation unit that receives the intermediate acoustic feature sequence and the intermediate character feature sequence and performs label estimation of blank symbols in the speech using a third neural network, thereby outputting a first output probability distribution; a label estimation unit that receives the intermediate acoustic feature sequence and the intermediate character feature sequence and performs label estimation of symbols in an output sequence using a fourth neural network, thereby outputting a second output probability distribution; an output probability integration unit that integrates the first output probability distribution and the second output probability distribution, and outputs a third output probability distribution; and a HAT loss calculation unit that receives the third output probability distribution and a correct symbol sequence and calculates a first loss for a HAT (Hybrid Autoregressive Transducer). an internal acoustic model loss calculation unit that inputs the third output probability distribution and the correct symbol sequence and calculates a second loss for an internal acoustic model using the same model parameters as those of the HAT loss calculation unit; a loss integration unit that outputs a third loss by integrating the first loss and the second loss; and a parameter update unit that updates model parameters in the first to fourth neural networks using the third loss.

2. A machine learning method executed by a speech recognition device that machine-learns a neural network for speech recognition, comprising: an audio distributed representation sequence conversion process that inputs an acoustic feature sequence and converts an intermediate acoustic feature sequence using a first neural network; a symbol distributed representation sequence conversion process that inputs a symbol sequence and converts it into an intermediate character feature sequence using a second neural network; a blank estimation process that inputs the intermediate acoustic feature sequence and the intermediate character feature sequence and performs label estimation of blank symbols in the speech using a third neural network, thereby outputting a first output probability distribution; a label estimation process that inputs the intermediate acoustic feature sequence and the intermediate character feature sequence and performs label estimation of symbols in the output sequence using a fourth neural network, thereby outputting a second output probability distribution; an output probability integration process that integrates the first output probability distribution and the second output probability distribution, and outputs a third output probability distribution that is a three-dimensional tensor; and a HAT loss calculation process that inputs the third output probability distribution and a correct symbol sequence and calculates a first loss for a HAT (Hybrid Autoregressive Transducer). an internal acoustic model loss calculation process that inputs the third output probability distribution and a correct symbol sequence and calculates a second loss for an internal acoustic model using the same model parameters as in the HAT loss calculation process; a loss integration process that outputs a third loss that integrates the first loss and the second loss; and a parameter update process that updates model parameters in the first to fourth neural networks using the third loss.

3. A program for causing a computer to execute the method according to claim 2.

4. A speech recognition device that performs speech recognition using a neural network, comprising: a speech embedding representation sequence conversion unit that receives an acoustic feature sequence as input and converts an intermediate acoustic feature sequence using a first neural network; a symbol embedding representation sequence conversion unit that receives a symbol sequence and converts it into an intermediate character feature sequence using a second neural network; a blank estimation unit that receives the intermediate acoustic feature sequence and the intermediate character feature sequence as input and performs label estimation of blank symbols in the speech using a third neural network, thereby outputting a first output probability distribution; a speech embedding representation sequence rejection unit that rejects the speech embedding representation sequence of a corresponding frame if the first output probability distribution is smaller than a first threshold; a label estimation unit that receives the intermediate acoustic feature sequence and the intermediate character feature sequence as input and performs label estimation of symbols in an output sequence using a fourth neural network, thereby outputting a second output probability distribution; and a calculation determination unit that causes the label estimation unit to execute processing if the first output probability distribution is equal to or greater than the first threshold and equal to or greater than the second threshold. an output probability integrating unit that integrates the first output probability distribution and the second output probability distribution to output a third output probability distribution.

5. A speech recognition method executed by a speech recognition device that performs speech recognition using a neural network, comprising: a speech distributed representation sequence conversion process that inputs an acoustic feature sequence and converts an intermediate acoustic feature sequence using a first neural network; a symbol distributed representation sequence conversion process that inputs a symbol sequence and converts it into an intermediate character feature sequence using a second neural network; a blank estimation process that inputs the intermediate acoustic feature sequence and the intermediate character feature sequence and performs label estimation of blank symbols in the speech using a third neural network, thereby outputting a first output probability distribution; a speech distributed representation sequence rejection process that rejects the speech distributed representation sequence of a corresponding frame if the first output probability distribution is smaller than a first threshold; a label estimation process that inputs the intermediate acoustic feature sequence and the intermediate character feature sequence and performs label estimation of symbols in an output sequence using a fourth neural network, thereby outputting a second output probability distribution; and a calculation and determination process that executes the label estimation process if the first output probability distribution is equal to or greater than the first threshold and the second threshold. an output probability integrating process of integrating the first output probability distribution and the second output probability distribution to output a third output probability distribution.

6. A program for causing a computer to execute the method according to claim 5.

Images