Sound recognition device and program

Abstract

To provide a sound recognition device that can reduce erroneous recognition of repeatedly outputting character (strings) uttered in the past.SOLUTION: A sound recognition device includes: a streaming transformer having L (L is an integer more than or equal to two) encoder layers having a layer structure and configured to input series of an acoustic feature quantity of a recognition target and output a recognition result corresponding to the series of the acoustic feature quantity of the recognition target; and a self-conditional CTC processing unit configured to generate a signal representing probability distribution information about a symbol constituting the recognition result based on an intermediate layer signal from an encoder layer of the l-th (1≤l≤L-1) layer of the streaming transformer and mix the intermediate layer signal and the signal representing the probability distribution information about the symbol, thereby supplying an input signal to the encoder layer of the (l+1)-th layer of the streaming transformer.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a voice recognition device and a program. Regarding. [Background technology]

[0002] There are two types of speech recognition processing methods that output recognition results in real time. The first method inputs acoustic features one by one and outputs the recognition results. The second method combines multiple acoustic features into one block, inputs the acoustic features of each block, and outputs the recognition results. In recent years, a speech recognition technology called Streaming Transformer has been proposed that processes each block. With the Streaming Transformer method, previously output recognition results can be output while taking into account previously output linguistic information by re-inputting the previously output recognition results into the model. In addition, the Streaming Transformer achieves high recognition performance by introducing a method called context embedding that inherits the acoustic information of previously processed blocks.

[0003] Non-Patent Document 1 describes a streaming transformer technology.

[0004] Non-Patent Document 2 describes a mechanism for inheriting context in an encoder (see FIG. 2, etc.).

[0005] Non-Patent Document 3 describes a CTC-based speech recognition technique. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Emiru Tsunoo, Yosuke Kashiwagi, Shinji Watanabe, “Streaming Transformer ASR with Blockwise Synchronous Beam Search”, In Proc. IEEE Spoken Language Technology Workshop (SLT), 2021. [Non-Patent Document 2] Emiru Tsunoo, Yosuke Kashiwagi, Toshiyuki Kumakura, Shinji Watanabe, “Transformer ASR with Contextual Block Processing”, In Proc. IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2019. [Non-Patent Document 3] Jumon Nozaki, Tatsuya Komatsu, “Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictions”, In Proc. INTERSPEECH, pp. 3735-3739, 2021. Summary of the Invention [Problem to be solved by the invention]

[0007] However, the streaming transformer technology has a problem in that it repeatedly outputs characters that were previously spoken, which can result in misrecognition.

[0008] The reason why the streaming transformer method causes misrecognition is as follows. In other words, in the streaming transformer method, a neural network called a decoder refers to the encoded information and continues to output recognition results until no new recognition results are output. In this method, the recognition result output process ends when a special symbol indicating the end of speech or the previously output character is output repeatedly. A drawback of this method is that the termination condition may not be satisfied due to repeated patterns, such as repeated strings of two characters, in which case the same pattern of recognition results may be output repeatedly. For these reasons, the recognition accuracy of the streaming transformer method decreases.

[0009] Here, an example of misrecognition when using the streaming transformer technology will be described.

[0010] The correct sentence is as follows: "As for the path to achieving 2% growth, in terms of the mechanism for achieving it, as I have said in the past, on the one hand, by lowering real interest rates, the economy will be stimulated, and wages and prices will rise through lowering the unemployment rate or narrowing the GDP gap. The mechanism by which this happens remains fundamentally unchanged."

[0011] On the other hand, the recognition results are as follows: "As for the path to achieving 2% growth, the mechanism for achieving it is, as I have said before, to stimulate the economy by lowering real interest rates, and to lower the unemployment rate or reduce the GDP gap, which in turn raises wages and prices. On the one hand, it's true that there are differences in how things are carried out, but on the other hand, there are also differences.

[0012] In the above example of the recognition result, the character strings "desu ne" and "hanbetsu wo" are repeatedly output, thus causing misrecognition.

[0013] The cause of the above misrecognition is due to the source-target attention mechanism of the Streaming Transformer. Each layer of the decoder in the Streaming Transformer has a source-target attention mechanism. In other words, one layer of the decoder in the Streaming Transformer uses the feature of the recognition result output in the past as a query, calculates weights for each vector of the encoder output, and propagates the weighted sum of each vector of the encoder output to the next decoder layer. This mechanism allows the Streaming Transformer to flexibly select the acoustic features to be output from the encoder, but there is a possibility that the encoder output of the same part may be repeatedly referenced, which may cause misrecognition by re-outputting characters or words that were previously output.

[0014] The present invention has been made based on the above-mentioned problem recognition, and aims to provide a speech recognition device and a program capable of reducing erroneous recognition in which previously spoken characters (strings) are output again (repeatedly), as seen in the recognition results output from a streaming transformer. [Means for solving the problem]

[0015] [1] In order to solve the above problems, a speech recognition device according to one aspect of the present invention has a streaming transformer having L (L is an integer equal to or greater than 2) encoder layers in a layered configuration, inputting a sequence of acoustic features to be recognized and context embedding, which is information about acoustic features from the sequence of acoustic features to be recognized, and outputting a recognition result corresponding to the sequence of acoustic features to be recognized; and l out ), a signal representing probability distribution information about the symbols constituting the recognition result is generated based on the intermediate layer signal, and the intermediate layer signal and the signal representing the probability distribution information about the symbols are mixed to generate an input signal (h l+1 in and a self-conditional CTC processor that provides the CTC as a

[0016] [2] Moreover, one aspect of the present invention is a speech recognition device according to [1] above, configured to use training data which is a set of pairs of a sequence of training acoustic features and a superscript text corresponding to the sequence of training acoustic features, input the sequence of training acoustic features of the training data to the streaming transformer, and enable optimization processing of values ​​of internal parameters of the streaming transformer and values ​​of internal parameters of the self-conditional CTC processing unit based on a loss between a recognition result output from the streaming transformer and the superscript text of the training data.

[0017] [3] In one aspect of the present invention, in the speech recognition device according to the above [1] or [2], the sequence of acoustic features to be recognized is a block including N acoustic features (N is a positive integer), and the encoder layer of the first layer of the streaming transformer inputs the block and simultaneously encodes N acoustic features included in a block further in the future than the block. ahead pieces(N ahead The (l+1)-th layer of the streaming transformer inputs the signal of the block to be recognized passed from the self-conditional CTC processing unit, and also reads ahead and inputs the signal passed from the self-conditional CTC processing unit based on a block further in the future than the block in question.

[0018] [4] In addition, according to one aspect of the present invention, in the speech recognition device according to any one of [1] to [3] above, the self-conditional CTC processing unit is configured to process an intermediate layer signal (h l out ), a softmax layer that outputs the probability distribution information by linearly transforming the output from the layer normalization layer and performing softmax processing, and l norm ) and the output from the softmax layer (z l ) to the encoder layer of the (l+1)th layer of the streaming transformer. l+1 in and a mixing unit that outputs the

[0019] [5] In addition, according to one aspect of the present invention, in the speech recognition device according to any one of [1] to [3] above, the self-conditional CTC processing unit is configured to process an intermediate layer signal (h l out), a softmax layer that linearly transforms the output from the layer normalization layer and performs softmax processing to output the probability distribution information, and l ) is input, and the signal (z l rnn ) and the output from the layer normalization layer (z l norm ) and the output from the recurrent neural network layer (z l rnn ) to the encoder layer of the (l+1)th layer of the streaming transformer. l+1 in ), wherein the recurrent neural network layer performs processing using a recurrent neural network also based on the latent vector resulting from processing the signals of a previous block by the recurrent neural network layer, and stores the latent vector resulting from processing the signals of the block to be processed for processing the next block.

[0020] [6] In addition, in one aspect of the present invention, in the speech recognition device according to the above [5], the mixing unit is l norm ) and the output from the recurrent neural network layer (z l rnn ) and perform linear transformation processing to calculate a one-dimensional coefficient (α), and the output (z l norm ) and the output from the recurrent neural network layer (z l rnn ) to the encoder layer of the (l+1)th layer of the streaming transformer. l+1 in ) is output.

[0021] [7] Also, one aspect of the present invention includes a streaming transformer having L (L is an integer equal to or greater than 2) encoder layers in a layered configuration, inputting a sequence of acoustic features to be recognized and inputting context embedding, which is information on acoustic features from the sequence of acoustic features to be recognized, and outputting a recognition result corresponding to the sequence of acoustic features to be recognized; and a streaming transformer that outputs an intermediate layer signal (h l out ), a signal representing probability distribution information about the symbols constituting the recognition result is generated based on the intermediate layer signal, and the intermediate layer signal and the signal representing the probability distribution information about the symbols are mixed to generate an input signal (h l+1 in ) and a self-conditional CTC processing unit that provides the speech recognition device with the speech recognition device. Effect of the Invention

[0022] According to the present invention, speech recognition is performed in the intermediate layer of the network by the self-conditional CTC process, and the probability distribution information on the characters (symbols) output from the intermediate layer is fed back to the network of the speech recognition model. This allows the speech recognition device to output the recognition result while taking the context into account. In other words, it is expected that the recognition accuracy will be improved. [Brief description of the drawings]

[0023] [Figure 1] 1 is a block diagram showing a schematic functional configuration of a voice recognition device according to a first embodiment of the present invention. [Diagram 2] 3 is a schematic diagram showing an example of a relationship between blocks and acoustic features in processing for each block in the speech recognition device according to the first embodiment; FIG. [Diagram 3] FIG. 11 is a block diagram showing a schematic functional configuration of a voice recognition device according to a second embodiment. [Figure 4] FIG. 11 is a block diagram showing a schematic functional configuration of a voice recognition device according to a third embodiment. [Diagram 5] FIG. 11 is a block diagram showing the internal functional configuration of a merge layer of a speech recognition device according to a third embodiment. [Figure 6] FIG. 1 is a block diagram showing an example of the internal configuration (computer) of a voice recognition device according to a first embodiment to a third embodiment (including modified examples). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Next, a number of embodiments of the present invention will be described with reference to the drawings. These embodiments are significant improvements on the speech recognition device using a streaming transformer described as the prior art.

[0025] In these embodiments, in order to solve the above-mentioned problems, a method is adopted in which the source-target attention mechanism of the streaming transformer is not used, and a recognition result is output for each input acoustic feature. One of the methods for outputting a recognition result for each acoustic feature is the technology of CTC (Connectionist Temporal Classification). In CTC, in order to fill the difference between the sequence length of the acoustic feature and the sequence length of the correct character or word, the correct character or word is consecutively made and blank symbols are introduced. In this way, the acoustic feature and the correct character or word are trained in one-to-one correspondence. When performing the recognition process, the consecutively output characters or words are converted into one, and the blank symbols are deleted, thereby outputting the final speech recognition result.

[0026] That is, in each of the following embodiments, a CTC-based speech recognition model is operated in real time to achieve one-to-one correspondence between acoustic features and recognition results. This prevents erroneous recognition due to erroneous reference to past information, which is seen in the source-target attention mechanism of the streaming transformer.

[0027] However, a speech recognition model based on CTC technology does not have a mechanism for outputting recognition results while utilizing linguistic information, as in the case of streaming transformers. Therefore, there is a possibility that high recognition accuracy cannot be obtained by simply applying CTC technology. Therefore, in each of the following embodiments, self-conditioning CTC is introduced. Self-conditioning CTC is a method for improving recognition accuracy by temporarily outputting recognition results in the intermediate layer of the model, and then inputting the information of the recognition results back into the model as linguistic information.

[0028] In each of the following embodiments (first to third embodiments), this self conditioning CTC operates in real time. In addition, in order to recognize speech while inheriting acoustic information of previously recognized speech segments, the context embedding method proposed in the Streaming transformer is also utilized.

[0029] Moreover, the speech recognition devices of the second and third embodiments are equipped with an RNN layer (RNN stands for Recurrent Neural Network) and perform RNN processing. This RNN layer performs processing based on the probability distribution vector for characters in the intermediate layer (layer normalization layer) output by the Self Conditioning CTC method. This allows the speech recognition devices of the second and third embodiments to perform recognition processing while inheriting longer-term context.

[0030] The speech recognition device of the third embodiment also includes a merge layer. The merge layer combines a vector output from the RNN layer with a vector output from an intermediate layer (layer normalization layer) and generates a one-dimensional scalar value (α) by linearly transforming the vector. The merge layer uses the scalar value (α) as a weight and has the function of determining each time whether the speech recognition model uses information on the vector that has been processed by the RNN.

[0031] By using the above-mentioned technique, each of the speech recognition devices of the second and third embodiments makes it possible to perform speech recognition while inheriting long-term context information in the self conditioning CTC technique.

[0032] Here, technical elements used in each embodiment of the present invention will be described.

[0033] [Technical element 1] CTC (Connectionist Time Series Classification) The CTC speech recognition model is an end-to-end speech recognition model that directly learns the relationship between speech and character strings without using phonetic symbols between the speech and character strings. In CTC speech recognition, the output symbol string is not generated by adding a blank string. <blank>By inserting a blank symbol and allowing the same symbol to be output consecutively, the acoustic feature frame and the output symbol string correspond to each other. The CTC speech recognition model outputs one symbol for each acoustic feature frame. In the CTC speech recognition model, the difference between the acoustic feature frame length and the sequence length between output symbols is determined by inserting a blank symbol. <blank>The system absorbs and learns by outputting characters in succession.

[0034] For example, the alignment of the output string "weather" for four frames of acoustic features is as follows: <blank>is replaced with an "_" (underscore). 1:_ _ Weather 2: Weather 3: Weather 4:Weather 5: Weather 6:Weather _ _ 7:_ Weather 8: Weather 9: Heavenly Heavenly Spirit 10: Weather 11:_ Weather 12: Weather 13:Weather 14: Weather 15: Weather 16: Weather 17: Heaven, heaven, heaven

[0035] When making inferences using the CTC speech recognition model, consecutive identical characters in the symbol string output by the CTC speech recognition model are compressed into a single character and blank symbols are deleted, resulting in a recognition result. However, if the correct recognized character string contains a sequence of identical characters, blank symbols must be inserted between those identical characters. As an example, if the correct recognized character string is "Ii Tenki" (fine weather), this string contains a sequence of identical characters "i". In this case, a blank symbol must be inserted between each "i".

[0036] The training of the CTC speech recognition model is as follows: For the output vector h (bold type) calculated from the final layer of the neural network in CTC speech recognition, the probability of correspondence with the output symbol string, i.e., alignment a (bold type), is calculated. Here, h (bold type) is expressed by equation (1), and a (bold type) is expressed by equation (2).

[0037]

number

[0038]

number

[0039] Here, T is the dimension of the output vector h (bold type). Also, V is the set of symbols that can be included in the output symbol. The probability of alignment a (bold type) in the output vector h (bold type) is calculated by the following formula (3).

[0040]

number

[0041] The sum of the probabilities of the output symbol y (in bold) over all possible alignments is given by equation (4) below.

[0042]

number

[0043] However, here β -1 (y(bold)) is the set of all possible alignments of output symbol y(bold) with h(bold). The CTC loss L is calculated using logarithms as in equation (5) below.

[0044]

number

[0045] The CTC speech recognition model is trained by updating (adjusting) the parameters of the neural network to minimize this loss L.

[0046] [Technical element 2] Self conditioning CTC Self-conditioning CTC was originally proposed as a method for performing speech recognition on an utterance-by-utterance basis rather than real-time speech recognition. Self-conditioning CTC was proposed as a method for improving the speech recognition accuracy of the CTC speech recognition model described above. In self-conditioning CTC, a CTC loss function is also set in the intermediate layer, and the model is trained based on the weighted sum of the CTC loss in the final layer and the CTC loss in the intermediate layer.

[0047] The probability vector for the output symbol calculated in the hidden layer is added to the hidden layer of the model after linear transformation processing. That is, it is as follows. Let the output vector of any l (el)th encoder layer be h l out (bold). If we express the layer normalization process as LayerNorm(·), the linear transformation process as Linear(·), and the softmax process as Softmax(·), then the input h l+1 in (Bold) are calculated using equations (6), (7), and (8) below.

[0048]

number

[0049]

number

[0050]

number

[0051] In other words, the Self-conditioning CTC stores information about the characters to be output in the middle layer. l (bold) is also used to improve the accuracy of the recognition results output in the final layer.

[0052] [Technical element 3] Streaming transformer encoder The streaming transformer model consists of an encoder and a decoder. The encoder receives acoustic features on a block-by-block basis. The streaming transformer uses a transformer structure to perform real-time speech recognition. The acoustic feature x of the bth block is (b) (bold) is expressed by the following equation (9). block is the block size, and L hop is the interval between block processing starts.

[0053]

number

[0054] In addition to the above acoustic features, the streaming transformer model also receives context embedding. The context embedding is the above acoustic feature x (b) (bold) or by averaging the acoustic features x (b) The context embedding c1 is input to the first layer of the speech recognition model. (b) (Bold) is expressed by the following formula (10). Note that Average(·) is the operation to calculate the average value, and Maxpool(·) is the max pooling operation. (b-1,j) is the positional encoding, and j is the index for the elements of the vector. model is the dimensionality of the context embedding.

[0055]

number

[0056] Note that positional encoding and Average(x (b) (bold)) or positional encoding and Maxpool(x (b) (boldface)) may be added. Hereafter, PE (b,j) Let us consider a vector consisting of elements of PE (b) It shall be written as (bold).

[0057] In other words, the streaming transformer model inherits the acoustic feature information contained in previous blocks as context embedding through self-attention.

[0058] Let the query, key, and value of the self-attention layer of the bth block in the first layer of the speech recognition model be Q1, respectively. (b) (bold), K1 (b) (bold), and V1 (b) (bold). In the streaming transformer, these queries, keys, and values ​​are combined with context embeddings to generate new queries, keys, and values, and processed using self-attention. The new queries, keys, and values ​​in the first layer are expressed as in Equation (11) below.

[0059]

number

[0060] Let the query, key, and value of the self-attention layer of the b-th block of the l-th encoder layer be Q l (b) (bold), K l (b) (bold), and V l (b) (bold). In the second and subsequent layers, the information (context information) from the previous (past) block is inherited, so the query, key, and value are as shown in formulas (12) and (13) below.

[0061]

number

[0062]

number

[0063] As described above, information on acoustic features in past blocks is carried over as context embedding through self-attention.

[0064] [First embodiment] FIG. 1 is a block diagram showing a schematic functional configuration of a speech recognition device according to a first embodiment. As shown in the figure, the speech recognition device 1 includes, in order from the input side, an encoder layer 101, an encoder layer 102, a layer normalization layer 111, a softmax layer 112, an adder 114, and an encoder layer 121. In the configuration of the speech recognition device 1, the encoder layer 101, the encoder layer 102, the layer normalization layer 111, and the encoder layer 121 are realized using a neural network. That is, the encoder layer 101, the encoder layer 102, the layer normalization layer 111, and the encoder layer 121 have parameters therein, and these parameters are updatable. That is, the speech recognition device 1 can perform machine learning (optimization of parameter values) using training data.

[0065] At least a part of each functional unit constituting the voice recognition device 1 can be realized, for example, by a computer and a program. Furthermore, each functional unit has a storage means as necessary. The storage means is, for example, a variable in the program, or a memory allocated by the execution of the program. Furthermore, non-volatile storage means such as a magnetic hard disk device or a solid state drive (SSD) may be used as necessary. Furthermore, at least a part of the functions of each functional unit may be realized as a dedicated electronic circuit rather than a program.

[0066] Information input to the speech recognition device 1 is a sequence (time series) of acoustic features. In FIG. 1, reference numeral 201 denotes an acoustic feature. For example, a log-mel spectrogram or the like can be used as the acoustic feature. The acoustic feature is expressed as a multidimensional numerical vector. A context embedding 202 (context information) is also input to the speech recognition device 1. The context embedding 202 is information calculated based on past acoustic features. Note that the method for calculating the acoustic feature from the speech to be recognized by the speech recognition device 1 (for example, sampling data representing a speech waveform) can itself be an existing technology.

[0067] In the illustrated example, the encoder layer 101, which is the lowest layer of the network of the speech recognition device 1, has N acoustic features (vectors) and N ahead The acoustic features to be looked-ahead and the context embedding 202 are input. ahead are positive integers. For example, N=64, N ahead = 16, etc., where N and N ahead The values ​​of are not limited to the values ​​exemplified here. Note that, in the time series arrangement of the acoustic features in Fig. 1, the left side is a relatively past acoustic feature, and the right side is a relatively future acoustic feature.

[0068] In the configuration shown in FIG. 1, the encoder layers 101, 102, and 121 constitute a streaming transformer. In general, the number of encoder layers (the number of layers) constituting the streaming transformer is L, where L is an integer equal to or greater than 2. Each encoder layer is configured using a neural network, and the internal parameters thereof can be optimized by machine learning using training data. In the configuration shown in FIG. 1, L=3.

[0069] In other words, the streaming transformer has a layered configuration of L encoder layers (L is an integer equal to or greater than 2), inputs a sequence of acoustic features to be recognized, and inputs context embedding, which is information about acoustic features that exist prior to the sequence of acoustic features to be recognized, and outputs a recognition result corresponding to the sequence of acoustic features to be recognized.

[0070] In addition, among the components shown in FIG. 1, the layer normalization layer 111, the softmax layer 112, and the adder 114 constitute a self-conditioning CTC processing unit. The self-conditioning CTC processing unit converts the hidden layer signal (h l out ), a signal representing probability distribution information about the symbols constituting the recognition result is generated based on the intermediate layer signal, and the intermediate layer signal and the signal representing the probability distribution information about the symbols are mixed to generate an input signal (h l+1 in ) In the example configuration shown in FIG. 1, L=3 and l=2. However, in general, the values ​​of L and l are not limited to these. The layer normalization layer 111, the softmax layer 112, and the addition unit 114 are configured using a neural network, and their internal parameters can be optimized by machine learning using training data.

[0071] In addition, among the encoder layers in the lth (l=1, 2, . . . , L-1) layer, the encoder layers other than the encoder layer that passes a signal to the self-conditional CTC processing unit (encoder layer 102 in the example shown in Figure 1) pass the output signal directly to the next encoder layer.

[0072] The encoder layer 101, the encoder layer 102, and the encoder layer 121 are the first, second, and third encoder layers, respectively. Each encoder layer is realized by a configuration similar to that of the streaming transformer described above. The signal h1 output from the encoder layer 101 out is the signal h2 input to the encoder layer 102. in The signal h2 output from the encoder layer 102 is out is input to the self-conditional CTC processor. Signal h3 output from the self-conditional CTC processor in is input to the encoder layer 121.

[0073] The layer normalization layer 111 normalizes the intermediate layer signal (h l out ) output from the encoder layer 102. In the example configuration shown in FIG. 1, the layer normalization layer 111 performs layer normalization of the intermediate layer signal (h l out The layer normalization layer 111 performs layer normalization of the signal z l norm Output.

[0074] The softmax layer 112 calculates the output signal z l norm Linearly transform z l norm ) and then performs softmax processing to output probability distribution information. The signal output by the softmax layer 112 is z l It is.

[0075] The adder 114 is also called a “mixer.” The adder 114 mixes the output (z l norm ) and the output from the softmax layer 112 (z l ) and the input signal (h l+1 in In the example configuration shown in FIG. 1, the encoder layer 102 is the (l+1)th encoder layer.

[0076] That is, the adder 114 mixes the signal representing the probability distribution information about the symbols constituting the recognition result with the intermediate layer signal. Specifically, the adder 114 mixes the signal representing the probability distribution information with the intermediate layer signal by adding them together, as shown in equation (8).

[0077] As shown in Fig. 1, the speech recognition device 1 is based on the model structure of a streaming transformer encoder and processes acoustic features and context embedding for each block. The speech recognition device 1 carries over information on acoustic features in past blocks as context embedding through self-attention.

[0078] However, the feature of the speech recognition device 1 (the difference from the streaming transformer) is that the intermediate layer of the streaming transformer performs calculations using the Self Conditioning CTC method shown in equations (6), (7), and (8).

[0079] The speech recognition device 1 is configured to optimize the values ​​of internal parameters by machine learning. The training data is a set of pairs of a sequence of training acoustic features and a supervised text corresponding to the sequence of training acoustic features. By using the training data, the speech recognition device 1 inputs the sequence of training acoustic features of the training data to the streaming transformer and calculates the loss between the recognition result output from the streaming transformer and the supervised text of the training data. The speech recognition device 1 is configured to enable optimization processing of the values ​​of internal parameters of the streaming transformer and the values ​​of internal parameters of the self-conditional CTC processing unit based on this calculation.

[0080] The loss used when training the model (self-conditioning CTC loss function) may be a weighted sum of the CTC loss in the final layer and the CTC loss in the intermediate layer.

[0081] The final speech recognition result by the speech recognition device 1 is obtained by the output from the final layer (encoder layer 121). By using the Self Conditioning CTC method, the speech recognition device 1 performs speech recognition processing while taking into account information about characters that cannot be taken into account by normal CTC.

[0082] Note that the (l+1)th and subsequent context embeddings c l+1 (b) (Bold) indicates the input to the (l+1)th layer, h l+1 in (bold) are used to calculate again using the following formula (14). Average(·) is a process that calculates the average value within a block.

[0083]

number

[0084] As shown in equation (14), in the speech recognition device 1, information on the probabilities of characters in the intermediate layer is also carried over to the next block as context information.

[0085] [About look ahead] The speech recognition device 1 of this embodiment performs look-ahead. That is, the speech recognition device 1 performs speech recognition processing while also referring to acoustic features at a time before (after) the acoustic feature to be recognized. As an example, when the number of acoustic features per block is 64, the speech recognition device 1 does not output the recognition results for the first 16 acoustic features (i.e., 16 acoustic features that follow the 64 acoustic features in time), but only uses them for attention calculation of other acoustic features. These 16 acoustic features at a time before (after) become targets for outputting the recognition results in the next block. This method of using acoustic features at a time before (after) the acoustic feature to be recognized is called look ahead. In the above example, the number of acoustic features to be looked ahead is 16. The streaming transformer can also use the look ahead mechanism. The look ahead method is also used in this embodiment. In Figure 1, the number of acoustic features per block is denoted as N, and the number of acoustic features to look ahead is denoted as N ahead It is expressed as follows.

[0086] A feature of this embodiment is that look ahead is used even in the (l+1)th layer and onwards after applying the self conditioning CTC. The self conditioning CTC adds the recognition probability for characters in the intermediate layer to the intermediate layer of the model. However, the recognition probability for the acoustic features of the part that is the target of look ahead can only be obtained after the processing of the next block. In other words, in this embodiment, a delay of one block occurs because look ahead is performed even in the (l+1)th layer and onwards after applying the self conditioning CTC. On the other hand, the advantage of the mechanism of this embodiment, which looks ahead even in the (l+1)th layer and onwards after applying the self conditioning CTC, is that speech recognition can be performed while taking into account the future recognition probability. With this mechanism, it is expected that accurate speech recognition results can be output in the final layer while taking into account not only the recognition probability in the intermediate layer of the current block but also the recognition probability in the intermediate layer of future blocks.

[0087] That is, in this embodiment, the sequence of acoustic features to be recognized is a block including N acoustic features (N is a positive integer). The encoder layer, which is the first layer of the streaming transformer, receives the acoustic features of the block to be recognized and also receives the N acoustic features included in the blocks further in the future than the block. ahead pieces(N ahead The (l+1)-th layer of the streaming transformer (encoder layer 121 in FIG. 1 ) inputs the signal of the block to be recognized passed from the self-conditional CTC processor, and also reads ahead and inputs the signal passed from the self-conditional CTC processor based on a block further in the future than the block.

[0088] FIG. 2 is a schematic diagram showing an example of the relationship between blocks and acoustic features in block-by-block processing by the speech recognition device 1. The acoustic features are numbered 1, 2, 3, ... from the past to the future. Similarly, the blocks are numbered 1, 2, 3, ... from the past to the future. In the example shown, the number of acoustic features (N) included in a block to be processed for speech recognition is 64, and the number of acoustic features to be looked ahead (N ahead ) is set to 16, and the block processing start interval (L hop ) is set to 16. However, N, N ahead , and L hop may be any positive integer.

[0089] As shown in the figure, in this example, in block numbered 1 (first block), the acoustic features to be recognized are numbered 1 to 64, and the acoustic features to be looked ahead are numbered 65 to 80. In block numbered 2 (second block), the acoustic features to be recognized are numbered 17 to 80, and the acoustic features to be looked ahead are numbered 81 to 96, and the context embedding is generated based on the acoustic features numbered 1 to 64. In block numbered 3 (third block), the acoustic features to be recognized are numbered 33 to 96, and the acoustic features to be looked ahead are numbered 97 to 112, and the context embedding is generated based on the acoustic features numbered 17 to 80. The same applies to blocks numbered 4 and onwards.

[0090] In general, in a block numbered b (b is a positive integer) (the bth block), the number of the acoustic feature to be recognized is (b-1)L hop +1, (b-1)L hop +N. The number of the acoustic feature to be looked ahead is (b-1)L hop +N+1, (b-1)L hop +N+N ahead In addition, when b ≥ 2, the number of the acoustic features used to generate the context embedding is (b-2)L hop From +1, (b-2)L hop Up to +N.

[0091] As described above, according to the first embodiment, the self-conditional CTC processor generates a signal representing the probability distribution information for the symbols constituting the recognition result based on the signal of the intermediate layer of the encoder layer, and feeds back the probability distribution information to the network of the speech recognition model again. This makes it possible to reduce the recognition errors of the speech recognition device of the first embodiment.

[0092] [Second embodiment] Next, a second embodiment of the present invention will be described. Note that the matters already described in the previous embodiment may not be described below. Here, the matters unique to this embodiment will be mainly described.

[0093] In the second embodiment, the recognition accuracy of the method described in the first embodiment is further improved by introducing an RNN. That is, in the second embodiment, the output from the intermediate layer (output probability for a character) z l By applying an RNN to (bold), time series information is incorporated and processed.

[0094] 3 is a block diagram showing a schematic functional configuration of a speech recognition device according to a second embodiment. As shown in the figure, the speech recognition device 2 includes, in order from the input side, an encoder layer 101, an encoder layer 102, a layer normalization layer 111, a softmax layer 112, an RNN layer 113, an adder 114, and an encoder layer 121. The RNN layer 113 is configured using an RNN (Recurrent Neural Network).

[0095] In the configuration of the speech recognition device 2, the encoder layer 101, the encoder layer 102, the layer normalization layer 111, the RNN layer 113, and the encoder layer 121 are realized using neural networks. That is, the encoder layer 101, the encoder layer 102, the layer normalization layer 111, the RNN layer 113, and the encoder layer 121 each have internal parameters, and these parameters are updatable.

[0096] The speech recognition device 2 according to this embodiment is characterized in that the self-conditional CTC processing unit includes an RNN layer 113 .

[0097] The RNN layer 113 is also called a “recurrent neural network layer.” The RNN layer 113 calculates the output (z l ) and the signal (z l rnn ) The RNN layer 113 also performs processing using a recurrent neural network based on the latent vector obtained as a result of processing the signal of the past (previous) block. The RNN layer 113 stores the latent vector obtained as a result of processing the signal of the block, and uses it in processing the signal of the next block. In FIG. 3, "M" input to the RNN layer 113 and output from the RNN layer 113 indicates that this latent vector is temporarily stored. In other words, the RNN layer 113 carries over information from the past side to the future side as a latent vector.

[0098] It should be noted that the RNN layer 113 can be realized using an RNN such as a Long Short-Term Memory (LSTM).

[0099] Other functions of the speech recognition device 2 are similar to or equivalent to those of the speech recognition device 1 in the first embodiment. That is, the encoder layer 101, the encoder layer 102, the layer normalization layer 111, the softmax layer 112, the adder 114, and the encoder layer 121 have the same functions as those in the first embodiment. However, in the speech recognition device 2 of this embodiment, the output from the softmax layer 112 is passed to the RNN layer 113. Also, the adder 114 (mixer) mixes the signal z l norm and the signal z l rnn The adder 114 mixes the signal resulting from the addition with the input signal (h l+1 in )

[0100] Specifically, the processes performed in each layer from the layer normalization layer 111 to the addition unit 114 are as expressed by the following equations (15), (16), (17), and (18).

[0101]

number

[0102]

number

[0103]

number

[0104]

number

[0105] In other words, as shown in the above equation (15), the layer normalization layer 111 normalizes the signal h l out After normalization (bold), the signal z l norm As shown in the above equation (16), the softmax layer 112 outputs the signal z l norm After performing the linear transformation process shown in (bold), the signal z l As shown in the above equation (17), the RNN layer 113 outputs the signal z l Based on (bold), the RNN is processed and the signal z l rnn As shown in the above equation (18), the adder 114 outputs the signal z l norm (bold) and the signal z output from the RNN layer 113 l rnn (bold) is linearly transformed and added to the signal h l+1 in (bold) This signal is output as l+1 in (Bold) is input to the (l+1)th layer, encoder layer 121.

[0106] As described above, when switching between blocks, the latent vector output from the RNN layer 113 at the end of the current block is input to the RNN layer 113 as a long-term memory for processing the next block (signal M in FIG. 3). In other words, the RNN layer 113 makes it possible to propagate past information across blocks to the future. Signal M is stored across blocks.

[0107] As described above, according to the second embodiment, in addition to the functions and effects of the first embodiment, the RNN layer 113 inputs the latent vector output from the RNN layer 113 at the end of the past block for processing in the RNN layer 113 of the next block. That is, it is possible to propagate past information across blocks.

[0108] [Third embodiment] Next, a third embodiment of the present invention will be described. Note that the description of the matters already described in the previous embodiments may be omitted below. Here, the description will focus on matters unique to this embodiment.

[0109] The third embodiment is characterized in that, instead of the processing of the adder 114 in the configuration of the second embodiment (the above equation (18)), a weighting coefficient (scalar) generated by the model is applied to each term in the addition. By using such a coefficient, the model itself determines how much importance is to be given to the first and second terms in the addition of equation (18). In this way, the third embodiment aims to improve recognition accuracy.

[0110] 4 is a block diagram showing a schematic functional configuration of a speech recognition device according to a third embodiment. As shown in the figure, the speech recognition device 3 includes, in order from the input side, an encoder layer 101, an encoder layer 102, a layer normalization layer 111, a softmax layer 112, an RNN layer 113, a merge layer 115, and an encoder layer 121. The merge layer 115 merges a signal output from the layer normalization layer 111 and a signal output from the RNN layer 113 based on a predetermined ratio (α). The value of α is calculated inside the merge layer 115.

[0111] The merge layer 115 is also called the “mixer.” The merge layer 115 combines the output (z l norm ) and the output from RNN layer 113 (z l rnn ) and perform linear transformation processing to calculate a one-dimensional coefficient (α), and the output (z l norm ) and the output from RNN layer 113 (z l rnn ) is mixed (weighted and added) with the (l+1)th layer (encoder layer 121) of the streaming transformer. l+1 in The internal configuration of the merge layer 115 will be described with reference to FIG.

[0112] 5 is a block diagram showing the internal functional configuration of the merge layer 115 in this embodiment. As shown in the figure, the merge layer 115 includes, from the input side, a concatenation layer 151, a linear transformation layer 152, a sigmoid function calculation unit 153, and a weighted addition unit 154.

[0113] The concatenation layer 151 calculates the output (z l norm ) and the output from RNN layer 113 (z l rnn ) are concatenated. The process performed by the concatenation layer 151 is to concatenate the columns of the elements of the two vectors. As a result, the concatenation layer 151 outputs one vector.

[0114] The linear transformation layer 152 performs linear transformation based on one vector passed from the concatenation layer 151 to obtain one-dimensional coefficients. The linear transformation layer 152 is configured using a neural network. The parameters (weights) at each node of this neural network can be optimized by learning.

[0115] The sigmoid function calculation unit 153 applies a sigmoid function to the one-dimensional coefficient calculated by the linear transformation layer 152. As a result, the sigmoid function calculation unit 153 outputs a one-dimensional coefficient (α) that is greater than or equal to 0.0 and less than or equal to 1.0. The sigmoid function calculation unit 153 passes the value of the coefficient α to the weighted addition unit 154.

[0116] The weighted addition unit 154 uses the value α passed from the sigmoid function calculation unit 153 as a weighting coefficient to add the output (z l rnn ) and the output (z l norm ) is added. A specific example of the calculation performed by weighted addition section 154 is shown in equation (24) below.

[0117] The specific process of this embodiment is as follows. The layer normalization layer 111 shown in FIG. 4 normalizes the output signal h l out Based on (bold), layer normalization is performed, and z l norm (bold). The layer normalization layer 111 outputs this output z l norm (bold) is passed to the softmax layer 112 and the merge layer 115. That is, as shown in the following equation (19).

[0118]

number

[0119] The softmax layer 112 is a layer that normalizes the signal z l norm (bold type), a linear transformation process is performed, and the result is then processed using a softmax function. As a result of this process, the softmax layer 112 converts the signal z l (bold). The softmax layer 112 outputs this signal z l (bold) is passed to the RNN layer 113. That is, as shown in the following equation (20).

[0120]

number

[0121] The RNN layer 113 receives the signal z l Based on (bold), the RNN is processed and the resulting signal z l rnn (bold). The RNN layer 113 outputs this signal z l rnn (bold) is passed to the merge layer 115. That is, as shown in the following equation (21).

[0122]

number

[0123] The merge layer 115 performs the following process. That is, the concatenation layer 151 shown in FIG. 5 performs the following process on the signal z l norm The vectors in bold and the signal z passed from the RNN layer 113 l rnn (bold) and the signal z l concat (bold). The concatenation layer 151 outputs this signal z l concat (bold type) is passed to the linear transformation layer 152. That is, as shown in the following equation (22). Note that Concat(·,·) is a function that combines vectors, which are arguments.

[0124]

number

[0125] The linear transformation layer 152 converts the signal z l concat The linear transformation layer 152 receives the signal Linear(z l concat (bold)) is passed to the sigmoid function calculation unit 153. l concat The number of dimensions of the above Linear(z l concat The sigmoid function is calculated using the value of (bold) as an input, and the resulting value α is calculated. That is, the processing of the linear transformation layer 152 and the sigmoid function calculation unit 153 is as shown in the following equation (23). Note that Sigmoid(·) is the sigmoid function.

[0126]

number

[0127] The lower limit of the value of α output from the sigmoid function calculation unit 153 is 0.0, and the upper limit is 1.0. In other words, inside the merge layer 115, the signal z l norm The vectors in bold and the signal z passed from the RNN layer 113 l rnn The value of α is calculated according to the vector (bold). The parameter values ​​of the weights (weights for linear conversion) of the linear transformation layer 152 are optimally determined by learning. In other words, inside the merge layer 115, an appropriate value of α is found based on the learning results.

[0128] The weighted addition unit 154 uses the above value α to calculate the signal z l norm The vectors in bold and the signal z passed from the RNN layer 113 l rnn The vectors in (bold) are weighted at a predetermined ratio and added together. Specifically, the weighted addition unit 154 performs calculations according to the following equation (24) to obtain h l+1 in (bold) is calculated.

[0129]

number

[0130] In other words, the above value α is the signal z l norm The vectors in bold and the signal z passed from the RNN layer 113 l rnn This is a factor for determining how much importance to give to the vector (in bold).

[0131] The weighted addition unit 154 adds the calculated signal h l+1 in (bold) is output. l+1 in (Bold) is the input to the (l+1)th layer, encoder layer 121.

[0132] In the processing expressed by the above equations (19) to (23), the one-dimensional coefficient α is generated based on a simplex vector, but the one-dimensional coefficient α may be generated from multiple time series vectors.

[0133] FIG. 6 is a block diagram showing an example of the internal configuration of the voice recognition device according to the first to third embodiments (including modified examples). The voice recognition devices 1, 2, and 3 can be realized using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions included in a program read from the RAM 902 or the like. The central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations according to each instruction. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. The RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with an external input / output device or the like. The input / output devices 904 and 905 exchange data with the central processing unit 901 via the input / output port 903. The bus 906 is a common communication path used inside the computer. For example, the central processing unit 901 reads and writes data from and to the RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port 903 via the bus 906.

[0134] At least some of the functions of the voice recognition device in each of the above-mentioned embodiments (including the modified examples) can be realized by a computer and a program. In that case, the program for realizing the functions may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the functions. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to a portable medium such as a flexible disk, an optical magnetic disk, a ROM, a CD-ROM, a DVD-ROM, a USB memory, and a storage device such as a hard disk built into a computer system. In other words, the term "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, the term "computer-readable recording medium" may include a medium that temporarily and dynamically holds a program, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. Furthermore, the above-mentioned program may be a program for realizing some of the above-mentioned functions, and may further be a program that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0135] Although a number of embodiments have been described above, the present invention can also be embodied in the following modified examples. Note that a number of embodiments and modified examples may be combined to the extent that they are possible to combine.

[0136] [Variation 1: Number of Encoder Layers] In each of the speech recognition devices according to the first to third embodiments, the number of encoder layers is three (encoder layers 101, 102, and 121). As a modification, an encoder having any number of layers (two or more) may be used.

[0137] [Variation 2: Presence or absence of look ahead in the (l+1)th layer and onwards] In each of the speech recognition devices according to the first to third embodiments, look ahead is used in the (l+1)th and subsequent layers. As a modification, look ahead may not be used in the (l+1)th and subsequent layers.

[0138] The speech recognition device according to any one of the first to third embodiments (including the modified examples) described above inputs a block of multiple acoustic features into a model, and outputs a recognition result for each acoustic feature in real time.

[0139] Furthermore, the speech recognition device according to any one of the first to third embodiments (including the modified examples) performs speech recognition processing while inheriting information on blocks that have previously been subjected to speech recognition processing by using context embedding.

[0140] In addition, the speech recognition device of any of the first to third embodiments (including the modified examples) also performs speech recognition in the intermediate layer by the method of self conditioning Connectionist Temporal Classification (self-conditioning CTC), and feeds back the probability distribution vector (probability distribution information) related to the characters (symbols) output from the intermediate layer to the network of the speech recognition model again. As a result, the speech recognition device outputs the recognition result while taking the context into consideration.

[0141] Moreover, the voice recognition devices of the second and third embodiments input the probability distribution vector (probability distribution information) regarding the above-mentioned characters (symbols) to a recurrent neural network (RNN) for processing.

[0142] The speech recognition device of the third embodiment also concatenates the fed-back probability distribution vector (probability distribution information) with the vector in the intermediate layer of the network, and processes the resulting vector in a linear transformation layer to generate a scalar value α. By multiplying the fed-back probability distribution vector (probability distribution information) by this scalar value α, the speech recognition model itself can select (or weight) whether to prioritize the probability distribution vector or the vector in the intermediate layer.

[0143] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included.

[0144] [Demonstration experiment 1] Demonstration experiments were conducted to verify the effects of each embodiment. Demonstration experiment 1 evaluated how the first embodiment improves the problem of erroneously repeatedly outputting past utterances, which occurs in the conventional technology (Streaming transformer). The speech recognition device of the first embodiment was compared with the conventional technology (Streaming transformer). The learning specifications are as follows:

[0145] The specifications of the model realizing the speech recognition device 1 of the first embodiment are 15 conformer layers, 8 attention heads, the number of dimensions of the linear transformation layer is 2048, and the number of output dimensions is 512. Note that self conditioning is performed between the 12th layer and the 13th layer.

[0146] The specifications of the conventional technology (Streaming transformer) were as follows: the encoder had 12 conformer layers, 8 attention heads, a linear transformation layer with 2048 dimensions, and an output dimension of 512. The decoder had 3 transformer layers, 8 attention heads, a linear transformation layer with 1024 dimensions, and an output dimension of 512.

[0147] The specifications common to the speech recognition device of the first embodiment and the conventional technology (Streaming transformer) are as follows: The number of acoustic features in one block, N, is 64, and the number of acoustic features to be looked ahead, N ahead was set to 16, and encoding was performed while shifting the blocks by 16 acoustic features (L hop =16). As training data, 4,500 hours of audio from programs previously broadcast by the Japan Broadcasting Corporation (NHK) and the corresponding subtitle text were used. As evaluation data, 10 hours of audio from programs previously broadcast by NHK and the manually transcribed text were used. In addition, an 80-dimensional log-mel filter bank was used as the acoustic feature. The output words were 4,500 types of word pieces.

[0148] The results of the demonstration experiment 1 are as follows: The character error rate (CER, in percent) in the first embodiment and the conventional technique are as shown in Table 1 below.

[0149] [Table 1]

[0150] As shown in Table 1, the character error rate when the method of the first embodiment was used was improved by 0.6 points compared to the character error rate of the conventional technique (Streaming transformer).

[0151] Specific examples of improvements to recognition results are as follows. Note that the words in parentheses are recognition errors. Evaluation speech: Prefectural Museum of Nature and Human Activities, Sanda City, Prefectural Oriental White Stork Park Conventional technology (Streaming transformer): Prefectural Kouno "Tori" no "Sato" Park Prefectural Kouno and Nature Museum in Sanda City First implementation: Prefectural "Stork Village" Park, Prefectural Museum of Nature and Human Activities in Sanda City

[0152] In both the conventional technology and the first embodiment, there are errors in the writing of hiragana and katakana, and errors in homonyms such as "go" and "sato." In the recognition results of the conventional technology (Streaming transformer), there is a misrecognition in which a previously spoken output of "kouno" is repeated and output in a place that is unrelated to the spoken voice. On the other hand, in the first embodiment, there is no error in repeating the output. Moreover, not only in the above utterance, but also overall (in other evaluation voices), in the first embodiment, there is no error in repeating the output.

[0153] [Demonstration experiment 2] We verified whether the recognition accuracy in each of the second and third embodiments is further improved compared to the first embodiment. In the demonstration experiment 2, the Corpus of Spontaneous Japanese (CSJ) was used as the training data and the evaluation data.

[0154] The model specifications are as follows: The specifications of the first embodiment are as explained in Demonstration Experiment 1 above. The specifications of the second embodiment are similar to those of the first embodiment described above in Demonstration Experiment 1. However, the RNN module introduced in the second embodiment is a single layer of Long short-term memory (LSTM) having a 1024-dimensional unit. The specifications of the third embodiment are the same as those of the second embodiment. However, the linear transformation module of formula (23) in the third embodiment is configured with two layers. In each of the two layers, projection from 2048 dimensions to 1024 dimensions and projection from 1024 dimensions to 1 dimension are performed.

[0155] The training data was 520 hours of speech and text pairs from CSJ. The evaluation data was eval1, eval2, and eval3 from CSJ. An 80-dimensional log-mel filter bank was used as the acoustic feature. The output words were 4500 types of word pieces.

[0156] The results of the demonstration experiment 2 are as follows: The character error rate (CER, unit is percent) in each of the first embodiment, the second embodiment, and the third embodiment is as shown in Table 2 below.

[0157] [Table 2]

[0158] Table 2 shows the character error rate (CER) for each of the evaluation sets eval1, eval2, and eval3 for the first, second, and third embodiments. As shown in this table, the second embodiment recorded a better character error rate than the first embodiment in all the evaluation sets eval1, eval2, and eval3. On the other hand, the method of the third embodiment has an improved character error rate compared to the first embodiment, but the character error rate is worse in the evaluation sets eval2 and eval3 when compared to the second embodiment.

[0159] In the third embodiment, the one-dimensional coefficient α is calculated in equations (19) to (23), but it may be possible to improve the recognition accuracy by generating a one-dimensional coefficient based on multiple time-series vectors. [Industrial Applicability]

[0160] The present invention can be widely used in various industries for the purpose of automatic speech recognition, etc. However, the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0161] 1,2,3 Voice Recognition Device 101 Encoder Layer 102 Encoder Layer 111 Layer Normalization Layer 112 Softmax Layer 113 RNN layer 114 Addition section (mixing section) 115 Merge layer (mixing part) 121 Encoder Layer 151 Connected layer 152 Linear Transformation Layer 153 Sigmoid function calculation unit 154 Weighted addition unit 201 Acoustic Features 202 Context Embedding (Context Information) 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus< / blank> < / blank> < / blank>

Claims

1. a streaming transformer having L encoder layers (L is an integer equal to or greater than 2) in a layered configuration, receiving a sequence of acoustic features to be recognized and a context embedding which is information on acoustic features from a time before the sequence of acoustic features to be recognized, and outputting a recognition result corresponding to the sequence of acoustic features to be recognized; The hidden layer signal (h ｌ ｏｕｔ ), a signal representing probability distribution information about symbols constituting the recognition result is generated based on the intermediate layer signal, and the intermediate layer signal and the signal representing the probability distribution information about the symbols are mixed to generate an input signal (h ｌ＋１ ｉｎ A self-conditional CTC processor that provides A voice recognition device comprising:

2. the streaming transformer is configured to use training data, which is a set of pairs of a sequence of training acoustic features and a superscript text corresponding to the sequence of training acoustic features, to input the sequence of training acoustic features of the training data to the streaming transformer, and to enable optimization processing of values ​​of internal parameters of the streaming transformer and values ​​of internal parameters of the self-conditional CTC processing unit based on a loss between a recognition result output from the streaming transformer and the superscript text of the training data; 2. The speech recognition device according to claim 1.

3. The sequence of the acoustic features to be recognized is a block including N acoustic features (N is a positive integer), The encoder layer of the first layer of the streaming transformer inputs the block and also receives N ａｈｅａｄ pieces (N ａｈｅａｄ is a positive integer) and inputs the acoustic features in advance. The encoder layer of the (l+1)th layer of the streaming transformer inputs a signal of the block to be recognized, which is passed from the self-conditional CTC processing unit, and also reads ahead and inputs a signal passed from the self-conditional CTC processing unit based on a block further in the future than the block.

2. The speech recognition device according to claim 1.

4. The self-conditional CTC processing unit includes: The hidden layer signal (h ｌ ｏｕｔ ) A layer normalization layer that performs layer normalization of the layer The output from the layer normalization layer is linearly transformed and softmax processed to obtain the probability distribution information (z ｌ ), and a softmax layer that outputs The output from the layer normalization layer (z ｌ ｎｏｒｍ ) and the output from the softmax layer (z ｌ ) to the encoder layer of the (l+1)th layer of the streaming transformer. ｌ＋１ ｉｎ a mixing unit for outputting the Equipped with 2. The speech recognition device according to claim 1.

5. The self-conditional CTC processing unit includes: The hidden layer signal (h ｌ ｏｕｔ ) A layer normalization layer that performs layer normalization of the layer a softmax layer that outputs the probability distribution information by linearly transforming the output from the layer normalization layer and performing softmax processing; The output from the softmax layer (z ｌ ) is input, and the signal (z ｌ ｒｎｎ ) and a recurrent neural network layer that outputs The output from the layer normalization layer (z ｌ ｎｏｒｍ ) and the output from the recurrent neural network layer (z ｌ ｒｎｎ ) to the encoder layer of the (l+1)th layer of the streaming transformer. ｌ＋１ ｉｎ a mixing unit for outputting the Equipped with the recurrent neural network layer performs processing using the recurrent neural network based also on the latent vector obtained as a result of processing the signal of the previous block by the recurrent neural network layer, and stores the latent vector obtained as a result of processing the signal of the block to be processed for processing the next block; 2. The speech recognition device according to claim 1.

6. The mixing section includes: The output from the layer normalization layer (z ｌ ｎｏｒｍ ) and the output from the recurrent neural network layer (z ｌ ｒｎｎ ) and perform linear transformation processing to calculate a one-dimensional coefficient (α), and the output (z ｌ ｎｏｒｍ ) and the output from the recurrent neural network layer (z ｌ ｒｎｎ ) to the encoder layer of the (l+1)th layer of the streaming transformer. ｌ＋１ ｉｎ ), 6. The speech recognition device according to claim 5.

7. a streaming transformer having L encoder layers (L is an integer equal to or greater than 2) in a layered configuration, receiving a sequence of acoustic features to be recognized and a context embedding which is information on acoustic features from a time before the sequence of acoustic features to be recognized, and outputting a recognition result corresponding to the sequence of acoustic features to be recognized; The hidden layer signal (h ｌ ｏｕｔ ), a signal representing probability distribution information about symbols constituting the recognition result is generated based on the intermediate layer signal, and the intermediate layer signal and the signal representing the probability distribution information about the symbols are mixed to generate an input signal (h ｌ＋１ ｉｎ A self-conditional CTC processor that provides A program for causing a computer to function as a speech recognition device comprising the above-mentioned.

Images