Voice recognition device, voice recognition method and program

Abstract

To provide a voice recognition device, a voice recognition method and a program, which can improve recognition performance even if learning environment of a recognizer differs.SOLUTION: A voice recognition device includes: an utterance section processing unit for determining an utterance section; a voice emphasis unit for determining an emphasis feature amount in which a voice component is emphasized with respect to an acoustic feature amount of an input voice signal using a first model for each frame; a hidden state processing unit for determining a hidden state feature amount sequence based on a target feature amount sequence using a second model; a sampling processing unit for sampling a sample value of the target feature amount sequence for a plurality of times using a third model indicating probability distribution of the target feature amount sequence corresponding to the emphasis feature amount sequence and the acoustic feature sequence in the utterance section, and determining an expectation value of the hidden state feature amount sequence from a sample value of the hidden state feature amount sequence; and an utterance processing unit for determining an utterance content of the utterance section based on the expectation value of the hidden state feature amount sequence using a fourth model, and the third model is probability distribution of beta distribution.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a voice recognition device, a voice recognition method, and a program. [Background technology]

[0002] Speech recognition has a wide variety of applications and is used in a variety of environments. It is known that when a noisy speech is used for speech recognition, the recognition rate is lower than that of a clean speech that is not contaminated by noise. To improve the recognition rate in noisy environments, speech enhancement is sometimes applied to speech recognition systems. With speech enhancement, the speech components of the recorded input speech components are enhanced, and the noise components are relatively reduced. Noise suppression can be considered as one form of speech enhancement.

[0003] Missing data speech recognition processing has been proposed as a method of applying speech enhancement to speech recognition. For example, in the methods described in Non-Patent Documents 1 and 2, an evidence model is applied. The evidence model is a model of the decoding process that provides statistical information from speech enhancement to speech recognition. The evidence model can be regarded as a mathematical model for evaluating the expected value of the classification score that gives the recognition result, and is expressed using a probability density function that is trained to reduce misclassification. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] AC Morris, J. Baker, and H. Bourlard, “FROM MISSING DATA TO MAYBE USEFUL DATA: SOFT DATA MODELLING FOR NOISE ROBUST ASR”, Proceedings of Workshop Innovation Speech Process, 2001 [Non-Patent Document 2] M. Kuhne, R. Togneri, and S. Nordholm, “Recognition with Applications in Reverberant Multi-Source Environments”, IEEE Transactions on Audio, Speech and Language Processing, vol. 19, No. 2, pp. 372-384, 2011 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the conventional technology, although performance improves when applied to a recognizer trained with a clean sound source, there is a problem in that performance does not improve when applied to a recognizer trained with a sound source containing noise.

[0006] The present invention has been made in consideration of the above problems, and has an object to provide a speech recognition device, a speech recognition method, and a program that can improve recognition performance even if the learning environment of the recognizer is different. [Means for solving the problem]

[0007] (1) In order to achieve the above object, a speech recognition device according to one embodiment of the present invention includes a speech section processing unit that determines a speech section based on acoustic characteristics of an input speech signal, a speech emphasis unit that determines, for each frame, emphasis features in which speech components are emphasized for acoustic features of the input speech signal using a first model, a hidden state processing unit that determines a hidden state feature series that is a series of hidden state features based on a target feature series that is a series of target features using a second model, a sampling processing unit that samples sample values ​​of a target feature series corresponding to an emphasis feature series that is a series of the emphasis features in the speech section and an acoustic feature series that is a series of the acoustic features multiple times using a third model that indicates a probability distribution of the target feature series corresponding to the emphasis feature series that is a series of the emphasis features in the speech section and the acoustic feature series that is a series of the acoustic features, and determines an expected value of the hidden state feature series from the sample values ​​of the hidden state feature series, and a speech processing unit that determines a speech content of the speech section based on the expected value of the hidden state feature series using a fourth model, where the third model is a beta distribution.

[0008] (2) Also, in one aspect of the present invention, in the speech recognition device of (1) above, the sampling processing unit may select the parameters of the beta distribution based on a score of a speech recognition result.

[0009] (3) In addition, in one aspect of the present invention, in the speech recognition device according to (1) or (2) above, the speech emphasis unit, the sampling processing unit, and the speech processing unit are The emphasis feature may be treated as a random variable, and the speech emphasis process and the speech recognition process may be integrated using Bayes' theorem.

[0010] (4) Also, one aspect of the present invention is a speech recognition device according to any one of (1) to (3) above, wherein the speech enhancement unit includes a speech enhancement encoder and a speech enhancement decoder, the hidden state processing unit includes a speech recognition encoder, and the utterance processing unit includes a speech recognition decoder, the speech enhancement encoder encodes an observation spectrum of the input speech signal to obtain latent variables, and outputs the latent variables to the speech enhancement decoder, and the speech enhancement decoder decodes the input observation spectrum and the latent variables to estimate speech features. and outputting the estimated acoustic feature to the third model, the third model performing interpolation on the estimated acoustic feature and the observed spectrum by sampling using the probability distribution of the beta distribution and outputting the interpolated estimated acoustic feature to the speech recognition encoder, the speech recognition encoder encoding the interpolated estimated acoustic feature and outputting the hidden state feature sequence to the speech recognition decoder, and the speech recognition decoder decoding the input hidden state feature sequence and outputting a speech recognition result.

[0011] (5) Also, in one aspect of the present invention, in the speech recognition device of (2) above, the sampling processing unit may calculate, for each sampling, a sum of maximum scores of the speech recognition result calculated using a character string to be speech-recognized, parameters of the third model or a smoothed encoded vector when the third model and an observed spectrum of the input speech signal are given, and the observed spectrum, and select parameters of the beta distribution based on the calculated sum.

[0012] (6) In order to achieve the above-mentioned object, a speech recognition method according to one aspect of the present invention is a speech recognition method in a speech recognition device, in which the speech recognition device executes: a speech section processing step of determining a speech section based on acoustic characteristics of an input speech signal; a speech emphasis step of determining, for each frame, emphasis features in which speech components are emphasized for acoustic features of the input speech signal using a first model; a hidden state processing step of determining a hidden state feature series that is a series of hidden state features based on a target feature series that is a series of target features using a second model; a sampling processing step of sampling sample values ​​of the target feature series multiple times using a third model that is a beta distribution probability distribution indicating a probability distribution of an emphasis feature series that is a series of the emphasis features in the speech section and a target feature series corresponding to the acoustic feature series that is a series of the acoustic features, and determining an expected value of the hidden state feature series from the sample values ​​of the hidden state feature series; and a speech processing step of determining a speech content of the speech section based on the expected value of the hidden state feature series using a fourth model.

[0013] (7) To achieve the above object, a program according to one aspect of the present invention is a program that causes a computer of a speech recognition device to function as the speech recognition method of (6) above. Effect of the Invention

[0014] According to one aspect of the present embodiment, even if the learning environment of the recognizer is different, the recognition performance can be improved. According to (1), (4), (5), (6), and (7), a probability model using a beta distribution is used for the third model, so that the distribution shape can be controlled or selected. As a result, according to (1), (6), and (7), the accuracy of speech recognition can be improved whether the model is trained with clean speech or robust noisy speech. According to (2), since the third model is selected using the score of the speech recognition result, the third model can be appropriately selected, and the accuracy of speech recognition can be improved. According to (3), the speech enhancement process and the speech recognition process can be integrated, thereby improving the accuracy of speech recognition. [Brief description of the drawings]

[0015] [Figure 1] 1 is a block diagram showing an example of the configuration of a voice recognition device according to an embodiment; [Diagram 2] The graph model of εu(s1:T) and samples extracted from it are shown. [Diagram 3] FIG. 13 is a diagram showing the feature curve and its projection onto the evidence model of this embodiment. [Figure 4] FIG. 13 is a diagram showing an example of samples extracted from each parameter setting of a beta distribution. [Diagram 5] FIG. 1 shows an overview of the overall model and the sampling process. [Figure 6] 4 is a flowchart showing a first example of a speech recognition process according to the embodiment. [Figure 7] 11 is a flowchart showing a second example of the voice recognition processing example according to the embodiment. [Figure 8] FIG. 13 is a diagram showing character error rates under each SNR condition using three speech recognition models to evaluate unsupervised model selection. [Figure 9] FIG. 13 is a diagram showing detailed character error rates for a test set under each SNR condition using a robust speech recognition model. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the drawings used in the following description, the scale of each component is appropriately changed so that each component is of a recognizable size. In addition, in all the drawings for explaining the embodiments, the same reference numerals are used for the parts having the same functions, and the repeated explanation is omitted. In addition, "based on XX" in this application means "based on at least XX," and includes cases where it is based on other elements in addition to XX. Furthermore, "based on XX" is not limited to cases where XX is used directly, but also includes cases where it is based on XX that has been calculated or processed. "XX" is any element (for example, any information).

[0017] <Outline of this embodiment> The conventional technology assumes that noisy speech is input to a speech recognition device trained with clean speech. For a speech recognition device trained with clean speech, speech enhancement processing for noisy speech reduces mismatch. On the other hand, for a speech recognition device previously trained with noisy speech, speech enhancement processing is unnecessary processing, and there is a possibility that it may worsen the mismatch.

[0018] For this reason, although speech enhancement processing is effective for a speech recognition device trained with clean speech, it may be better not to apply speech enhancement processing to a robust speech recognition device. For this reason, in this embodiment, the probability distribution shape of the probability distribution model used when estimating the expected value of the feature of the speech signal is made selectable.

[0019] <Example of voice recognition device configuration> 1 is a block diagram showing an example of the configuration of a voice recognition device according to the present embodiment. As shown in FIG. The control unit 110 includes, for example, a feature analysis unit 112, a speech enhancement unit 114, a speech segment processing unit 116, a sampling processing unit 118, a hidden state processing unit 120, a speech processing unit 122, and a model learning unit 130.

[0020] The sound collection unit 20 is a microphone and includes an electro-acoustic converter that collects sounds arriving at the unit and converts the amplitude of the sound pressure into a voltage. The sound collection unit 20 outputs an electrical signal having a signal value indicating the converted voltage to the voice recognition device 10. A voice signal collected by a sound collection unit 20 is input to the voice recognition device 10 as an input voice signal.

[0021] <Outline of processing in each part of the voice recognition device> An overview of the processing performed by each unit of the speech recognition device 10 (feature analysis unit 112, speech enhancement unit 114, speech segment processing unit 116, sampling processing unit 118, hidden state processing unit 120, speech processing unit 122, and model learning unit 130) will be described.

[0022] The feature analysis unit 112 calculates, for each frame, an acoustic feature of the input audio signal input from the sound collection unit 20. The feature analysis unit 112 outputs the acoustic feature calculated for each frame to the voice enhancement unit 114, the speech segment processing unit 116, and the sampling processing unit 118.

[0023] The speech emphasizing unit 114 determines, for each frame, an emphasis feature in which the speech component is emphasized with respect to the acoustic feature of the input speech signal using the first model. The speech emphasizing unit 114 outputs the calculated emphasis feature to the sampling processing unit 118. The first model will be described later.

[0024] The speech section processing unit 116 determines a speech section based on the acoustic characteristics of the input audio signal for each frame acquired from the feature analysis unit 112. The speech section processing unit 116 outputs the determined speech section to the sampling processing unit 118.

[0025] The sampling processing unit 118 samples the sample values ​​of the target feature sequence multiple times using a third model (evidence model) indicating a probability distribution of the emphasis feature sequence, which is a sequence of emphasis features in an utterance section, and the target feature sequence corresponding to the acoustic feature sequence, which is a sequence of acoustic features, and determines an expected value of the hidden state feature sequence from the sample values ​​of the hidden state feature sequence. The sampling processing unit 118 uses the evidence model for each utterance. The sampling processing unit 118 estimates the feature expected value using a probability distribution model. In this embodiment, a probability distribution model is created using a beta distribution. The distribution shape of the beta distribution changes depending on two parameters. Furthermore, in this embodiment, an optimal probability distribution model is selected using an ASR (Automatic Speech Recognition; hereinafter, ASR is also referred to as "speech recognition") score. The probability distribution model, a method for selecting the model, and a method for calculating the ASR score will be described later. The sampling processing unit 118 outputs the sample values ​​of the target feature sequence to the hidden state processing unit 120. The sampling processing unit 118 outputs the determined hidden state feature sequence to the speech processing unit 122.

[0026] The hidden state processing unit 120 determines a hidden state feature sequence, which is a sequence of hidden state features, based on a target feature sequence, which is a sequence of target features, using the second model. The first model will be described later. The hidden state processing unit 120 outputs sample values ​​of the hidden state feature sequence to the sampling processing unit 118.

[0027] The speech processing unit 122 determines the content of the speech in the speech section based on the expected value of the hidden state feature sequence using the fourth model, which will be described later.

[0028] The model learning unit 130 uses pre-constructed training data to learn, for example, the second model and the fourth model. The learning method and the teacher data will be described later.

[0029] <Conventional E2E> First, we provide an overview of conventional end-to-end (E2E) speech recognition, missing data frameworks, and evidence models. Deep learning using the E2E model replaces machine learning systems that require multiple stages of processing from input data to outputting results with a single large neural network equipped with multiple layers and modules that perform various processing. Conventional missing-data speech recognition (e.g., Gaussian Mixture Model (GMM)-HMM (hidden Markov model) and Deep Neural Network (DNN)-HMM) cannot be applied to recent end-to-end speech recognition because the assumption of inter-frame independence does not match these new architectures.

[0030] End-to-end phonetic recognition is performed using a sequence of speech feature vectors s of length T. 1:T = [s1,…,s T ] to a string c of length L 1:L = [c1,…,c L ] is directly estimated. t ∈R D (R is a double underlined letter) is a D-dimensional vector. s 1:T From c 1:L to the posterior probability p(c 1:L |s 1:T ) is modeled by a DNN and obtained by supervised learning. A recent connectionist temporal classification (CTC) / attention architecture for end-to-end speech recognition consists of a shared encoder, a CTC, and an attention-decoder network. The shared encoder network is a network that extracts speech features s 1:T Let T' be the encoded hidden vectors h 1:T’ = [h1,…,h T’ ], h t ∈R D’ (t=1,…,T') (R is a double underline character). The CTC and attention decoder networks use different methods to estimate the posterior probability pctc (c 1:L |h 1:T’ ) and p att (c 1:L |h 1:T’ ) is expressed.

[0031] The score function J in the decoding process is expressed by the following equation (1): asr is the weight w i ∈[0,1],(i=1,2,3), where h 1:T’ is expressed by the following equation (2).

[0032]

number

[0033]

number

[0034] Equation (1), p lm (c 1:L ) represents the language model. The string that maximizes this score function is searched for. 1:T’ (·),p ctc ,p att ,p lm ) can be implemented with any modern neural network model, e.g. a transformer.

[0035] In speech recognition, missing data / features refers to corrupted or distorted speech features caused by noise signals or signal processing techniques such as speech enhancement. Missing data speech recognition is a method to recognize such distorted speech features s^ 1:T It is possible to recognize it by utilizing the uncertainty of In the speech enhancement scenario, the speech enhancement module computes a single-channel, noisy observed spectrum x in the short-time Fourier transform (STFT) domain. 1:T = [x1,…,x T ],x t ∈C F From (t=1,…,T) (C is a double-underlined character), the distorted speech feature s^ 1:TEstimate the noise spectrum x 1:T The features extracted from u 1:T = [u1,…,u T ],u t ∈R D (t=1,…,T) (R is a double underline). In what follows, we assume that the frame index t and length L of the sequential variables are the same in both the STFT domain and the feature domain, and that the speech features are Mel-filter bank for speech recognition.

[0036] Missing data approaches generally smooth the target function or target variable using an evidence model (pdf) that represents the uncertainty in the input features. Small values ​​of the evidence pdf represent large uncertainty. Smoothing is implemented by evaluating the expectation of the target function or variables with respect to the evidence pdf ε and the parameter set Θ. 1:T and the observed spectrum x1:T is included in Θ, i.e., {s^ 1:T ,x 1:T}∈Θ. For example, the target function is a likelihood function, e.g., the acoustic model p(s 1:T |c 1:L ), then smoothing is implemented by evaluating the expectation of the function on the evidence pdf ε:

[0037]

number

[0038] The assumption or decomposition of (conditional) independence between frames (e.g. HMMs) is usually made for high-dimensional variables s 1:T To simplify the integral above, i.e., s t is applied to the likelihood and / or evidence model to evaluate each pdf of The missing data approach for end-to-end speech recognition based on the encoder-decoder model has been experimentally demonstrated to be effective for the encoded vector h 1:T’It has been shown that is effective as a target variable. End-to-end speech recognition usually considers the correlation of features between frames due to highly nonlinear networks such as self-attention networks, so frame-independent assumptions cannot be applied to the evaluation of the expectation calculation. For this reason, the expectation is directly approximated by the Monte Carlo method as shown in the following equation (4), where n is an integer between 1 and N.

[0039]

number

[0040] In formula (4), {s 1:T (n)} n=1 N is ε(s 1:T ;Θ), and the encoded vector is smoothed by the samples. Here, non-MD-ASR (i.e., conventional ASR) uses a Dirac delta function ε(s 1:T ;Θ)=δ(s 1:T -s^ 1:T )

[0041] (Uniform evidence model for each utterance) Next, we explain the utterance-wise uniform evidence model. In the Missing Data (MD) framework for end-to-end speech recognition, we define an utterance-based evidence model ε(s 1:T ;Θ), whose dimensionality is usually high (D × T dimensions). Since sampling from such a large dimensional function is almost impossible, latent space models are introduced into the utterance-unit evidence to reduce the sampling complexity. In order to maintain the continuity of features between frames, the evidence model for an utterance unit is usually decomposed into a pdf format as shown in the following equation (5) (note that Θ is omitted in equation (5)).

[0042]

number

[0043] In equation (5), η is a latent variable, and p(s 1:T |η) conditional independence between t and d is assumed. 1:T )=Π t p(s t ) cannot be adopted as an utterance-based evidence model because samples extracted from this pdf will result in noisy features due to frame independence, which will severely degrade speech recognition performance.

[0044] The basic evidence model is an utterance-based uniform PDF that is an extension of the frame-based uniform PDF. This PDF is expressed as follows: 1:T and the observed feature u 1:T Only the latent variable η is used, and s^ 1:T Tou 1:T Also, ε δ - (η;η m ) is the following equation (7), and ε u - (η;η l ; u ) is expressed by the following equation (8).

[0045]

number

[0046]

number

[0047]

number

[0048] In formulas (6) to (8), s t,d, s^ t,d , u t,d are s t , s^ t , u t is the d-th dimension of . Also, ε δ- ( δ- is the scalar delta pdf, ε u- ( u- (-sign on B) corresponds to the scalar uniform pdf. η u , η l is η u > l The distribution boundary is specified by and I(η) (I is a double-underline character) is η∈[η l ,η u ] is an indicator function that is equal to 1 if ] and 0 otherwise.

[0049] Specifically, we subtract η from the uniform distribution and calculate s according to the delta function. t,d The samples of η can be deterministically chosen in even intervals in [0,1] with any resolution. u (s 1:T ) and samples extracted from it. g1 is a graph model of the uniform pdf for each utterance. g2 is a sample extracted in the feature space from the graph model of g1. Each axis of g2 is an appropriate dimension of the feature vector. The posterior probability of speech enhancement p(s 1:T |x 1:T ) can be integrated with the uniform pdf if the statistics of additional speech enhancement (SE) are available. u (s 1:T ) and the mixed pdf of the posterior probability is Θ={x 1:T ,u 1:T ,s^ 1:T ,π1,π2} and is defined as follows:

[0050]

number

[0051] In equation (9), π1 and π2 are mixture weights that satisfy π1+π2=1. Independence between frames is usually expressed as p(s 1:T |x 1:T )=Π t p(s t The posterior distribution of speech enhancement is assumed to be x1:T. However, this assumption leads to the generation of discontinuous features between frames when extracting samples. Therefore, such a posterior distribution of speech enhancement cannot be used as an evidence pdf directly, and a uniform evidence model is required as a probabilistic floor.

[0052] (Flexible evidence models for speech recognition models) Next, we will explain the flexible evidence model for the speech recognition model used in this embodiment. FIG. 3 is a diagram showing a feature curve and its projection onto the evidence model of this embodiment. Reference symbol g11 is a performance-feature curve, and reference symbol g12 is its projection onto the evidence model of this embodiment. In reference symbol g10, the horizontal axis is speech recognition performance (ASR performance), and the vertical axis is the posterior probability ε of the prior pdf. In reference symbol g20, the horizontal axis is the posterior probability ε of the prior pdf. B- ( B- is the - symbol above B). In addition, in g11, the line g11 is the robust model (Type 1), and the line g12 is the clean model (Type 2). In g10, the line g13 is (Type 3). In g12, the line g21 is (Type 1) ε B- ( B- is the -sign above B)(η;6,1), is (Type2)ε B- ( B- is the -sign on B (η;1,1). In the sign g20, the sign g23 is (Type3)ε B- ( B- is the -sign above B(η;1,6).

[0053] The structure of the feature space of speech recognition models is important for designing new flexible evidence models. For the speech recognition performance curve, as shown in Fig. 3, there are two end points, i.e., the enhanced speech feature s^ 1:T and observed noisy speech feature u 1:T We focus on the line segment defined by This curve is u 1:T Type 1 has a peak near s^ 1:T We assume that the models can be broadly divided into three types: Type 1, Type 2, and Type 3. Speech recognition models trained with clean speech with no (or little) noise correspond to Type 2, and robust speech recognition models correspond to Type 1. The traditional uniform pdf mentioned above can only capture Type 2 curves and the corresponding uncertainty. For a robust speech recognition model, it is expected that Type 1 pdfs will capture the inverse phenomenon of uncertainty.

[0054] For this reason, in this embodiment, an utterance-based evidence model based on a beta distribution is introduced to express the performance feature curve in the relative coordinate space, as shown by reference symbol g20 in FIG. 3. The structure of the evidence model in this embodiment is similar to the utterance-based uniform PDF given by equation (6), but the prior distribution follows the beta distribution as shown in the following equation (10). Note that in equation (1), ε B- ( B- is the -sign above B)(η;α,β) is the following equation (11).

[0055]

number

[0056]

number

[0057] In equation (11), η∈[0,1]. In equations (10) and (11), α and β are shape parameters of the beta distribution. For example, by setting α=6 and β=1, a Type 1 curve is obtained. When α=β=1, the distribution is equivalent to a uniform pdf. Also, ε B Since is a scalar density function, we can also deterministically select samples of η in even intervals in [0,1] from the cumulative density function by applying the inverse transformation method.

[0058] FIG. 4 shows an example of samples extracted from each parameter setting of the beta distribution. In FIG. 4, the density of the samples is s^ 1:T Tou 1:T The code g31 is an example of Type 1, the code g32 is an example of Type 2, and the code g33 is an example of Type 3. The codes g31 to g32 represent the difference between the samples extracted from each pdf type.

[0059] (Unsupervised selection of appropriate evidence models) In the flexible model, given an evidence model and a target speech set D (D is cursive) = {x 1:T (i)} i=1 Nd It is necessary to select appropriate parameters (e.g., α and β) for a given set of models (usually a test set). As mentioned above, typical parameter sets and patterns of evidence models are usually limited, so we use maximum likelihood estimation (no backpropagation of NN) to find a preset model and parameters M = {M j} j=1 Nm (M is cursive) Choose the most appropriate one. This criterion is the length-normalized ASR score of the following formula (12). That is, in this embodiment, the score of formula (12) is used as the ASR score to select the third model (or the parameters of the third model).

[0060]

number

[0061] In formula (12), L i is the length of the i-th recognized character, and E (E is a double-underline character) [h 1:T’ |M j ,x 1:T (i) ] (M is in cursive) is the jth parameter / model M j (M is in cursive) and the i-th observed spectrum x 1:T (i) represents the smoothed encoded vector given

[0062] (Bayesian Integration with Speech Enhancement) Since the posterior probability is also affected by the given speech recognition model, in this embodiment, we introduce the integration using Bayes' theorem of the evidence model with the new posterior probability model for speech enhancement. This implication means that the mixed integration by Equation (9) may not work well for robust ASR models.

[0063] Since the posterior model of this embodiment does not assume frame independence, samples can be extracted directly from the speech enhancement pdf without a mixture model. Figure 5 shows an overview of the whole model and the sampling process. As shown in Figure 5, the speech enhancement encoder 141 has an observed signal x 1:T is input, and the encoded latent variable z is output to the speech enhancement decoder 142. The speech enhancement decoder 142 outputs the encoded enhancement feature s^ 1:T to the evidence model 143. The evidence model 143 receives the observed signal and the output of the decoder 142 for speech enhancement, and outputs the interpolation result to the encoder 144 for speech recognition. The encoder 144 for speech recognition outputs the feature amount, which is the encoding result, to the decoder 145 for speech recognition. The decoder 145 for speech recognition outputs the speech recognition result. The encoder 144 for speech recognition is provided in, for example, the hidden state processing unit 120. The decoder 145 for speech recognition is provided in, for example, the speech processing unit 122.

[0064] The symbol g51 represents the output of the encoder for speech enhancement, which is the latent space. The top row represents the most likely point, and the bottom row represents the unscented transform described below. The double arrow is the scaling parameter γ. The code g52 is the feature space, which is the output of the decoder for speech enhancement, and the estimated acoustic feature s^ 1:T It is. The symbol g53 is the feature space, the output of the evidence model, and the estimated acoustic features s^ 1:T Interpolated to {s^ 1:T} n=1 N It is. The symbol g54 represents the output of the encoder 144 for speech recognition, which is a hidden vector space and has the mean of the samples (1 / n)Σ n=1 N (h 1:T’ )s^ 1;T (n) It is.

[0065] In this embodiment, the emphasis feature s^ 1:T As a random variable, an accurately integrated evidence model is derived as shown in the following formula (13) (the parameter set Θ is omitted in formula (13)). That is, in this embodiment, as shown in FIG. 5, the speech enhancement process and the speech recognition process are integrated using Bayes' theorem by treating the enhancement feature as a random variable.

[0066]

number

[0067] In equation (13), p(s^ 1:T |x 1:T ) corresponds to the posterior pdf of speech enhancement, and ε(s 1:T |s^ 1:T ,x 1:T ) is, for example, ε U (U is cursive) or ε B(B corresponds to cursive writing). These two evidence models 1:T Tou 1:T (x 1:T The posterior pdf is weighted by the first-order evidence model. The ancestral sampling method uses ε f (s 1:T |x 1:T ) is applied to extract samples from p(ŝ 1:T |x1 :T ) to s 1:T * Then, extract ε(s 1:T |s^ 1:T * ,x 1:T ) to s 1:T * The superscript * indicates that the sample was actually taken. Sampling from the posterior pdf of speech enhancement is simplified by assuming a new latent space model of the speech enhancement process, as shown in Eq. (14). Note that in Eq. (14), ŝ 1:T is expressed by the following equation (15): In addition, in equation (15), z is expressed by the following equation (16): In addition, in equation (16), n is another noise component.

[0068]

number

[0069]

number

[0070]

number

[0071] In equation (14), z∈R K(R is a double underline character) is a K-dimensional latent vector (latent variable), and z is a zero-mean noise random variable with small variance that can almost ignore fluctuations, as shown in equation (16). The function f in equation (15) is the function of speech enhancement and x 1:T represents a deterministic process involving feature extraction from . When the dimension K is small, sampling of the latent variables z is easy due to the Gaussian assumption. The variational posterior pdf q(z|x 1:T ) is used in place of the prior distribution during the actual sampling stage, as described below.

[0072] In this embodiment, the stochastic sampling process can be eliminated by applying an uncented transform (UT) (see Reference 1) to the latent variable z.

[0073] Reference 1; SJ Julier and JK Uhlmann, “A New extension of the Kalman filter to nonlinear systems”, in Proc. of Signal Processing, Sensor Fusion, and Target Recognition VI, vol. 3068, 1997, pp. 182-193

[0074] Post-mortem pdf p(s^ 1:T |x 1:T ) is expressed as the following equation (17), under the assumption of n, by the sigma point {z (k)} k=1 2K+1 and its weight {W (k)} k=1 2K+1 This can be efficiently approximated by a limited number of samples given by

[0075]

number

[0076] The definition of the weight using the parameter κ follows Reference 1. Also, f(x 1:T ,z (k) )} k=1 2K+1 and {W (k)} k=1 2K+1 Given ε(s 1:T |s^ 1:T ,x 1:T Deterministic sampling from is achieved by systematic resampling (see Ref. 2) without initial drawing.

[0077] Reference 2; R. Douc, O. Cappe, and E. Moulines, “Comparison of resampling schemes for particle filtering”, Proc. of International Symposium on Image and Signal Processing and Analysis, pp. 64-69, 2005

[0078] (Building a speech enhancement model) To learn the functions and parameters of (14) and (15) from the training data, a VAE-like framework is applied. Given the variational posterior probability q and function f by the DNN, we apply the observed noise spectrum x 1:T , the reference spectrum y 1:T , and its feature quantity s 1:T Learning can be performed by maximizing a lower bound on the log-likelihood given

[0079] The function f (called the decoder) consists of a spectrum mask process and a feature extraction process g. Mask-based speech enhancement applies a soft mask to the observed spectrum x t By multiplying with t (See Reference 3.)

[0080] Reference 3; D. Wang and J. Chen, “Supervised speech separation based on deep learning: An overview”, IEEE / ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 10, pp. 1702-1726, 2018

[0081] That is, as shown in the following equations (18) and (20), yt ≒ M○x t where the circles represent element-wise multiplication. The soft mask vector M is x 1:T or a function of its subset modeled by a DNN, where each element of M is in [0,1]. Also, in equation (18), n t y is expressed by the following equation (19). In addition, in equation (20), n t s is expressed as follows (21).

[0082]

number

[0083]

number

[0084]

number

[0085]

number

[0086] In equation (18), where log(·) and |·| denote the element-wise logarithm and absolute function, and L denotes the zero-mean Laplace distribution. x ~t is the x from frame (tk) to (t+k) t ε represents a small variance for regularization, which is n t y and t s This means that there is little process noise. Variational posterior matrix q(z|x 1:T )=N(m,Λ) (called the encoder) (N in cursive) is approximated by a VAE framework and used to sample z. The mean vector m and diagonal covariance matrix Λ of q are encoded by a DNN similar to the one used for the mask function M, by sliding window processing, a network that estimates the frame-wise mean and variance, and average pooling of the outputs. 1:T It is estimated from

[0087] Sigma points other than the mean are selected by adding a scaling parameter γ to the i-th element of m, i.e., m i ±γ. The variance estimated by the DNN is empirically ignored for the sigma points, i.e. the drawn samples are approximately the same as the mean vector m, and the smoothing effect of MD-ASR does not occur.

[0088] <Processing Procedure> Next, a description will be given of an example of a processing procedure performed by the voice recognition device 10. Fig. 6 is a flowchart showing a first example of a voice recognition processing example according to the present embodiment.

[0089] (Step S1) The feature analysis unit 112 analyzes an acoustic feature amount indicating an acoustic characteristic of an input speech signal for each frame.

[0090] (Step S2) The speech emphasizing unit 114 performs speech emphasizing processing on the acoustic feature acquired for each frame using the first model, and determines an emphasized feature in which the speech component is emphasized.

[0091] (Step S3) The speech section processing unit 116 detects a speech section consisting of a plurality of frames using a known voice detection method based on the acoustic characteristics of each frame.

[0092] (Step S4) The sampling processor 118 repeats the processes of steps S5 and S6 N times (samples N times). The sampling processor 118 constructs an emphasis feature sequence consisting of emphasis features in the detected speech section, and constructs a speech feature sequence consisting of speech features. The sampling processor 118 determines a probability distribution of a target feature sequence corresponding to the constructed emphasis feature sequence and speech feature sequence according to the third model.

[0093] (Step S5) The sampling processing unit 118 samples the sample values ​​of the target feature sequence using the determined probability distribution model.

[0094] (Step S6) The sampling processing unit 118 causes the hidden state processing unit 120 to calculate sample values ​​of the hidden state feature sequence for the sample values ​​of the target feature sequence sampled using the second model.

[0095] (Step S7) The sampling processing unit 118 determines whether the processes of steps S5 and S6 have been repeated N times. If they have been repeated N times, the process proceeds to step S8. If they have not been repeated N times, the process returns to step S5.

[0096] (Step S8) The sampling processing unit 118 calculates the average value of the sample values ​​of the N hidden state feature sequence as the expected value of the hidden state feature sequence.

[0097] (Step S9) The sampling processing unit 118 performs a decoding process and calculates a length normalized score using equation (12). The sampling processing unit 118 uses the average value of the length normalized scores (equation (12)) as a score for model selection.

[0098] (Step S10) The sampling processing unit 118 selects the parameters of a probability distribution model based on the calculated score, and selects a model.

[0099] (Step S11) The speech processing unit 122 uses the fourth model to calculate the posterior probability for the expected value of the hidden state feature sequence for each candidate of the utterance content. The speech processing unit 122 estimates the candidate of the utterance content with the maximum calculated posterior probability as the utterance content in the utterance section. After the process, the speech recognition device 10 ends the process.

[0100] 7 is a flowchart showing a second example of the speech recognition processing example according to the present embodiment. In the following processing, the speech processing unit 122 (decoder) includes, for example, an attention decoder network 122a (not shown), a CTC decoder network 122c (not shown), and an utterance content estimation unit 122d (not shown) (for example, see Japanese Patent Application No. 2022-134800).

[0101] In this way, when the attention decoder network 122a and the CTC decoder network 122c are included, the expected value E[h 1:T’ In this configuration, the expected value E[h 1:T’ ] (E is a double-underlined character) to the attention decoder network 122a and the CTC decoder network 122c. The attention decoder network 122a outputs the posterior probability patt The posterior probability patt to the speech content estimation unit 122d. The CTC decoder network 122c outputs the posterior probability p ctc The posterior probability p ctc to the utterance content estimation unit 122d.

[0102] The attention decoder network 122a and the CTC decoder network 122c each calculate a posterior probability p att (c 1:L|E[h1:T']), p ctc (c1:L|E[h1:T']) (E is a double-underlined character). Then, the utterance content estimation unit 122d of the utterance processing unit 122 calculates the posterior probability patt (c 1:L |E[h 1:T’ ]) (E is a double underlined letter) and the posterior probability p ctc (c 1:L |E[h 1:T’ ]) (E is a double underline) is the weighted sum of the logarithm of each candidate speech content to calculate the speech recognition score J asr The speech content estimation unit 122d calculates the speech recognition score J asr The candidate for the utterance content for which the maximum value is obtained can be estimated as the utterance content. In equation (22), w1 and w2 are each a predetermined real value between 0 and 1, and are normalized so that their sum is 1. In this case, in equation (12), J in equation (22) asr is used.

[0103]

number

[0104] (Step S21) The feature analysis unit 112 analyzes an acoustic feature amount indicating an acoustic characteristic of an input speech signal for each frame.

[0105] (Step S22) The speech emphasizing unit 114 performs speech emphasizing processing on the acoustic feature acquired for each frame using the first model, and determines an emphasized feature in which the speech component is emphasized.

[0106] (Step S23) The speech section processing unit 116 detects a speech section consisting of a plurality of frames by using a known voice detection method based on the acoustic characteristics of each frame.

[0107] (Step S24) The sampling processing unit 118 repeats the processes of steps S25, S26, and S27 N times (samples N times). The sampling processing unit 118 configures an emphasis feature sequence consisting of emphasis features in the detected speech section, and configures a speech feature sequence consisting of speech features. The sampling processing unit 118 determines a probability distribution of a target feature sequence corresponding to the configured emphasis feature sequence and speech feature sequence according to the third model.

[0108] (Step S25) The sampling processing unit 118 samples the sample values ​​of the target feature sequence using the determined probability distribution model.

[0109] (Step S26) The sampling processing unit 118 causes the hidden state processing unit 120 to calculate sample values ​​of the hidden state feature sequence for the sample values ​​of the target feature sequence sampled using the second model.

[0110] (Step S27) The sampling processing unit 118 outputs the sample values ​​of the hidden state feature sequence to the CTC decoder network 122c of the utterance processing unit 122, and causes the utterance processing unit 122 to calculate the sample values ​​of the posterior probability for each candidate utterance content as CTC posterior probability sample values.

[0111] (Step S28) The sampling processing unit 118 determines whether the processes of steps S25, S26, and S27 have been repeated N times. If they have been repeated N times, the process proceeds to step S29. If they have not been repeated N times, the process returns to step S25.

[0112] (Step S29) The sampling processing unit 118 calculates the average value of the sample values ​​of the N hidden state feature sequence as the expected value of the hidden state feature sequence.

[0113] (Step S30) The attention decoder network 122a of the speech processing unit 122 calculates, for each candidate of the utterance content, an expectation value of the posterior probability for the expectation value of the hidden state feature sequence as an attention posterior probability expectation value.

[0114] (Step S31) The utterance content estimation unit 122d of the utterance processing unit 122 calculates an average value of the CTC posterior probability sample values ​​for each candidate utterance content as a CTC posterior probability expected value.

[0115] (Step S32) The sampling processing unit 118 calculates a score using formula (12). The sampling processing unit 118 uses the average value of the length normalized scores (formula (12)) as a score for model selection.

[0116] (Step S33) The sampling processing unit 118 selects the parameters of a probability distribution model based on the calculated score, and selects a model.

[0117] (Step S34) The speech processing unit 122 calculates the weighted average of the attention posterior probability expectation value and the CTC posterior probability expectation value for each candidate of the speech content as the speech recognition score. The speech processing unit 122 estimates the candidate of the speech content with the maximum calculated speech recognition score as the speech content in the speech section. After the processing, the speech recognition device 10 ends the processing.

[0118] <Details of the processing in each part of the voice recognition device> Next, the processing performed by each unit of the speech recognition device 10 (the feature analysis unit 112, the speech enhancement unit 114, the speech section processing unit 116, the sampling processing unit 118, the hidden state processing unit 120, the speech processing unit 122, and the model learning unit 130) will be described in detail.

[0119] (Feature Analysis Department) The feature analysis unit 112 acquires an input audio signal input from the sound collection unit 20. The input audio signal is a digital signal indicating a time series of signal values ​​sampled at a predetermined sampling frequency. The sampling frequency is, for example, 16 kHz. The feature analysis unit 112 calculates an acoustic feature for each analysis window having a predetermined window length. The analysis window is a section in which the acoustic characteristics of the audio signal are analyzed at one time. For example, a Hanning window is used as the analysis window. The window length corresponds to the period to be analyzed, that is, a frame. The window length is, for example, 512 samples. The feature analysis unit 112 moves the analysis window section by a predetermined hop length at regular time intervals. The hop length corresponds to the period in which the analysis window is moved at one time. The hop length may be any positive real number equal to or less than the window length. The hop length is, for example, 128 samples. The feature analysis unit 112 can apply, as the acoustic feature, a feature representing a frequency characteristic, such as Short Time Fourier Transform parameters (STFT) or a Mel filter bank.

[0120] (Speech enhancement section) The speech enhancement unit 114 performs speech enhancement (SE) processing on the acoustic features of the input speech signal output by the feature analysis unit 112 using a first model to enhance speech components, and calculates the acoustic features with the enhanced speech components as enhanced features for each frame. Emphasizing speech components means relatively enhancing the speech components compared to other components. Noise suppression processing may be applied as the speech enhancement processing. For example, a deep neural network (DNN) can be applied as the first model.

[0121] (Speech section processing unit) The speech section processing unit 116 can use a known voice detection method (VAD: Voice Activity Detection) in detecting a speech section. The speech section processing unit 116 judges whether or not each frame is a speech section. For example, the speech section processing unit 116 uses a preset speech section determination model to determine a frame in which the speech section probability calculated based on the acoustic feature is equal to or greater than a predetermined speech section probability threshold as a speech section, and judges a frame in which the speech section probability is less than the threshold as a non-speech section. The speech section processing unit 116 can determine a series of non-speech section frames that are continuous for a number of consecutive frames equal to or greater than a preset lower limit as a non-speech section. The speech section processing unit 116 can determine a section consisting of one or more frames sandwiched between non-speech sections before and after the series as a speech section. As a result, even a speech section including a temporary silent section is detected as a series of speech sections without being divided by a silent section. The speech section determination model may be implemented as a part of the first model or the third model. The speech section processing unit 116 outputs the determined speech section to the sampling processing unit 118 .

[0122] (Sampling processing section) The sampling processing unit 118 receives the speech feature for each frame from the feature analysis unit 112, and receives the emphasis feature for each frame from the speech emphasis unit 114. The sampling processing unit 118 receives the speech section from the speech section processing unit 116. The sampling processing unit 118 arranges the emphasis feature for each frame in the speech section in that order to form an emphasis feature sequence. The sampling processing unit 118 arranges the speech feature for each frame in the speech section in that order to form a speech feature sequence. The sampling processing unit 118 samples the sample values ​​of the target feature sequence multiple times using a third model. The third model will be described later. The sampling processing unit 118 outputs the sample values ​​of the target feature sequence to the hidden state processing unit 120. The sampling processor 118 receives sample values ​​of the hidden state feature sequence from the hidden state processor 120 as a response to the sample values ​​of the target feature sequence. The sampling processor 118 determines an expected value obtained by averaging the sample values ​​of the hidden state feature sequence between samples as the hidden state feature sequence for the speech section. The sampling processor 118 outputs the determined hidden state feature sequence to the speech processor 122.

[0123] (Hidden state processing unit) The hidden state processor 120 receives sample values ​​of the target feature sequence from the sampling processor 118 for each sample. The hidden state processor 120 uses a second model to calculate sample values ​​of the hidden state feature sequence for the sample values ​​of the target feature sequence. The second model may be, for example, a model corresponding to a shared encoder network that is part of a known connectionist temporal classification (CTC) / attention architecture. A parameter set trained to provide a hidden state feature sequence for a speech feature sequence of a known clean speech may be applied to the second model.

[0124] The fourth model may have, for example, a CTC / attention architecture. The CTC / architecture includes a CTC decoder network and an attention decoder network. The speech content or its candidates are constructed using a label string including one or more labels. A label can be a character, a syllable, a word, or any other unit related to the representation of the speech content. The label string may be expressed using text. When searching for a recognition result from a set of candidates for speech information, for example, a known beam search technique can be used. The control unit 110 may store the speech information that is the recognition result determined by the speech processing unit 122, may use it for other processing, or may output it to other devices.

[0125] (Model Learning Section) The model learning unit 130 learns the second and fourth models using pre-configured training data. In this disclosure, "model learning" or "learning a model" means determining a parameter set used in a calculation based on the model. The training data includes a plurality of different data sets, and each data set includes known input data and output data, and is configured by associating them. In learning a certain model, the model learning unit 130 recurrently updates the parameter set so that the difference between the calculation value obtained by calculation on the input value constituting the input data and the output value constituting the output data corresponding to the input data is reduced (minimized) for the entire training data. When the difference becomes equal to or less than a predetermined judgment threshold value, or when the number of updates reaches a predetermined number, the model learning unit 130 stops the model learning at that point, and sets the obtained parameter set to the function unit related to each model.

[0126] The model learning unit 130 learns the second model and the fourth model simultaneously. Each data set constituting the training data used for learning includes, as input data, speech features derived from a speech signal in a certain speech section, and includes, as output data, speech information indicating known speech content in that speech section. In learning the second model and the fourth model, input data derived from a speech signal indicating clean speech may be used. This output data may be represented by a vector value in which the posterior probability for a candidate for speech information that provides a correct answer is 1, and the posterior probability for other speech information is 0.

[0127] As a loss function indicating the magnitude of the difference between the calculated value and the output value, for example, binary cross entropy can be used. In updating the parameter set, for example, backpropagation based on steepest gradient or stochastic gradient descent, or a variant thereof (for example, Adam Optimizer) can be used.

[0128] The first model used in the voice enhancement process can be trained independently of the second and fourth models used in the voice recognition process. The training of the first model will be described later together with a specific example of the voice enhancement unit 114. Depending on the configuration of the third model described later, training is not necessarily required. Some or all of the parameters of the third model may be derived from calculated values ​​obtained in the course of training the first model.

[0129] <Evaluation> (Evaluation conditions, evaluation method) Evidence for the Present Embodiment: To evaluate the models and evaluate whether unsupervised model selection can select an appropriate model for each speech recognition model, multiple speech recognition models trained with different training data were prepared. Furthermore, the experimental conditions for the speech recognition model are open in terms of speakers, reverberation, and noise sources. We also believe that the language spoken in the dataset (e.g., English or Japanese) does not have a significant impact on the noise robustness of speech recognition or speech enhancement.

[0130] For the speech recognition and speech enhancement datasets, experiments were conducted using speech data from the Corpus of Spontaneous Japanese (CSJ) (see Reference 4). The training set included approximately 660 hours of speech (full set) for speech recognition and 190 hours (3-16 seconds of data from the core set) for speech enhancement.

[0131] Reference 4; K. Maekawa, “Corpus of spontaneous Japanese: Its design and evaluation”, in Proc. of ISCA & IEEE Workshop on Spontaneous Speech Processing and Recognition, 2003

[0132] The official CSJ evaluation set (eval{1,2,3}, total 5 hours) was used as the test set for MD-ASR. Each speech data was divided into utterances. The sampling rate was 16 kHz. The WHAM! corpus (see Reference 5) was used as a general background noise source for speech recognition training.

[0133] Reference 5; G. Wichern, J. Antognini, et al., “WHAM!: Extending speech separation to noisy environments”, in Proc. Interspeech, Sep. 2019

[0134] The ProSoundEffects (PSE) Corpus 1 (see Reference 6) was used as an application-specific non-speech source in both the training set of the SE model and the phonetic-English recognition test set. The PSE corpus includes environmental sounds, animal calls, and music signals. We also created a validation set from other data that was not included in either the training or test sets. The validation sets consisted of three models: (a) a clean speech model (clean), (b) a clean and reverberant speech model (clean+rev.), and (c) a clean and reverberant noisy speech model (clean+rev.+bgn.).

[0135] Reference 6; I. Kavalerov, S. Wisdom, et al., “Universal sound separation”, in Proc. of IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, 2019, pp. 175-179.

[0136] The reverberant speech signal was convolved with real impulse responses (RT20 640ms) measured from 576 positions in the room to simulate a real acoustic environment. The mixed SNR of the WHAM! background noise was randomly set to 0, 5, 10, 15, and 20dB. The total speech data used for model (c) was 660×3=1980 hours. The training data for the speech enhancement model was a mixture of reverberant speech and non-speech signals. Specifically, each speech signal was combined with a randomly selected non-speech signal from the PSE data. The mixed SNR was randomly set to -5, 0, 5, 10, and 15 dB. The reference spectrum was the direct sound (first arrival sound) frame of the reverberant speech.

[0137] The test set data was also a mixture of reverberant speech with different SNRs and raw non-speech signals to evaluate the robustness of the MD-ASR of the present embodiment. Different impulse responses were recorded from a microphone in a different room at a distance of 1.5 m. The speech source signals were from the CSJ test set, and the non-speech signals were from PSE data that was not used in the training set of the speech enhancement model. The SNRs were -5, 0, 5, and 10 dB.

[0138] (Example of the DNN and MD-ASR configuration used in the evaluation) We will explain the configuration of the DNN and MD-ASR used in the evaluation. First, the ASR model used the Transformer encoder-decoder acoustic model of ESPnet (see Reference 7). The STFT parameters were a 512-point Hanning window and a 128-point shift. For the speech features, we used an 80-dimensional Mel filter bank and applied global mean-variance normalization. To improve the robustness of the model, we also applied speech rate perturbation and SpecAug techniques. For details on the network, RNN language model, and hyperparameters, see the CSJ recipe and configuration file for ESPnet.

[0139] Reference 7; S. Watanabe, T. Hori, S et al., “ESPnet: End-to-end speech processing toolkit”, in Proc. of Interspeech, 2018, pp. 2207-2211.

[0140] The speech enhancement model was built from scratch using the PyTorch library and its predefined classes (see Reference 8). The STFT parameters are the same as those in ESPnet. The feature extraction network consisted of an 80-dimensional filter bank network, absolute and power activation functions, a concatenation of 32 frames before and after the central frame, and positional encoding.

[0141] Reference 8; A. Paszke, S. Gross, F. Massa, et al., “PyTorch: An imperative style, high-performance deep learning library”, in Proc. of Advances in Neural Information Processing Systems 32, 2019, pp. 8024-8035.

[0142] To estimate the mask M and the latent variable z, respectively, the configuration utilizes a 6-layer and 2-layer transformer encoder with 8 heads and 512 hidden nodes as the main network, followed by a fully connected layer. The nonlinear function of the mask head is a sigmoid function, and the nonlinear function of the frame-by-frame variance of z before average pooling is an absolute value function. In addition, gradient clipping (see Reference 9) and Adam (see Reference 10) are applied as configurations in the evaluation.

[0143] Reference 9; R. Pascanu, T. Mikolov, et al., “On the difficulty of training recurrent neural networks”, in Proc. of International Conference on International Conference on Machine Learning - Volume 28, 2013, pp. III-1310-III-1318. Reference 10; DP Kingma and J. Ba, “Adam: A method for stochastic optimization”, in Proc. of International Conference on Learning Representations, 2015.

[0144] All parameters were updated for each utterance with a mini-batch size of 1. The learning rate was 1.0 × 10 -4 The parameters for 50 epochs were used for evaluation. The decoding parameters are almost the same as in the ESPnet CSJ recipe. The search beam size is 20, and the weights of CTC, attention, and language model are 0.5, 0.5, and 0.3, respectively. The number of samples N used for the isolated evidence model (uniform, beta, etc.) is 8. The sampling parameters for the other cases are N=40, π1=0.125, and κ=0.5. The EB and γ parameters (α,β) are evaluated for the combinations {(1,1),(2,1),(4,1),(6,1),(8,1)} and {4,8}, respectively. The model selection method of this embodiment automatically selects the best result from these parameter sets for each evaluation set and SNR.

[0145] (Results between different speech recognition models for evaluation of unsupervised model selection) Figure 8 shows the character error rate (CER:%) under each SNR condition using three speech recognition models to evaluate unsupervised model selection. Note that CER is the sum of eval1, eval2, and eval3 test sets. Also, UT stands for unscented transformation. Symbol g101 is the evaluation condition, symbol g102 is the evaluation result of the above-mentioned validation set (a) clean speech model (clean speech recognition model), symbol g103 is the evaluation result of the validation set (b) clean reverberant speech model, symbol g104 is the evaluation result of the validation set (c) clean reverberant noisy speech model (robust speech recognition model), and symbol g105 is the average of the results of validation sets (a) to (c). In symbols g101 to g105, the rows indicate each parameter, the formula used for calculation, etc.

[0146] The smaller the CER, the better the speech recognition performance. The total number of characters in eval{1,2,3} was 115,745. In Fig. 8, the “Avg” column shows the average performance for each speech recognition model at all SNRs, and the “Total” column shows the average performance for all speech recognition models. “No proc.” in code g101 means that the noisy speech signal is directly recognized. The baselines were the method using only speech enhancement and the method using speech enhancement using the uniform evidence model.

[0147] As shown in Figure 8, the evaluation results showed that in the case of the non-robust speech recognition model, speech enhancement significantly degraded performance by 14 points compared to “No proc.” This is due to the mismatch between speech enhancement and ASR uncertainty. The uniform evidence model improved CER, but did not surpass “No proc.” The same phenomenon was observed in the comparative speech recognition model (validation set (b)) at 5 dB and 10 dB. That is, the model using the uniform distribution is effective for clean speech recognition, but has the opposite effect on robust speech recognition. And the model using the beta distribution has the opposite effect on clean speech recognition, but is effective for robust speech recognition.

[0148] As shown in the last line of FIG. 8, in this embodiment, it was confirmed that the performance was improved for both models by selecting the parameters α and β using the speech recognition score.

[0149] The evaluation of the evidence model separation will be described. The evidence model based on the beta distribution of this embodiment outperformed the “No proc” by 1.3 points in “Avg.” using the robust speech recognition model. Bayesian integration using speech enhancement (UT and beta) further improved the CER by about 0.2 points in “Avg.”. However, these settings did not work well for the phonetic recognition model trained with clean speech. Even in this case, the integration of the uniform evidence model and speech enhancement by the method of this embodiment outperformed the “speech enhancement only” method at all SNRs.

[0150] This shows that the ASR score of DNNs can be trusted to some extent in model selection.This result shows that the ASR score of DNNs can be trusted to some extent in model selection.

[0151] (Detailed results using the robust speech recognition model for the evidence model evaluation of this embodiment) Fig. 9 is a diagram showing detailed character error rates (CER (%)) of test sets under each SNR condition using a robust speech recognition model. In Fig. 9, symbol g201 denotes the evaluation condition, symbol g202 denotes the evaluation result of test set eval1, symbol g203 denotes the evaluation result of test set eval2, symbol g204 denotes the evaluation result of test set eval3, and symbol g205 denotes the average of the results of test sets eval1 to 3. In symbols g201 to g205, the rows indicate each parameter, a formula used for calculation, etc.

[0152] In addition, in Figure 9, the total number of characters in the test sets eval1, eval2, and eval3 was 43,897, 43,623, and 28,225 characters, respectively. The “Avg.” column shows the average performance across all SNRs, and the “Total” column shows the average performance across the eval set. “No proc.” shows the case where the noisy speech signal was directly recognized. The baselines are the method using only speech enhancement and speech enhancement using the uniform evidence model. The CERs of 8.5, 6.8, and 7.3 for “No proc” under 10 dB conditions are close to the CERs for clean speech, so the improvement under low SNR conditions is important in this evaluation.

[0153] The overall performance trends among different test sets (eval1, eval2, eval3) are similar to the results in Fig. 8. In the independent evaluation of the evidence model, the beta distribution evidence model of this embodiment outperformed the “No proc.” by 1.3 points in total and by more than 3 points in the -5 dB condition. The CER of speech enhancement was worse than “No proc.” Therefore, the evidence model of the present embodiment can take advantage of the poor results of speech enhancement. Bayesian integration with speech enhancement (UT and β) of the present embodiment was able to improve the CER by about 0.2 points on some test sets.

[0154] As described above, it has been found that the evidence model of this embodiment based on the beta distribution can perform well even in a robust speech recognition model, especially under noisy low SNR conditions. That is, according to the method and configuration of this embodiment, it is possible to use a speech recognition model trained in advance with clean speech, as well as a phonetic recognition model trained with noisy speech. And, according to this embodiment, it is possible to reduce the uncertainty mismatch between speech enhancement and the speech recognition model.

[0155] The evidence model of this embodiment filters the uncertainty of speech enhancement and selects effective parameters according to the ASR score without supervision. As a result, the evaluation results showed that this embodiment can improve the CER of enhanced speech even under low SNR (-5-dB) conditions using a robust speech recognition model.

[0156] All or part of the processing performed by the speech recognition device 10 described above may be processed by an external device such as a server connected via a network, or may be processed on the cloud.

[0157] In addition, a program for realizing all or part of the functions of the voice recognition device 10 of the present invention may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to perform all or part of the processes performed by the voice recognition device 10. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer system" also includes a WWW system equipped with a homepage providing environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" also refers to a storage device that holds a program for a certain period of time, such as a volatile memory (RAM) inside a computer system that becomes a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.

[0158] The above program may also be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium, or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above program may also be for realizing part of the above-mentioned functions. Furthermore, it may be a so-called difference file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0159] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0160] 10... speech recognition device, 110... control unit, 112... feature analysis unit, 114... speech enhancement unit, 116... speech section processing unit, 118... sampling processing unit, 120... hidden state processing unit, 122... speech processing unit, 130... model learning unit

Claims

1. a speech segment processing unit that determines a speech segment based on an acoustic characteristic of an input speech signal; a speech emphasizing unit that determines, for each frame, an emphasis feature in which a speech component is emphasized with respect to an acoustic feature of the input speech signal using a first model; a hidden state processing unit that determines a hidden state feature sequence that is a sequence of hidden state feature values ​​based on a target feature sequence that is a sequence of target feature values ​​by using a second model; a sampling processing unit that samples sample values ​​of a target feature sequence corresponding to an emphasis feature sequence, which is a sequence of the emphasis feature sequences within an utterance section, and an acoustic feature sequence, which is a sequence of the acoustic feature sequences, a plurality of times using a third model that indicates a probability distribution of the target feature sequence, and determines an expected value of the hidden state feature sequence from the sample values ​​of the hidden state feature sequence; a speech processing unit that determines a speech content of the speech section based on an expected value of the hidden state feature sequence using a fourth model, The third model is a probability distribution of a beta distribution.

2. the sampling processing unit selects parameters of the beta distribution based on a score of a speech recognition result; 2. The speech recognition device according to claim 1.

3. The voice enhancement unit, the sampling processing unit, and the speech processing unit are The speech enhancement process and the speech recognition process are integrated using Bayes' theorem by treating the enhancement feature as a random variable.

2. The speech recognition device according to claim 1.

4. the speech enhancement unit includes a speech enhancement encoder and a speech enhancement decoder; The hidden state processing unit includes an encoder for speech recognition, The speech processing unit includes a decoder for speech recognition, the speech enhancement encoder encodes an observed spectrum of the input speech signal to obtain latent variables, and outputs the latent variables to the speech enhancement decoder; the speech enhancement decoder decodes the input observation spectrum and the latent variables to estimate speech features, and outputs the estimated acoustic features to the third model; the third model performs interpolation on the estimated acoustic feature and the observed spectrum by sampling using a probability distribution of the beta distribution, and outputs the interpolated estimated acoustic feature to the speech recognition encoder; the speech recognition encoder encodes the interpolated estimated acoustic features and outputs the sequence of hidden state features to the speech recognition decoder; the speech recognition decoder decodes the input hidden state feature sequence and outputs a speech recognition result.

2. The speech recognition device according to claim 1.

5. The sampling processing unit includes: calculating, for each sampling, a sum of maximum scores of the speech recognition result calculated using a character string to be speech-recognized, parameters of the third model or the third model and an observed spectrum of the input speech signal, a smoothed encoded vector, and the observed spectrum, and selecting parameters of the beta distribution based on the calculated sum; 3. The speech recognition device according to claim 2.

6. A speech recognition method for a speech recognition device, comprising: The speech recognition device, a speech segment processing step of determining a speech segment based on an acoustic characteristic of an input speech signal; a speech emphasizing step of determining, for each frame, an emphasis feature in which a speech component is emphasized with respect to an acoustic feature of the input speech signal using a first model; a hidden state processing step of determining a hidden state feature sequence, which is a sequence of hidden state features, based on a target feature sequence, which is a sequence of target features, by using a second model; sampling sample values ​​of the target feature sequence a plurality of times using a third model that is a beta distribution probability distribution indicating a probability distribution of an emphasis feature sequence that is a sequence of the emphasis feature in an utterance section and a target feature sequence that corresponds to the acoustic feature sequence that is a sequence of the acoustic feature; a sampling process step of determining an expected value of the hidden state feature sequence from sample values ​​of the hidden state feature sequence; and executing an utterance processing step of determining an utterance content of the utterance section based on an expected value of the hidden state feature sequence using a fourth model. Speech recognition methods.

7. The computer of the voice recognition device The speech recognition method according to claim 6 is performed as follows. program.

Images