Voice recognition device and method

Abstract

The invention relates to a computer-implemented speech recognition method comprising the following steps: temporal windowing (WD) of a sound signal into a plurality of sound sequences; feature extraction (EXT) from each of the sound sequences; assignment (CLAS) of each sound sequence to a phoneme from among a plurality of phonemes by providing the extracted features of the sound sequence to a phonetic classification network comprising layers of impulse neurons; and word determination (NLU) of one or more words from the phonemes to which the sound sequences are assigned. Figure 2

Description

Title of the invention: Voice recognition device and method Technical field

[0001] The field of the invention is that of automatic speech recognition (also called voice recognition) which makes it possible to analyze a sound signal from a recording of a human voice in order to transcribe it in the form of a text that can be used by a machine. Prior art

[0002] With today's computing processors and the large volume of training data, deep learning models are enjoying great success in the field of speech recognition. However, deep learning models are not energy efficient and are unsuitable for embedded applications where high-performance processors are not available. These models require too much resource to be efficiently embedded (we often talk about models requiring 7-8 GB of resource). Statement of the invention

[0003] An objective of the invention is to have a low-power voice recognition solution, easily embedded and requiring no connection to a computing cloud or an external storage system.

[0004] To this end, the invention proposes a computer-implemented method of voice recognition, comprising the following steps: - time windowing of a sound signal into a plurality of sound sequences; - extraction of characteristics from each of the sound sequences; - for each of the sound sequences, assignment of the sound sequence to a phoneme from among a plurality of phonemes by providing the extracted features of the sound sequence to a phonetic classification network that includes layers of spiking neurons; - determination of one or more words from the phonemes to which the sound sequences are assigned.

[0005] Some preferred but non-limiting aspects of this method are as follows: - temporal windowing is Hamming windowing; - the extraction of characteristics from a sound sequence includes the calculation of cepstral coefficients; - cepstral coefficients are coefficients on the Mel scale; - the determination of one or more words includes the application of a hidden Markov model having as observable states the phonemes to which the sound sequences are assigned; - the layers of spiking neurons of the classification neural network comprise hidden layers in which the spiking neurons are integration and leaky firing neurons each characterized by a firing threshold and a leaky factor, the spiking neurons of the same hidden layer sharing the same firing threshold and the same leaky factor; - the layers of impulse neurons of the classification neural network comprise an input layer in which the impulse neurons are each configured to generate a train of impulses distributed according to a Poisson law with a firing rate proportional to the value of one of the characteristics extracted from a sound sequence provided to it as input; - the classification neural network includes an output layer made up of integration neurons, each associated with one of the different phonemes.

[0006] The invention also relates to a computer program product comprising instructions which, when the program is executed by a computer, cause the latter to implement the steps of the method according to the invention as well as a voice recognition device which comprises a processor configured to implement the steps of the method according to the invention. Brief description of the drawings

[0007] Other aspects, aims, advantages and characteristics of the invention will appear better on reading the following detailed description of preferred embodiments thereof, given by way of non-limiting example, and made with reference to the appended drawings in which:

[0008] - [Fig.l] is a diagram of a voice recognition device according to a mode of possible realization of the invention;

[0009] - [Fig.2] is a diagram illustrating steps of a voice recognition method according to a possible embodiment of the invention;

[0010] - [Fig.3] illustrates a possible realization of a phonetic classification network with impulse neurons used within the framework of the invention.

[0011] DETAILED DESCRIPTION OF PARTICULAR EMBODIMENTS

[0012] With reference to [Fig.l], the invention relates to a recognition device voice 1 which comprises an audio signal processing module 10, a voice recognition module 20 and a natural language interpretation module 30. This Device 1 comprises a processor configured to implement the steps which are described subsequently. Signal Processing Module 10

[0013] The audio signal processing module is configured to process an incoming sound signal So, which can be provided to it via a microphone or through an audio file, and to provide as output a set of MFCC characteristics of this signal.

[0014] With reference to [Fig.2], this module 10 is more particularly configured to obtain the sound signal during a step OB, to carry out a time windowing of the sound signal So into a plurality of sound sequences during a step WD and to carry out an extraction of characteristics of each of the sound sequences during a step EXT.

[0015] The sound signal So is most often non-stationary. However, it is made up of a sequence of phonemes whose properties do not vary for a duration ranging from 5 to 100 milliseconds. It is thus possible to assume that this signal is stationary over a window of a few milliseconds.

[0016] The windowing carried out during the WD step can be a Hamming windowing defined by the following expression: yn < h( n) = 0.54- 0.46œs( ) •

[0017] Hamming windowing has the advantage of smoothing the transition of the audio signal on the edges whereas other types of windowing (rectangular type for example), even if they do not distort the signal over the interval considered, cause a sudden variation on the edges which will induce many significant harmonics and consequently disrupt the inference and learning of the network.

[0018] The value of N is for example adapted to cover 25 ms of the signal (N=0.025*v with v=16 kHz for example the sampling frequency).

[0019] In one possible embodiment, in order not to lose the information corresponding to the low amplitudes of the windowing function, the windows are superimposed for a period of time Q (for example 10 ms) adapted to the length of the window.

[0020] The expression of the function corresponding to the k-th window is therefore the following: [0021 ] V n< N, sk(n) - s( k^Q + n^h^n) where sk denotes the k-th sound sequence from windowing (we also speak of a truncated signal to designate such a sound sequence).

[0022] The values ​​of Q and N as well as the windowing function are adaptable. Those given above are classic for performing speech recognition and allow to find a good compromise between the quantity of calculations and the precision of the results. As already mentioned, the Hamming window avoids significant

[0023]

[0024]

[0025]

[0026]

[0027]

[0028]

[0029]

[0030]

[0031]

[0032]

[0033]

[0034]

[0035] discontinuities at the edges of the window but can for example be replaced by a Hann window which has similar properties. A feature vector is then extracted from each of the sound sequences during the EXT step. Extracting the features of a sound sequence may include calculating cepstral coefficients. These cepstral coefficients are preferably coefficients on the Mel scale (MFCC coefficients for "Mel-Frequency Cepstral Coefficients"). For example, there are 13 MFCC coefficients, which ensures good performance for a speech recognition system using these MFCC coefficients. A first step in calculating the MFCC is to calculate the Fourier transform Sk of the truncated signal, for example using a fast Fourier transform. This transform is completed with zeros so that it has 512 points. Since a sound signal has real values, its transform is symmetrical: V n< N, Sk(Nn) =Sk(n) = Sk( -n). It is therefore possible to consider only the first M = 257 points. The power spectral density of the window is then calculated according to: Vn<< Pk(n) =^\Sk(n)\2 This function is then filtered with Nfilters=23 Mel-scale bandpass filters. This filtering returns a real number per filter that gives an estimate of the spectral power of the signal in a frequency range. The filters are defined as follows. First, a minimum and maximum frequency are set, which is then converted to a Mel scale. 0 Hz and the Nyquist frequency (8 kHz) are chosen, respectively, which gives 0 mel and 2840 mel. The conversion from Hertz to mel is done as follows: m = Mel (f) = 1127*Zw (1 + 7^0) The center frequency of each filter is then calculated by spacing them linearly between 0 and 2840 mel. By converting these values ​​back to Hertz, we obtain filters spaced logarithmically according to the frequency scale. To be able to be applied to the signal spectrum, these values ​​must be aligned with the FFT parameters: ym < Nfutres, fm = round ( x 512 Each value f ,m< Nfentres represents the frequency corresponding to the peak of a filter. We define the m-th filter by:

[0036]

[0037]

[0038]

[0039]

[0040]

[0041]

[0042]

[0043]

[0044]

[0045]

[0046] r ' < i W' ■ Vfce / -f , ml"' ml f -f J m m-\ rrr 1 / Once filtered, we obtain the following powers: V m <Nfiltre„ Pk(m) = Ï^Pk(*) *Hm(i) A final step is implemented by calculating the cepstral coefficients in an attempt to eliminate speaker-dependent features. The cepstrum of a signal is defined as follows: K1 [ logF [ s„ ] ] • The cepstrum is therefore the spectrum of a spectrum. The speaker-dependent harmonics are therefore transformed into higher-order cepstral coefficients. By retaining only the first coefficients, we eliminate the noise created by the vocal apparatus on the sound. The first 13 coefficients therefore form the MFCCs. In one possible implementation, we do not use a discrete Fourier transform but rather a discrete cosine transform because it is more efficient when it comes to compressing data. V k < 13, Ck = In [ Pk ( n ) ] *cos [ ( n + 4 ) ] Voice Recognition Module 20 The speech recognition module 20 is configured to transform the MFCC characteristics into a sequence of PHO phonemes. For this purpose, it uses a spike neural network (SNN). Humans achieve auditory perception using a computational paradigm that is completely different from that of classical deep learning. Indeed, auditory perception in humans involves sparse, asynchronous events that are transmitted as electrical impulses in the brain. Although the transmission speed between biological neurons (a few milliseconds) is lower than that of artificial neurons, the human brain performs complex speech recognition at a speed higher than that of artificial networks and learns languages ​​in a different way from classical networks. The invention, motivated by these observations on how the human brain learns and recognizes speech, proposes to imitate the human learning and recognition system with an automatic recognition system based on a spiking neural network (SNN). In such an SNN, the exchange between neurons is based on the integration of impulses and the descent of activation, like biological neurons. The encoding is therefore temporal and binary.

[0047]

[0048]

[0049]

[0050]

[0051]

[0052]

[0053]

[0054]

[0055] Because it operates with asynchronous and time-distributed binary signals, such a network leads to greater computational efficiency on embedded hardware. Binary encoding also allows such a network to be easily deployed on an electronic board, such as a Field Programmable Gate Array (FPGA). With reference to [Fig.2], the module 30 is more particularly configured to, during a CLAS step, assign each of the sound sequences to a phoneme from among a plurality of phonemes by providing, for each of the sound sequences, the characteristics extracted from the sound sequence to a phonetic classification network which comprises layers of impulse neurons. Referring to [Fig.3], the phonetic classification network comprises an input layer I composed of spiking neurons, an output layer O and one or more hidden layers H composed of spiking neurons between the input layer I and the output layer O. In a possible embodiment, the phonetic classification network comprises two hidden layers. The spiking neurons of the input layer I and those of the hidden layers H are preferably leaky integrate-and-fire neurons (LIF neurons for Leaky Integrate-and-Fire), each characterized by a firing threshold and a leakage factor. Such a LIF neuron can be described by: + (1) L sî^' >0 (2) 0, otherwise where is the membrane potential of a post-neuron i at time t, Ki is the leakage factor of post-neuron i with a value between 0 and 1, wtj is the weight connecting pre-neuron j to post-neuron i, is the output of pre-neuron j at time t, oH is the output of post-neuron i at time t-1 and Ç. is the firing threshold of post-neuron i. The first term in equation (1) denotes the leakage in the membrane potential, the second term integrates the weighted inputs received from the pre-neurons, and the third term accounts for the reduction in potential when the neuron generates a spike. The firing threshold directs the average integration time of the input, and the leakage regulates how much of the potential from the previous instant is retained. The pre-activation for each neuron is the weighted sum of the y & .w. inputs. j J received by the neuron. Generally, the firing threshold of a neuron is defined as the maximum pre-activation of the neuron, which provides good accuracy but at the cost of high latency. In one possible embodiment, the thresholds at each layer are determined, not by using the maximum pre-activation, but by taking a certain percentile of each layer's pre-activation distribution (e.g., the 99.7 percentile). This reduces latency with a slight decrease in accuracy.

[0056] Finally, as will be described later with the expressions for calculating the gradient of the parameters of each layer, the network is trained with error backpropagation to optimize the weights, membrane leakage and thresholds. In particular, the spatial and temporal allocation of credits can be performed by the temporal unfolding of the network and by backpropagation in BTTP time (for Back Propagation Through Time in English).

[0057] Input layer

[0058] For each of the sound sequences, the 13 MFCC cepstral coefficients extracted from the sound sequence are applied directly to the input layer I at each time interval t. The input layer I thus comprises 13 neurons.

[0059] The input layer acts as both a feature extractor and a spike generator, which accumulates the MFCC coefficient values ​​and generates the output spikes. More particularly, the spiking neurons of the input layer are each configured to generate a train of pulses distributed according to a Poisson law with a firing rate proportional to the value of that of the features extracted from a sound sequence provided to it as input.

[0060] Such generation of a pulse train is similar to rate-coding, but the firing rate is here defined as a function of the membrane leakage and the trigger threshold, which are all learned by gradient descent. In order to achieve the conversion of the values ​​of the MFCC coefficients into pulse trains, rate-coding uses a Poisson generator to generate random numbers at each time interval and compares these random numbers to the values ​​of the MFCC coefficients to produce, when a value of an MFCC coefficient is greater than one of these random numbers, a pulse of the pulse train.

[0061] Output layer

[0062] The output layer O comprises as many neurons as there are phonemes that can be assigned to the sound sequences (i.e. the number of classes in which a sound sequence can be assigned). For example, the output layer O comprises 33 neurons corresponding to the 33 phonemes of the French language. Each of the neurons in the output layer provides an assignment score PBPHO1, PbPHO2....PbPHO32, PbPHO33 of a sound sequence to one of the phonemes, in the form of a probability

[0063]

[0064]

[0065]

[0066]

[0067]

[0068]

[0069]

[0070]

[0071]

[0072]

[0073]

[0074]

[0075]

[0076] of belonging to a given class or a binary score of belonging or not to a given class. The neurons in the output layer are accumulator neurons that only accumulate incoming inputs without any leakage and do not generate an output impulse. They can be described by: where vi is a vector containing the membrane potential of the N output neurons, N is the number of classes (phonemes in our case), is the weight matrix connecting the output layer to the previous layer, and is a vector containing the spike signals of the previous layer ( l -1 ). In one possible implementation, the cumulative values ​​in the neurons of the output layer are compared to threshold values ​​and each provides a binary value depending on the result of the comparison. The output layer thus forms an activation layer, with a softmax activation function. The weight learning can be done as follows using a loss function defined on vi at the last time interval T. The cross-entropy loss can be used which exploits the softmax function calculated on rf: L-= oy where ® is the vector containing the softmax values, L is the loss function, and y is the one-hot encoding vector of the training phoneme class label. The weight update is calculated as follows: a VV dL dL V / a V / ar az, 34, - 3434, -\®-y)Wi Where 7 is the learning rate. Hidden layers In a preferred embodiment of the invention, the LIF neurons of the same hidden layer share the same trigger threshold and the same leakage factor. This reduces the number of network parameters to be trained, even though the Applicant was unable to observe any significant improvement by assigning an individual threshold / leakage to each neuron. The neurons in the hidden layers can then be described as follows:

[0077] ^ = J.1et çi of'= 1 0, otherwise

[0078] Where Ki is a real value representing the leakage factor common to all neurons in layer 1 and where C; is a real value representing the trigger threshold common to all neurons in layer l.

[0079] During training, the weight update is calculated as follows:

[0080] y èL yyi , l 0^? C^l-}

[0081] is the discontinuous gradient that can be approximated by the gradient of substitution: A i I and M dtf, fat > where is a constant ~7-ymax 0,1- £, - —(^4 _ _ denoting the maximum value of the gradient.

[0082] The update of the threshold C} is then calculated as follows:

[0083]

[0084] _Y _Y — — — - V j

[0085] Finally, the leak update Ki is calculated as follows:

[0086] = and a _Vt)£_V5L^^_V ÔL ôofi j 1 AkI - LtdKl - Lt^ 3^ - 2-^ U

[0087] The invention thus exploits a low-latency phoneme classification neural network whose training can be carried out by gradient descent in order to optimize both the membrane leakage, the trigger threshold and the weights of the network. The membrane leakage and the threshold of each layer of the network are optimized with end-to-end backpropagation to achieve competitive accuracy at reduced latency.

[0088] The values ​​of the spatio-temporal coefficients of a speech signal are directly applied to the input layer of the network without the need to convert them into pulse trains. The input layer is trained to convert the inputs into pulses where the LIF neurons integrate the inputs and generate an output spike when the membrane potential exceeds the firing threshold. This eliminates the need for a generator function (and associated overhead) used in other coding schemes.

[0089] Learned membrane leakage controls the input information flow and attenuates irrelevant inputs to increase activation sparsity in dense layers of the network. The reduced latency combined with high and sparse activation allows for a significant improvement in computational efficiency. Membrane leakage and a suitable threshold for each layer leads to a great improvement in activation parsimony and energy efficiency.

[0090] The invention thus proposes a voice recognition solution with low resource consumption and low latency. It can be used to control systems (robot for example) by voice command with low resource consumption (memory and energy). Natural Language Interpretation Module 30

[0091] The natural language interpretation module 30 is configured to transform the PHO phoneme sequence into a TXT text. With reference to [Fig.2], it determines one or more words from the PHO phoneme sequence during an NLU step.

[0092] Determining one or more words may include applying a hidden Markov model having as observable states the phonemes to which the sound sequences are assigned.

[0093] A Markov chain consists of several states as well as a probability of transition from one state to another. The probability of being in a state depends only on the previous state. We can thus, from any state, look at the probability of transition to all the other states.

[0094] A hidden Markov model is applicable to sentence reconstruction: by taking the PHO phonemes at the output of the voice recognition module 20 as observable states and the existing words as hidden states, it is possible, from the transition of the observable phonemes, to find the original word. The type of Markov model used is preferably a hidden model of the left-right type which has the particularity of only going in one direction, with a loop to prevent a phoneme which is repeated at the output of the voice recognition module 20 from being present several times.

[0095] It should be noted that it is also possible to find a word by looking at each phoneme that belongs to it. While this method is simpler to implement, it has the disadvantage that if a phoneme is added, made an error, or removed, it is impossible to find the word. Conversely, a left-right hidden Markov model provides a probability for each word, thus making it possible to find a word even in the presence of an error in the sequence of phonemes.

[0096] The module 30 is further configured, after having been able to identify a word, to check that the sentence makes sense by taking into account that words may be phonetically close. N-grams can be used for this. Furthermore, it is possible to eliminate errors which led to the determination of words which should not have been identified as such.

[0097] In order to obtain the most accurate result possible, it may be more interesting to look for which sequence of phonemes is the most probable. The Viterbi algorithm can be used to calculate the most probable sequence with significantly less complexity than if all possible phoneme lists had to be tested. To do this, we analyze, for each state, the probability of moving from one state to another as well as the probability of the next state and we repeat the action until the entire phoneme sequence is covered.

[0098] The invention is not limited to the device and method previously described but also extends to a computer program product comprising instructions which, when the program is executed by a computer, lead the latter to implement the steps of the method previously described.

Claims

Claims

1. A computer-implemented method of speech recognition, comprising the following steps: - time windowing (WD) of a sound signal (So) into a plurality of sound sequences; - extraction (EXT) of features (MFCC) from each of the sound sequences;- for each of the sound sequences, assignment (CLAS) of the sound sequence to a phoneme among a plurality of phonemes by providing the characteristics extracted from the sound sequence to a phonetic classification network which comprises layers (E, H) of impulse neurons, the layers of impulse neurons of the classification neural network comprise hidden layers (H) in which the impulse neurons are integration and trigger neurons with leakage each characterized by a trigger threshold and a leakage factor, the impulse neurons of the same hidden layer sharing the same trigger threshold and the same leakage factor; - determination (NLU) of one or more words from the phonemes to which the sound sequences are assigned.;

2. The method of claim 1, wherein the temporal windowing is Hamming windowing.

3. Method according to one of claims 1 and 2, in which the extraction of characteristics of a sound sequence comprises the calculation of cepstral coefficients.

4. The method of claim 3, wherein the cepstral coefficients are coefficients on the Mel scale.

5. Method according to one of claims 1 to 4, in which the determination of one or more words comprises the application of a hidden Markov model having as observable states the phonemes to which the sound sequences are assigned.

6. A method according to one of claims 1 to 5, wherein the layers of spiking neurons of the classification neural network comprise an input layer (I) in which the spiking neurons are each configured to generate a train of pulses distributed according to a Poisson law with a firing rate proportional to the value of one of the characteristics extracted from a sound sequence supplied to it as input.

7. Method according to one of claims 1 to 6, in which the classification neural network comprises an output layer (0) consisting of integration neurons each associated with one of the different phonemes.

8. Computer program product comprising instructions which, when the program is executed by a computer, cause the latter to implement the steps of the method according to one of claims 1 to 7.

9. Voice recognition device (1), comprising a processor configured to implement the steps of the method according to one of claims 1 to 7.

Images