Speech recognition method and apparatus, and computer-readable storage medium

Abstract

a voice recognition device and a computer-readable storage medium are also disclosed. The voice recognition method includes the steps of: acquiring a first linear frequency spectrum corresponding to training sounds having different sampling rates; determining a maximum sampling rate and other sampling rates among the different sampling rates; determining maximum frequency-domain serial numbers of the first linear frequency spectrum corresponding to the other sampling rates as first and second frequency-domain serial numbers, respectively; configuring amplitude values ​​corresponding to frequency-domain serial numbers greater than the first frequency-domain serial number and less than or equal to the second frequency-domain serial number in the first linear frequency spectrum corresponding to the other sampling rate to be zero, to obtain a second linear frequency spectrum corresponding to the other sampling rate; determining first and second voice features according to first mel-spectral features of the first linear frequency spectrum corresponding to the maximum sampling rate and second mel-spectral features of the second linear frequency spectrum corresponding to the other sampling rate, respectively; and training a machine learning model using the first and second voice features.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application is based on and claims priority to China Patent Application Publication No. 201910904271.2, filed on September 24, 2019, the disclosure of which is hereby incorporated in its entirety into this application.

[0002] The present disclosure relates to the field of machine learning, and in particular to a speech recognition method and apparatus, and a computer-storable medium. [Background technology]

[0003] In recent years, with the rapid development of artificial intelligence technology, intelligent customer service systems have been widely applied in many service fields such as e-commerce, logistics, and finance. ASR (Automatic Speech Recognition) is one of the core technologies of intelligent customer service systems, which is intended to convert users' voice signals into text information for the convenience of subsequent natural language processing. Therefore, accurate and real-time ASR is an important prerequisite for intelligent customer service systems to ensure that they can accurately understand customers' intentions and provide service responses. However, in real intelligent customer service situations, the acoustics to be recognized may have various complex factors such as accents, dialects, and background noise interference, which poses a major challenge to high-precision speech recognition.

[0004] Regarding the issue of speech recognition in the context of intelligent customer service, in conventional methods, a machine learning model for speech recognition is usually decomposed into multiple serial sub-models, where each sub-model is a machine learning model for realizing a part of the function of speech recognition. In multiple serial sub-models, the output of the previous sub-model is the input of the next sub-model. These serial sub-models include, but are not limited to, an acoustic model, a pronunciation dictionary model, and a language model. After processes such as sub-model design, data annotation, and sub-model training, a final machine learning model is obtained.

[0005] However, the conventional method has two main problems. First, a large amount of annotation data is required to train each sub-model, and the more sub-models there are, the higher the time and labor costs for obtaining annotation data become. Second, speech recognition is performed by using a machine learning model obtained by multiple serial sub-models, which results in the error of the sub-models being easily transmitted cumulatively. For example, the accuracy of each sub-model is 0.9, and the accuracy of the machine learning model obtained by three serial sub-models is 0.9. 3 = 72.9%. The application effect of the final machine learning model is significantly limited by the adoption of the multiple serial sub-model method.

[0006] With the rapid development of DL (Deep Learning) technology, compared with the machine learning model adopting multiple serial sub-models in the traditional method, the machine learning model based on end-to-end DNN (Deep Neural Network) significantly improves its performance. DNN-based speech recognition adopts only one machine learning model, the input end of the model is speech acoustic data, and the output end of the model is text recognition result. Such a model effectively solves the problems in the machine learning model of the traditional method, such as high sub-model design cost, high data annotation cost of multiple sub-models, and low speech recognition accuracy brought about by the continuous accumulation of errors of multiple sub-models.

[0007] At present, the frameworks of DNN-based machine learning models mainly include the Connectionist Temporal Classification (CTC) framework, the Listen Attention Spell (LAS) framework, and the Recurrent Neural Network Transducer (RNNT) framework, and later network structures of DNN-based machine learning models evolve based on these three frameworks.

[0008] Since the network structure for performing speech recognition by using a machine learning model is usually large, obtaining large-scale acoustic data and annotation data as training data is an important prerequisite for training a good machine learning model for speech recognition. Therefore, whether or not large-scale acoustic data can be obtained as training data is a crucial factor in the speech recognition performance of a machine learning model.

[0009] However, unlike obtaining data such as images and texts, audio data contains confidential information such as privacy and copyright, and it is usually difficult to obtain large-scale audio data easily. Furthermore, in the process of obtaining annotation data, it is necessary to manually listen to the audio data at least once to complete the annotation of the audio data, which makes the annotation cost high and makes it difficult to obtain large-scale annotation data.

[0010] Furthermore, even if large-scale acoustic data are obtained, these acoustic data usually come from different acquisition devices, and as a result, the sampling rates of these acoustic data are not all the same, and it is difficult to simultaneously apply acoustic data with different sampling rates to training a machine learning model.

[0011] In view of this, in the method of performing speech recognition by using a machine learning model, acoustic data of a single sampling rate is adopted to train and obtain a machine learning model corresponding to the sampling rate. However, since the speech feature distributions of acoustic data of different sampling rates are different, it is often difficult for a machine learning model trained and obtained under a certain speech feature distribution to be effectively applied to another speech feature distribution, so that a DNN-based machine learning model trained and obtained by using acoustic data of a certain sampling rate has poor speech recognition effect with respect to acoustic data of another sampling rate.

[0012] In order to train a machine learning model by using larger-scale acoustic data, in the related art, the acoustic data with different sampling rates undergoes downsampling or upsampling, thereby converting the different sampling rates into the same sampling rate, and then the audio features of the acoustic data are extracted, and then the machine learning model is trained by using the extracted audio features. Summary of the Invention [Means for solving the problem]

[0013] According to a first aspect of the present disclosure, a speech recognition method is provided, the method including the steps of: obtaining a first linear spectrum corresponding to a training sound of a different sampling rate, an abscissa of the first linear spectrum being a spectrum-sequence serial number, an ordinate of the first linear spectrum being a frequency-domain serial number, and a value of a coordinate point determined by the abscissa and the ordinate being an original amplitude value corresponding to the training sound; determining a maximum sampling rate and another sampling rate other than the maximum sampling rate among the different sampling rates; determining a maximum frequency-domain serial number of the first linear spectrum corresponding to the other sampling rate as a first frequency-domain serial number; determining a maximum frequency-domain serial number of the first linear spectrum corresponding to the maximum sampling rate as a second frequency-domain serial number; and determining the other sampling rate to obtain a second linear spectrum corresponding to the other sampling rate. setting amplitude values ​​corresponding to each frequency domain serial number that is greater than the first frequency domain serial number and less than or equal to the second frequency domain serial number in a first linear spectrum corresponding to the maximum sampling rate to zero; determining a first speech feature of an acoustic to be trained for a maximum sampling rate according to a first mel-spectral feature of the first linear spectrum corresponding to the maximum sampling rate; determining a second speech feature of an acoustic to be trained for another sampling rate according to a second mel-spectral feature of a second linear spectrum corresponding to the other sampling rate; and training a machine learning model by using the first speech feature and the second speech feature.

[0014] In some embodiments, determining first speech features of the acoustic to be trained at the maximum sampling rate includes performing a global normalization process on the first mel-spectral features to obtain the first speech features, and determining second speech features of the acoustic to be trained at another sampling rate includes performing a local normalization process on the second mel-spectral features to obtain the second speech features.

[0015] In some embodiments, the local normalization process includes the steps of: obtaining a mel-spectral frequency corresponding to the maximum linear spectral frequency according to a maximum linear spectral frequency corresponding to the acoustic to be trained of the other sampling rate; calculating a maximum mel filter serial number corresponding to the mel-spectral frequency; obtaining a first amplitude value corresponding to each other mel filter serial number in the second mel-spectral feature, where the other mel filter serial number is a mel filter serial number less than or equal to the maximum mel filter serial number; calculating an average and a standard deviation of all the first amplitude values ​​as a local average and a local standard deviation, respectively; calculating a first difference between each of the first amplitude values ​​and its local average; calculating a ratio of each first difference to the local standard deviation as a normalized first amplitude value corresponding to each first amplitude value; and replacing each first amplitude value in the second mel-spectral feature with the normalized first amplitude value corresponding to each first amplitude value.

[0016] In some embodiments, the global normalization process includes obtaining second amplitude values ​​corresponding to each mel filter sequence number in the first mel-spectral feature, calculating a mean and standard deviation of all the second amplitude values ​​as a global mean and a global standard deviation, calculating second differences between each of the second amplitude values ​​and the global mean, calculating a ratio of each second difference to the global standard deviation as a normalized second amplitude value corresponding to each second amplitude value, and replacing each second amplitude value in the first mel-spectral feature with the normalized second amplitude value corresponding to each second amplitude value.

[0017] In some embodiments, obtaining the first linear spectra corresponding to the training sounds having different sampling rates includes obtaining the first linear spectra corresponding to the training sounds having different sampling rates by using a short-time Fourier transform, respectively.

[0018] In some embodiments, the step of obtaining a first linear spectrum corresponding to the training sound having a different sampling rate includes the steps of obtaining an audio signal oscillogram of the training sound having a different sampling rate, performing pre-emphasis processing on the audio signal oscillogram of the training sound having a different sampling rate, and obtaining a first linear spectrum corresponding to the training sound having a different sampling rate according to the audio signal oscillogram after the pre-emphasis processing.

[0019] In some embodiments, the speech recognition method further includes performing a mel-filtering transform on the first linear spectrum corresponding to the maximum sampling rate and the second linear spectrum corresponding to the other sampling rate by using a plurality of unit triangle filters to obtain first and second mel-spectral features, respectively.

[0020] In some embodiments, the machine learning model includes a deep neural network (DNN) model.

[0021] In some embodiments, the different sampling rates include 16 kHz and 8 kHz.

[0022] In some embodiments, the speech recognition method further includes the steps of obtaining speech to be recognized, determining speech features of the speech to be recognized, and inputting the speech features of the speech to be recognized into a machine learning model to obtain a speech recognition result.

[0023] In some embodiments, the step of determining speech characteristics of the sound to be recognized includes the steps of determining a maximum frequency domain serial number of a first linear spectrum of the sound to be recognized as a third frequency domain serial number, setting amplitude values ​​corresponding to each frequency domain serial number in the first linear spectrum of the sound to be recognized that is greater than the third frequency domain serial number and less than or equal to the second frequency domain serial number to zero to obtain a second linear spectrum of the sound to be recognized, and determining speech characteristics of the sound to be recognized according to mel-spectral characteristics of the second linear spectrum of the sound to be recognized.

[0024] According to a second aspect of the present disclosure, a speech recognition device is provided, the speech recognition device including: a first acquisition module configured to acquire first linear spectra corresponding to sounds to be trained at different sampling rates, where an abscissa of the first linear spectrum is a spectrum sequence serial number, an ordinate of the first linear spectrum is a frequency domain serial number, and a value of a coordinate point determined by the abscissa and the ordinate is an original amplitude value corresponding to the sounds to be trained; a first determination module configured to determine a maximum sampling rate and another sampling rate other than the maximum sampling rate among the different sampling rates; a second determination module configured to determine a maximum frequency domain serial number of the first linear spectrum corresponding to the other sampling rate as a first frequency domain serial number; and a second determination module configured to determine a maximum frequency domain serial number of the first linear spectrum corresponding to the maximum sampling rate as a second frequency domain serial number. a third determining module configured to determine an amplitude value corresponding to each frequency domain serial number that is greater than the first frequency domain serial number and less than or equal to the second frequency domain serial number in the first linear spectrum corresponding to the other sampling rate to zero to obtain a second linear spectrum corresponding to the other sampling rate; a fourth determining module configured to determine a first speech feature of the to-be-trained acoustic of the maximum sampling rate according to a first mel-spectral feature of the first linear spectrum corresponding to the maximum sampling rate; a fifth determining module configured to determine a second speech feature of the to-be-trained acoustic of the other sampling rate according to a second mel-spectral feature of the second linear spectrum corresponding to the other sampling rate; and a training module configured to train a machine learning model by using the first speech feature and the second speech feature.

[0025] According to a third aspect of the present disclosure, there is provided a speech recognition apparatus comprising a memory and a processor coupled to the memory, the processor configured to perform a speech recognition method according to any of the above embodiments based on instructions stored in the memory.

[0026] According to a fourth aspect of the present disclosure, there is provided a computer-storable medium having stored thereon computer program instructions which, when executed by a processor, implements a speech recognition method according to any of the embodiments.

[0027] The accompanying drawings, which constitute a part of this specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the disclosure.

[0028] The present disclosure can be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0029] [Figure 1] FIG. 2 is a flow diagram illustrating a speech recognition method according to some embodiments of the present disclosure. [Diagram 2] FIG. 1 is a flow diagram illustrating steps of obtaining first linear spectra corresponding to training sounds with different sampling rates according to some embodiments of the present disclosure. [Figure 3A] FIG. 2 illustrates an audio signal oscillogram of a training sound with a sampling rate of 8 kHz, according to some embodiments of the present disclosure. [Figure 3B] FIG. 2 illustrates an audio signal oscillogram of an acoustic to-be-trained having a sampling rate of 16 kHz, according to some embodiments of the present disclosure. [Figure 4A] FIG. 2 illustrates an audio signal oscillogram of a training sound with a sampling rate of 8 kHz after pre-emphasis processing according to some embodiments of the present disclosure. [Figure 4B]FIG. 2 illustrates an audio signal oscillogram of a training sound with a sampling rate of 16 kHz after pre-emphasis processing, according to some embodiments of the present disclosure. [Figure 5A] FIG. 1 illustrates a first linear spectrum corresponding to a training sound having a sampling rate of 8 kHz, in accordance with some embodiments of the present disclosure. [Figure 5B] FIG. 2 illustrates a first linear spectrum corresponding to a training sound having a sampling rate of 16 kHz, in accordance with some embodiments of the present disclosure. [Figure 6] FIG. 13 illustrates a second linear spectrum corresponding to a training sound having a sampling rate of 8 kHz, in accordance with some embodiments of the present disclosure. [Figure 7] FIG. 1 is a flow diagram illustrating steps for determining a first audio feature of a training sound with a maximum sampling rate according to some embodiments of the present disclosure. [Figure 8] FIG. 2 illustrates a first mel-spectral feature of a first linear spectrum corresponding to 16 kHz, in accordance with some embodiments of the present disclosure. [Figure 9A] FIG. 1 is a flow diagram illustrating a global normalization process according to some embodiments of the present disclosure. [Figure 9B] FIG. 1 illustrates a first audio feature corresponding to a training sound having a sampling rate of 16 kHz, according to some embodiments of the present disclosure. [Figure 10] FIG. 11 is a flow diagram illustrating steps of determining a second audio feature of an acoustic to be trained for another sampling rate according to some embodiments of the present disclosure. [Figure 11] FIG. 13 illustrates a second mel-spectral feature of a second linear spectrum corresponding to 8 kHz in accordance with some embodiments of the present disclosure. [Figure 12A] FIG. 1 is a flow diagram illustrating a local normalization process according to some embodiments of the present disclosure. [Figure 12B] FIG. 13 illustrates a second audio feature corresponding to a training sound having a sampling rate of 8 kHz, according to some embodiments of the present disclosure. [Figure 13] 1 is a block diagram illustrating a speech recognition device according to some embodiments of the present disclosure. [Figure 14] FIG. 13 is a block diagram showing a speech recognition device according to another embodiment of the present disclosure. [Figure 15] FIG. 1 is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] Next, various exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that the relative configurations, numerical expressions and numerical values ​​of the components and steps described in these embodiments do not limit the scope of the present disclosure unless otherwise specified.

[0031] However, it should be understood that the sizes of the various parts shown in the drawings have not been drawn to scale for ease of illustration.

[0032] The following description of at least one exemplary embodiment is merely exemplary in nature and is in no way intended to limit the disclosure, its application, or uses.

[0033] Techniques, methods, and apparatus known to those skilled in the art may not have been discussed in detail but should, where appropriate, be considered part of this specification.

[0034] In all examples shown and discussed herein, any particular values ​​should be construed as merely illustrative and not limiting, as such values ​​may differ in other examples of example embodiments.

[0035] It should be noted that like reference numerals and letters indicate like items in subsequent drawings, so that once a particular item is defined in one drawing, it need not be further discussed in subsequent drawings.

[0036] In the related art, a large amount of original acoustic data information is lost during downsampling, and other acoustic data information is inserted during upsampling, which makes it impossible to accurately extract the original speech features of acoustic data with different sampling rates, and therefore makes it difficult to improve the speech recognition performance of the machine learning model.

[0037] In view of this, the present disclosure provides a speech recognition method for accurately extracting original speech features from acoustic data with different sampling rates, thereby improving the speech recognition performance of a machine learning model.

[0038] FIG. 1 is a flow diagram illustrating a speech recognition method according to some embodiments of the present disclosure.

[0039] 1, the speech recognition method includes steps S110 to S180. For example, the speech recognition method is performed by a speech recognition device.

[0040] In step S110, a first linear spectrum corresponding to the training sound with different sampling rates is obtained. The abscissa of the first linear spectrum is the spectrum sequence serial number, and the ordinate is the frequency domain serial number. The value of the coordinate point determined by the abscissa and ordinate of the first linear spectrum is the original amplitude value corresponding to the training sound. 。

[0041] The different sampling rates include, for example, 16 kHz and 8 kHz. That is, the sounds to be trained include sounds with a sampling rate of 16 kHz and sounds with a sampling rate of 8 kHz. Although the present disclosure only takes the most common sampling rates of 16 kHz and 8 kHz as examples, it should be understood that the sampling rate can also include other sampling rates such as 44.1 kHz, 50 kHz, etc.

[0042] In some embodiments, the sampling rate is indicated by s, and s ∈ [1,2]. s ∈ [1,2] indicates that the value of the sampling rate s is 1 or 2. Under the condition that the value of the sampling rate s is 1, the sampling rate of the sound to be trained is 8 kHz. Under the condition that the value of the sampling rate s is 2, the sampling rate of the sound to be trained is 16 kHz.

[0043] For example, there may be multiple training sounds with the same sampling rate. In some embodiments, the number of training sounds with a sampling rate of 8 kHz is N 1 where N 1 is a positive integer. The number of sounds to be trained with a sampling rate of 16 kHz is N 2 where N 2 is a positive integer.

[0044] In some embodiments, the first linear spectra corresponding to the training audio with different sampling rates are obtained by using short-time Fourier transform (STFT) respectively. For example, the time domain features of the audio can be converted into frequency domain features by using short-time Fourier transform. In the field of speech recognition, the frequency domain features have better speech feature representation, and different speech features are more easily distinguished in the spectrum. Therefore, converting the time domain features of the training audio into frequency domain features is more beneficial for training the machine learning model.

[0045] In some embodiments, the step of obtaining first linear spectra corresponding to training sounds with different sampling rates in step S110 of FIG. 1 may be implemented in a manner as shown in FIG.

[0046] FIG. 2 is a flow diagram illustrating steps of obtaining first linear spectra corresponding to training sounds with different sampling rates according to some embodiments of the present disclosure.

[0047] As shown in FIG. 2, the step of obtaining a first linear spectrum corresponding to a training sound with a different sampling rate includes steps S111 to S113.

[0048] In step S111, audio signal oscillograms of the training audio with different sampling rates are obtained, which represent the time domain features of the audio.

[0049] For example, audio signal oscillograms of the training sounds with sampling rates of 8 kHz and 16 kHz are obtained, respectively.

[0050] The speech signal oscillograms of the training sounds with sampling rates of 8 kHz and 16 kHz will be described in detail below in conjunction with FIG. 3A and FIG. 3B.

[0051] FIG. 3A shows an audio signal oscillogram of an acoustic to-be-trained having a sampling rate of 8 kHz, according to some embodiments of the present disclosure.

[0052] As shown in Fig. 3A, the abscissa of the speech signal oscillogram of the training sound with a sampling rate of 8 kHz is the sampling point sequence number, and the ordinate is the normalized amplitude value of the speech signal. The speech signal oscillogram shown in Fig. 3A represents in a visualized form the discrete speech signal of the training sound with a sampling rate of 8 kHz.

[0053] In some embodiments, the discrete audio signal of the audio signal oscillogram of the training acoustic having a sampling rate of 8 kHz shown in FIG. 3A is

[0054]

number

[0055] where i∈[1,N 1 ].

[0056]

number

[0057] The number 1 in the table is the value of the sampling rate s.

[0058] For example, the number of consecutive sampling points of the discrete audio signal in the audio signal oscillogram of FIG. 3A is denoted by k.

[0059]

number

[0060] where:

[0061]

number

[0062] is a discrete audio signal

[0063]

number

[0064] In some embodiments, each sampling point corresponds to one sampling time moment, so that

[0065]

number

[0066] also denotes the total number of sampling time moments.

[0067] For example, the discrete audio signal of the audio signal oscillogram of the training acoustic having a sampling rate of 8 kHz shown in FIG.

[0068]

number

[0069] The amplitude value of the kth sampling point in is

[0070]

number

[0071] It can be shown that:

[0072] FIG. 3B shows an audio signal oscillogram of an acoustic to-be-trained having a sampling rate of 16 kHz according to some embodiments of the present disclosure.

[0073] As shown in Fig. 3B, the abscissa of the speech signal oscillogram of the training sound with a sampling rate of 16 kHz is the sampling point sequence number, and the ordinate is the normalized amplitude value of the speech signal. The speech signal oscillogram shown in Fig. 3B represents in a visualized form the discrete speech signal of the training sound with a sampling rate of 16 kHz.

[0074] In some embodiments, the discrete audio signal of the audio signal oscillogram of the training acoustic having a sampling rate of 16 kHz shown in FIG. 3B is

[0075]

number

[0076] where j∈[1,N 2].

[0077]

number

[0078] The number 2 in the figure is the value of the sampling rate s.

[0079] For example, the sequential number of sampling points of the discrete audio signal in the audio signal oscillogram of FIG. 3B is denoted by k'.

[0080]

number

[0081] where:

[0082]

number

[0083] is a discrete audio signal

[0084]

number

[0085] In some embodiments, each sampling point corresponds to one sampling time moment, so that

[0086]

number

[0087] also denotes the total number of sampling time moments.

[0088] For example, the discrete audio signal of the audio signal oscillogram of the training acoustic having a sampling rate of 16 kHz shown in FIG.

[0089]

number

[0090] The amplitude value of the k'th sampling point in is

[0091]

number

[0092] It can be shown that:

[0093] Returning to FIG. 2, after the speech signal oscillograms of the training sounds with different sampling rates are obtained, step S112 is performed. In step S112, pre-emphasis processing is performed on the speech signal oscillograms of the training sounds with different sampling rates, respectively. The speech signal oscillograms after pre-emphasis processing still represent the time domain features of the sound. By performing pre-emphasis processing on the speech signal, the signal-to-noise ratio of the speech signal is improved, the high frequency resolution of the speech is enhanced, and the speech features of the original training sounds are better restored.

[0094] For example, pre-emphasis processing is performed on audio signal oscillograms with sampling rates of 8 kHz and 16 kHz, respectively.

[0095] The audio signal oscillograms after pre-emphasis processing performed on the audio signal oscillograms of the training sound with sampling rates of 8 kHz and 16 kHz in FIGS. 3A and 3B will be described in detail below in conjunction with FIGS. 4A and 4B.

[0096] FIG. 4A shows an audio signal oscillogram of a training sound with a sampling rate of 8 kHz after pre-emphasis processing according to some embodiments of the present disclosure.

[0097] As shown in FIG. 4A, the abscissa of the speech signal oscillogram of the training sound with a sampling rate of 8 kHz after pre-emphasis processing is the sampling point sequence number, and the ordinate is the normalized amplitude value after pre-emphasis processing.

[0098] For example, the discrete audio signal in the pre-emphasis processed audio signal oscillogram shown in FIG.

[0099]

number

[0100] It can be shown that α is the pre-emphasis weight, 0<α<1. For example, α=0.97.

[0101] FIG. 4B shows an audio signal oscillogram of a training sound with a sampling rate of 16 kHz after pre-emphasis processing according to some embodiments of the present disclosure.

[0102] As shown in FIG. 4B, the abscissa of the speech signal oscillogram of the training sound with a sampling rate of 16 kHz after pre-emphasis processing is the sampling point sequence number, and the ordinate is the normalized amplitude value after pre-emphasis processing.

[0103] For example, the discrete audio signal of the audio signal oscillogram after pre-emphasis processing shown in FIG.

[0104]

number

[0105] It can be shown that α' is the pre-emphasis weight, 0<α'<1. For example, α'=0.97.

[0106] Returning to FIG. 2, after pre-emphasis processing is performed on the speech signal oscillograms of the training sounds with different sampling rates, step S113 is performed. In step S113, a first linear spectrum corresponding to the training sounds with different sampling rates is obtained according to the speech signal oscillograms after pre-emphasis processing. For example, a first linear spectrum corresponding to the training sounds with sampling rates of 8 kHz and 16 kHz is obtained according to the speech signal oscillograms after pre-emphasis processing shown in FIG. 4A and FIG. 4B.

[0107] In some embodiments, the discrete audio signal of the pre-emphasis processed audio signal oscillogram shown in FIG. 4A and FIG. 4B may be

[0108]

number

[0109] and

[0110]

number

[0111] For example, the discrete audio signals of the audio signal oscillograms after pre-emphasis processing shown in FIG. 4A and FIG. 4B are

[0112]

number

[0113] and

[0114]

number

[0115] is the first linear spectrum corresponding to the training sound with sampling rates of 8 kHz and 16 kHz.

[0116]

number

[0117] and

[0118]

number

[0119] To obtain the four short-time Fourier transforms,

[0120]

number

[0121] and

[0122]

number

[0123] By using 1 Point Short-Time Fourier Transform and n 2 Point short-time Fourier transform of discrete audio signal

[0124]

number

[0125] and

[0126]

number

[0127] This is performed for each.

[0128] For example, the size of the STFT sliding window is 20 ms, and the sliding stride of the sliding window is 10 ms. It should be understood that the sliding stride of the window is only necessary to ensure a certain overlap area between the sliding windows.

[0129] In some embodiments, the discrete speech signals after pre-emphasis processing for training sounds with sampling rates of 8 kHz and 16 kHz have 20 ms x 8 kHz = 160 points and 20 ms x 16 kHz = 320 points in a sliding window of 20 ms, respectively. 1 and n 2 The value of must be an exponent of 2 to realize parallel processing and improve processing speed. However, the smallest number that is greater than 160 and 320 for exponent 2 is 256 and 512, respectively. Therefore, n 1 = 256 and n 2 =512.

[0130] Calculated n 1 and n 2 According to, the discrete speech signals after pre-emphasis processing performed on the training sounds with sampling rates of 8 kHz and 16 kHz can undergo 256-point STFT and 512-point STFT, respectively.

[0131] According to the sampling theorem, the training sound with a sampling rate of 8kHz can only have a maximum of 4kHz of spectral information, and the 256-point STFT divides -4kHz to 4kHz evenly into 256 parts. By combining the positive frequency 4kHz and the negative frequency -4kHz, the resulting dimension of the spectral sequence is 256÷2+1=129.

[0132] By using the 256-point STFT, a 129-dimensional spectrum sequence of each discrete speech signal with a sampling rate of 8 kHz can be obtained as the first linear spectrum of the training sound with a sampling rate of 8 kHz.

[0133] FIG. 5A shows a first linear spectrum corresponding to a training sound having a sampling rate of 8 kHz according to some embodiments of the present disclosure.

[0134] As shown in Fig. 5A, the abscissa of the first linear spectrum corresponding to the training sound with a sampling rate of 8 kHz is the spectrum sequence serial number, and its ordinate is the frequency domain serial number. The value of the coordinate point determined by the abscissa and ordinate is the amplitude value corresponding to the training sound with a sampling rate of 8 kHz. Each frequency domain serial number corresponds to one frequency range.

[0135] For example, the linear spectrum frequency range of a training sound having a sampling rate of 8 kHz is 0 to 4 kHz, and the frequency domain serial number of a first linear spectrum corresponding to the training sound having a sampling rate of 8 kHz is 0 to 128, that is, 0 to 4 kHz is divided into 128 segments, and each frequency domain serial number corresponds to one linear spectrum frequency range.

[0136] According to Fig. 4B, it can be seen that the maximum number of sampling points of the discrete speech signal after pre-emphasis processing is 20000, that is, the 8kHz training sound has a total of 20000 sampling points. 8kHz indicates 8000 sampling points per second, and the total duration of the 8kHz sound with 20000 sampling points is 20000 / 8000=2.5 ​​seconds, or 2500 milliseconds.

[0137] For example, one stride of the sliding window is 10 milliseconds, and the number of slide strides is 2500 / 10-1=249 strides. Therefore, the spectral sequence serial numbers of the first linear spectrum corresponding to the sound to be trained with a sampling rate of 8 kHz are 0 to 248. The frequency domain serial numbers are 0 to 128. Each spectral sequence serial number and each frequency domain serial number uniquely determine one amplitude value. The 129 amplitude values ​​corresponding to each spectral sequence serial number constitute a spectral sequence corresponding to the spectral sequence serial number. Similarly, by the 512-point STFT, a 257-dimensional spectral sequence of each discrete speech signal with a sampling rate of 16 kHz can be obtained as the first linear spectrum of the sound to be trained with a sampling rate of 16 kHz.

[0138] FIG. 5B illustrates a first linear spectrum corresponding to a training sound having a sampling rate of 16 kHz, according to some embodiments of the present disclosure.

[0139] As shown in FIG. 5B, the abscissa of the first linear spectrum corresponding to the sound to be trained with a sampling rate of 16 kHz is the spectral sequence serial number, and its ordinate is the frequency domain serial number. The value of the coordinate point determined by the abscissa and the ordinate is an amplitude value corresponding to the sound to be trained with a sampling rate of 16 kHz. The spectral sequence serial numbers of the first linear spectrum corresponding to the sound to be trained with a sampling rate of 16 kHz range from 0 to 498, and the frequency domain serial numbers range from 0 to 256. Each spectral sequence serial number and each frequency domain serial number uniquely determines one amplitude value. The 257 amplitude values ​​corresponding to each spectral sequence serial number constitute a spectral sequence corresponding to the spectral sequence serial number.

[0140] Returning to FIG. 1, after the first linear spectrum corresponding to the training sound with different sampling rates is obtained, step S120 is performed. In step S120, among the different sampling rates, a maximum sampling rate and other sampling rates other than the maximum sampling rate are determined. For example, taking the training sound with different sampling rates of 8 kHz and 16 kHz as an example, the maximum sampling rate is 16 kHz and the other sampling rates are 8 kHz.

[0141] In step S130, the maximum frequency domain serial number of the first linear spectrum corresponding to other sampling rates is determined as the first frequency domain serial number. In some embodiments, the maximum frequency domain serial number of the first linear spectrum corresponding to 8kHz is determined as the first frequency domain serial number. For example, take the training sound with different sampling rates of 8kHz and 16kHz according to FIG. 5A as an example, the first frequency domain serial number is 128.

[0142] In step S140, the maximum frequency domain serial number of the first linear spectrum corresponding to the maximum sampling rate is determined as the second frequency domain serial number. In some embodiments, the maximum frequency domain serial number of the first linear spectrum corresponding to 16kHz is determined as the second frequency domain serial number. For example, take the training sound with different sampling rates of 8kHz and 16kHz according to FIG. 5B as an example, the second frequency domain serial number is 256.

[0143] In step S150, to obtain a second linear spectrum corresponding to another sampling rate, an amplitude value corresponding to each frequency domain serial number that is greater than the first frequency domain serial number and less than or equal to the second frequency domain serial number in the first linear spectrum corresponding to the other sampling rate is set to 0.

[0144] For example, in Fig. 5A, to obtain a second linear spectrum corresponding to 8 kHz, as shown in Fig. 6, the amplitude value corresponding to each frequency domain serial number that is greater than 128 and less than or equal to 256 is set to 0. The number of frequency domain serial numbers of the second linear spectrum is the same as the number of frequency domain serial numbers of the first linear spectrum corresponding to 16 kHz, so that mixed training can be performed on the machine learning model by using training sounds with different sampling rates.

[0145] FIG. 6 shows a second linear spectrum corresponding to a training sound having a sampling rate of 8 kHz according to some embodiments of the present disclosure.

[0146] As shown in Fig. 6, the abscissa of the second linear spectrum corresponding to the training sound with a sampling rate of 8 kHz is the spectrum sequence serial number, and its ordinate is the frequency domain serial number. The value of the coordinate point determined by the abscissa and ordinate is the amplitude value corresponding to the sound with a sampling rate of 8 kHz.

[0147] Compared with FIG. 5A, in FIG. 6, the amplitude values ​​corresponding to the frequency domain serial numbers 129 to 255 are added, and the amplitude values ​​are all 0. The zero padding does not destroy the original spectrum information of the sound, and does not destroy the original speech features. Since the zero padding is performed on the first linear spectrum corresponding to the 8 kHz sound, the frequency range of the second linear spectrum of the 8 kHz sound to be trained and the frequency range of the first linear spectrum of the 16 kHz sound to be trained are both 0 to 8 kHz.

[0148] Returning to Fig. 1, after the second linear spectrum corresponding to the other sampling rate is obtained, step S160 is performed. In step S160, the first speech feature of the maximum sampling rate to be trained acoustic is determined according to the first mel-spectrum feature of the first linear spectrum corresponding to the maximum sampling rate. The mel-spectrum feature is also called a mel spectrogram.

[0149] FIG. 7 is a flow diagram illustrating steps for determining a first audio feature of a maximum sampling rate training sound according to some embodiments of the present disclosure.

[0150] As shown in FIG. 7, the step of determining the first speech feature of the maximum sampling rate training sound includes steps S161 to S162.

[0151] In step S161, a mel-filtering transform is performed on the first linear spectrum corresponding to the maximum sampling rate by using a plurality of unit triangular filters or a plurality of unit square-wave filters to obtain a first mel-spectral feature, where both the unit triangular filters and the unit square-wave filters are linear mel filters.

[0152] Those skilled in the art should understand that the mel spectrum feature is another spectral feature obtained by filtering and transforming the linear spectrum feature. Transforming the linear spectrum feature into the mel spectrum feature by the mel filtering transform is transforming the linear spectrum feature into a log-linear spectrum. However, in speech recognition, it is generally considered that the speech information of the low frequency part is richer than that of the high frequency part due to the difference between the features of high and low frequencies. Due to the mel spectrum feature obtained by performing the mel filtering transform on the linear spectrum, the resolution of the mel spectrum corresponding to the low frequency part of the sound is higher, and the resolution of the mel spectrum corresponding to the high frequency part is relatively lower, which is more beneficial for the extraction of speech features in the process of speech recognition. In this case, the high frequency and the low frequency are relative concepts, that is, all frequencies higher than the low frequency are high frequencies. For example, if 0 to 2 kHz is a low frequency, then a frequency higher than 2 kHz is a high frequency.

[0153] In some embodiments, the mapping relationship between the mel spectral frequencies and the linear spectral frequencies is:

[0154]

number

[0155] or f=700(10 m / 2595 −1), where f is the linear spectral frequency and m is the mel spectral frequency.

[0156] In some embodiments, the number of linear mel filters can be designed to obtain the mel spectrum feature of the corresponding dimension feature. For example, a mel filtering transformation is performed on the first linear spectrum corresponding to the maximum sampling rate by using multiple unit triangular filters. In some embodiments, the number of unit triangular filters is 80.

[0157] For example, the expression

[0158]

number

[0159] Each unit triangle filter is calculated by using H n (g) shows the nth unit triangle filter, where n is an integer between 1 and 80. n is the center frequency of each unit triangular filter. g is the linear spectrum frequency. f(m n ) is the center frequency m n For example, f(m n ) is the formula f=700(10 m / 2595 -1).

[0160] In some embodiments of the present disclosure, each unit triangular filter has a center frequency m n The calculation process will be described in detail later.

[0161] For example, the linear spectral frequency corresponding to the maximum mel-spectral frequency of each unit triangular filter is 8 kHz, i.e.

[0162]

number

[0163] The bandwidth of each unit triangular filter is

[0164]

number

[0165] The center frequency of each unit triangular filter is

[0166]

number

[0167] It is.

[0168] For example, for audio with a sampling rate of 16 kHz, the unit triangular filter H n After (g) is calculated, 80 unit triangle filters H 1 (g)~H n (g) is used to perform a mel-filtering transform on the first linear spectrum shown in FIG. 5B. For example, the mel-filtering transform formula

[0169]

number

[0170] By using the above, the first mel-spectral feature is calculated, as shown in FIG.

[0171]

number

[0172] is a matrix of second amplitude values ​​in the first mel-spectral feature.

[0173] FIG. 8 illustrates a first mel-spectral feature of a first linear spectrum corresponding to 16 kHz, according to some embodiments of the present disclosure.

[0174] As shown in Fig. 8, the abscissa of the first mel-spectral feature of the first linear spectrum corresponding to the sound to be trained with a sampling rate of 16 kHz is the spectral sequence serial number, which is the same as the spectral sequence serial number of the first linear spectrum. The ordinate of the first mel-spectral feature is the Mel filter serial number. The value of the coordinate point determined by the abscissa and the ordinate is the second amplitude value corresponding to the sound to be trained with a sampling rate of 16 kHz.

[0175] 7, after the first mel-spectral feature is obtained, execution continues with step S162, in which a global normalization process is performed on the first mel-spectral feature to obtain a first speech feature.

[0176] FIG. 9A is a flow diagram illustrating a global normalization process according to some embodiments of the present disclosure.

[0177] As shown in FIG. 9A, the global normalization process includes steps S1621 to S1625.

[0178] In step S1621, the second amplitude values ​​of the first mel spectral feature corresponding to each mel filter serial number are obtained. For example, taking the maximum sampling rate of 16 kHz as an example, the matrix of the second amplitude values ​​of the first mel spectral feature in FIG. 8 is as follows:

[0179]

number

[0180] It is.

[0181] In step S1622, the mean and standard deviation of the second amplitude values ​​are calculated as the global mean and the global standard deviation. For example, taking the maximum sampling rate of 16 kHz as an example, the global mean and the global standard deviation corresponding to the first mel-spectrum feature in FIG. 8 are respectively:

[0182]

number

[0183] and

[0184]

number

[0185] It is.

[0186] In step S1623, a second difference between each of the second amplitude values ​​and its global average is calculated. For example, taking the maximum sampling rate of 16 kHz as an example, the second difference corresponding to the first mel-spectral feature in FIG. 8 is:

[0187]

number

[0188] It is.

[0189] In step S1624, the ratio of each second difference to the global standard deviation is calculated as the normalized second amplitude value corresponding to each second amplitude value. For example, taking the maximum sampling rate of 16 kHz as an example, the matrix of the second amplitude values ​​of the first mel-spectral feature in FIG.

[0190]

number

[0191] The matrix of normalized second amplitude values ​​corresponding to

[0192]

number

[0193] It is.

[0194] In step S1625, each second amplitude value in the first mel-spectral feature is replaced with a normalized second amplitude value corresponding to each second amplitude value. For example, taking a maximum sampling rate of 16 kHz as an example,

[0195]

number

[0196] By using the above, the second amplitude value of the first mel-spectral feature in Fig. 8 is replaced with the normalized second amplitude value to obtain the first audio feature shown in Fig. 9B. The first mel-spectral feature after the global normalization process is the first audio feature.

[0197] FIG. 9B illustrates a first audio feature corresponding to a training sound having a sampling rate of 16 kHz, according to some embodiments of the present disclosure.

[0198] As shown in Fig. 9B, the abscissa of the first audio feature is the spectral sequence serial number, which is the same as the spectral sequence serial number of the first linear spectrum corresponding to the training sound of 16 kHz. Its ordinate is the Mel-filter serial number. The value of the coordinate point determined by the abscissa and ordinate is the second amplitude value after global normalization corresponding to the training sound with a sampling rate of 16 kHz.

[0199] In the present disclosure, by performing a global normalization process on the first mel-spectrum features, the convergence effect of training a machine learning model is improved and the influence of idiosyncratic data on training a machine learning model is suppressed.

[0200] Returning to Fig. 1, after the first speech feature of the maximum sampling rate to be trained is determined, step S170 is executed. It should be understood that the order of executing steps S160 and S170 can be set as necessary, and the execution order shown in Fig. 1 is merely an example.

[0201] In step S170, a second speech feature of the training acoustic of the other sampling rate is determined according to the second mel-spectrum feature of the second linear spectrum corresponding to the other sampling rate.

[0202] FIG. 10 is a flow diagram illustrating steps of determining a second audio feature of an acoustic to be trained for another sampling rate according to some embodiments of the present disclosure.

[0203] As shown in FIG. 10, the step of determining the second speech feature of the to-be-trained acoustic of another sampling rate includes steps S171 to S172.

[0204] In step S171, a mel-filtering transform is performed on the second linear spectrum corresponding to another sampling rate by using multiple unit triangular filters or multiple unit square wave filters to obtain a second mel-spectral feature.

[0205] For example, for a training sound with a sampling rate of 8 kHz, the unit triangle filter H n After [g] is calculated, 80 unit triangle filters H 1 [g]~H nBy using [g], a mel-filtering transform is performed on the second linear spectrum, as shown in Figure 6. For example, as shown in Figure 11, the second mel-spectral feature is expressed by the mel-filtering transform equation

[0206]

number

[0207] It is calculated by using

[0208]

number

[0209] is a matrix of first amplitude values ​​in the second mel-spectral feature.

[0210] FIG. 11 illustrates a second mel-spectral feature of a second linear spectrum corresponding to 8 kHz, according to some embodiments of the present disclosure.

[0211] As shown in Fig. 11, the abscissa of the second mel-spectrum feature of the second linear spectrum corresponding to the training sound with a sampling rate of 8 kHz is the spectral sequence serial number, which is the same as the spectral sequence serial number of the second linear spectrum. Its ordinate is the mel filter serial number. The value of the coordinate point determined by the abscissa and the ordinate is the first amplitude value corresponding to the training sound with a sampling rate of 8 kHz.

[0212] Returning to Fig. 10, after the second mel-spectral feature is obtained, step S172 is executed. In step S172, a local normalization process is performed on the second mel-spectral feature to obtain a second speech feature.

[0213] FIG. 12A is a flow diagram illustrating a local normalization process according to some embodiments of the present disclosure.

[0214] As shown in FIG. 12A, the local normalization process includes steps S1721 to S1727.

[0215] In step S1721, according to the maximum linear spectrum frequency corresponding to the sound of other sampling rates, the Mel spectrum frequency corresponding to the maximum linear spectrum frequency is obtained. The maximum linear spectrum frequency is also the maximum audio signal frequency.

[0216] For example, taking another sampling rate of 8 kHz as an example, according to the sampling theorem, the maximum linear spectrum frequency of the second linear spectrum corresponding to the training sound with a sampling rate of 8 kHz is 4 kHz.

[0217] In step S1722, the maximum Mel spectrum filter sequence number corresponding to the Mel spectrum frequency is calculated. For example, taking another sampling rate of 8 kHz as an example, the Mel spectrum frequency corresponding to 4 kHz is

[0218]

number

[0219] The formula is

[0220]

number

[0221] According to the Mel spectrum frequency m 1 The corresponding Mel filter sequence number is

[0222]

number

[0223] It is.

[0224] In step S1723, a first amplitude value of a second mel spectral feature corresponding to each other Mel filter serial number is obtained, where the other Mel filter serial numbers are Mel filter serial numbers that are less than or equal to the maximum Mel filter serial number.

[0225] For example, taking another sampling rate of 8 kHz as an example, the other Mel filter serial numbers are 1 to 61. In some embodiments, the matrix of the first amplitude values ​​corresponding to the other Mel filter serial numbers of the second Mel spectral feature in FIG.

[0226]

number

[0227] It is.

[0228] In step S1724, the mean and standard deviation of the first amplitude value are calculated as the local mean and local standard deviation, respectively. For example, taking another sampling rate of 8 kHz as an example, the local mean and local standard deviation corresponding to the second mel-spectrum feature in FIG. 11 are respectively:

[0229]

number

[0230] and

[0231]

number

[0232] It is.

[0233] In step S1725, a first difference between each of the first amplitude values ​​and its local average is calculated. For example, taking another sampling rate of 8 kHz as an example, the first difference corresponding to the second Mel-spectrum feature in FIG. 11 is:

[0234]

number

[0235] It is.

[0236] In step S1726, the ratio of each first difference to the local standard deviation is calculated as the normalized first amplitude value corresponding to each first amplitude value. For example, taking another sampling rate of 8 kHz as an example, the matrix of the first amplitude values ​​corresponding to the other Mel filter serial numbers in the second Mel spectrum feature in FIG. 11 is

[0237]

number

[0238] and the corresponding matrix of normalized first amplitude values ​​is

[0239]

number

[0240] It is.

[0241] In step S1727, each first amplitude value in the second mel-spectral feature is replaced with a normalized first amplitude value corresponding to each first amplitude value.

[0242] For example, take another sampling rate of 8kHz as an example:

[0243]

number

[0244] By using the normalized first amplitude values ​​corresponding to other Mel filter sequence numbers in the second Mel-spectral feature of FIG. 11 are replaced with the normalized first amplitude values ​​to obtain a second audio feature as shown in FIG. 12B.

[0245] In the case of the second Mel-spectral feature of FIG. 11, in which the other 19 dimensions, not the 1st to 61st dimensions, are formed by zero padding, the first amplitude value remains unchanged. For example,

[0246]

number

[0247] It is.

[0248]

number

[0249] is a matrix of first amplitude values ​​of the 62nd to 80th dimensions of the second mel-spectral feature.

[0250]

number

[0251] is a matrix of the first amplitude values ​​of the second mel-spectral feature of 19 dimensions, that is, the 62nd to 80th dimensions, after the local normalization process.

[0252] FIG. 12B is a diagram illustrating a second audio feature corresponding to a training sound having a sampling rate of 8 kHz, according to some embodiments of the present disclosure.

[0253] As shown in Fig. 12B, the abscissa of the second audio feature is the spectral sequence serial number, which is the same as the spectral sequence serial number of the second linear spectrum. The ordinate is the Mel-filter serial number. The value of the coordinate point determined by the abscissa and the ordinate is the first amplitude value after local normalization corresponding to the training sound with a sampling rate of 8 kHz.

[0254] Since the high frequency zero padding is performed on the first linear spectrum of the sound to be trained with another sampling rate in step S150 as shown in Fig. 1, in the present disclosure, the problem of suppressing the original speech features of the sound to be trained caused by the normalization process for the features with a large number of averages and standard deviations of 0, which is brought about by zero padding the second mel-spectral features, is effectively avoided by performing the local normalization process on the second mel-spectral features, and the suppression of the original speech features of the sound to be trained can be better prevented. In this case, the high frequency indicates the linear spectral frequency corresponding to the frequency domain serial number that is greater than the first frequency domain serial number and less than or equal to the second frequency domain serial number.

[0255] Returning to FIG. 1, after the first and second voice features are determined, step S180 is executed. In step S180, a machine learning model is trained by using the first and second voice features. In some embodiments, the machine learning model includes a Deep Neural Network (DNN) model. For example, the DNN model is an end-to-end based DNN model.

[0256] For example, taking different sampling rates of 8 kHz and 16 kHz as an example, the machine learning model is trained by using the first audio feature as shown in FIG. 9B and by using the second audio feature as shown in FIG. 12B.

[0257] It should be understood that in the above embodiments, Figures 5A to 6, 8, 11, 9B, and 12B are merely schematic diagrams, and here the magnitude of the amplitude value is determined by the brightness.

[0258] In some embodiments, after the machine learning model is trained, speech recognition for the sound to be recognized can also be performed by using the machine learning model.

[0259] Speech recognition of the sound to be recognized using a machine learning model is implemented, for example, in the following manner.

[0260] First, the sound to be recognized is obtained, for example, at one of different sampling rates.

[0261] Then, the speech characteristics of the sound to be recognized are determined.

[0262] In some embodiments, first, the maximum frequency domain serial number of the first linear spectrum of the sound to be recognized is determined as the third frequency domain serial number. Then, to obtain the second linear spectrum of the sound to be recognized, the amplitude value corresponding to each frequency domain serial number in the first linear spectrum of the sound to be recognized that is greater than the third frequency domain serial number and less than or equal to the second frequency domain serial number is set to zero. Finally, the speech features of the sound to be recognized are determined according to the mel-spectrum features of the second linear spectrum of the sound to be recognized.

[0263] It should be understood that in the speech recognition process, in order to perform speech recognition by using the speech features of the sound to be recognized, it is also necessary to obtain the speech features of the sound to be recognized by using a method similar to the method of obtaining the speech features of the sound to be trained.

[0264] Finally, the speech features of the audio to be recognized are input to a machine learning model to obtain a speech recognition result, for example, the audio to be recognized is converted into text information, and the text information is the speech recognition result.

[0265] In the present disclosure, by performing zero padding on the first linear spectrum corresponding to the training sound of other sampling rates less than the maximum sampling rate, the effect of mixed training on the training sound of different sampling rates is obtained on the basis of not adding or subtracting voice features, and the original voice features of the different sampling rate sound data can be more accurately extracted, so that the recognition performance of the machine learning model is improved. Compared with the method of training the machine learning model by using the sound of a single sampling rate, in the present disclosure, the problem of insufficient training samples caused by the limited number of sounds of a single sampling rate is solved, and the problem of low universality of the machine learning model caused by the low speech recognition effect of the machine learning model trained by using the sound data of a single sampling rate on the sound data of another sampling rate is solved, so that the universality of the machine learning model is improved.

[0266] FIG. 13 is a block diagram illustrating a speech recognition device according to some embodiments of the present disclosure.

[0267] As shown in FIG. 13, the speech recognition device 13 includes a first acquisition module 131, a first decision module 132, a second decision module 133, a third decision module 134, a setting module 135, a fourth decision module 136, a fifth decision module 137, and a training module 138.

[0268] The first acquisition module 131 is configured to acquire a first linear spectrum corresponding to the training sound with a different sampling rate, where the abscissa of the first linear spectrum is a spectrum sequence serial number, the ordinate of the first linear spectrum is a frequency domain serial number, and the value of the coordinate point determined by the abscissa and the ordinate is an original amplitude value corresponding to the training sound for performing step S110, for example as shown in FIG. 1.

[0269] The first determination module 132 is configured to determine, among the different sampling rates, a maximum sampling rate and other sampling rates other than the maximum sampling rate, to perform, for example, step S120 as shown in FIG. 1.

[0270] The second determination module 133 is configured to determine the maximum frequency domain serial number of the first linear spectrum corresponding to the other sampling rate as the first frequency domain serial number, for example, to perform step S130 shown in FIG. 1.

[0271] The third determination module 134 is configured to determine a maximum frequency domain serial number of the first linear spectrum corresponding to a maximum sampling rate as the second frequency domain serial number, for example, to perform step S140 shown in FIG. 1.

[0272] The setting module 135 is configured to set amplitude values ​​corresponding to each frequency domain serial number that is greater than the first frequency domain serial number and less than or equal to the second frequency domain serial number in the first linear spectrum corresponding to the other sampling rate to zero, for example, to perform step S150 shown in FIG. 1, in order to obtain a second linear spectrum corresponding to the other sampling rate.

[0273] The fourth determination module 136 is configured to determine a first speech feature of the acoustic to be trained at the maximum sampling rate according to the first mel-spectrum feature of the first linear spectrum corresponding to the maximum sampling rate, for example, to perform step S160 as shown in FIG. 1.

[0274] The fifth determination module 137 is configured to determine a second speech feature of the training acoustic of the other sampling rate according to the second mel-spectrum feature of the second linear spectrum corresponding to the other sampling rate, for example, to perform step S170 shown in FIG. 1.

[0275] The training module 138 is configured to train a machine learning model by using the first audio feature and the second audio feature, for example, to perform step S180 as shown in FIG. 1 .

[0276] FIG. 14 is a block diagram illustrating a voice recognition device according to another embodiment of the present disclosure.

[0277] As shown in Fig. 14, the speech recognition device 14 includes a memory 141 and a processor 142 coupled to the memory 141. The memory 141 is configured to store instructions for performing a corresponding embodiment of a speech recognition method. The processor 142 is configured to perform the speech recognition method in any embodiment of the present disclosure based on the instructions stored in the memory 141.

[0278] FIG. 15 is a block diagram illustrating a computer system for implementing some embodiments of the present disclosure.

[0279] As shown in Figure 15, computer system 150 may take the form of a general purpose computing device. Computer system 150 includes a memory 1510, a processor 1520, and a bus 1500 connecting different system components.

[0280] The memory 1510 may include, for example, a system memory, a non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, other programs, etc. The system memory may include a volatile storage medium, such as a random access memory (RAM) and / or a cache memory. The non-volatile storage medium may store, for example, instructions for performing at least one corresponding embodiment of the speech recognition method. Non-volatile storage media may include, but are not limited to, a magnetic disk memory, an optical memory, a flash memory, etc.

[0281] The processor 1520 may be implemented by discrete hardware components such as a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gates or transistors, etc. Thus, each module such as the judgment module and the decision module may be implemented by a central processing unit (CPU) executing instructions in a memory that perform the corresponding steps, or may be implemented by a dedicated circuit that performs the corresponding steps.

[0282] Bus 1500 may use any of a variety of bus architectures, including, but not limited to, Industry Standard Architecture (ISA) bus, MicroChannel Architecture (MCA) bus, and Peripheral Component Interconnect (PCI) bus.

[0283] The computer system 150 may also include an input / output interface 1530, a network interface 1540, a storage interface 1550, and the like. These interfaces 1530, 1540, 1550, as well as the memory 1510 and the processor 1520 may be connected via a bus 1500. The input / output interface 1530 may provide a connection interface for input / output devices such as a display, a mouse, a keyboard, and the like. The network interface 1540 provides a connection interface for various network devices. The storage interface 1550 provides a connection interface for external storage devices such as a floppy disk, a USB disk, an SD card, and the like.

[0284] Various aspects of the present disclosure have been described herein with reference to flow diagrams and / or block diagrams of methods, apparatus, and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flow diagrams and / or block diagrams, and combinations of blocks, can be implemented by computer readable program instructions.

[0285] These computer-readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable apparatus to generate a machine, whereby execution of the instructions by the processor generates an apparatus for implementing the functions specified in one or more blocks in the flow diagrams and / or block diagrams.

[0286] These computer-readable program instructions are also stored in a computer-readable memory and enable a computer to operate in a particular manner to produce an article of manufacture that includes instructions that implement the functions specified in one or more blocks in the flow diagrams and / or block diagrams.

[0287] The present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects.

[0288] The speech recognition method and apparatus, and computer-storable medium in the above embodiments can accurately extract original speech features of acoustic data with different sampling rates, thereby improving the speech recognition performance of the machine learning model.

[0289] Up to now, the speech recognition method and device, and computer-storable medium according to the present disclosure have been described in detail. Some details well known in the art have not been described to avoid obscuring the concept of the present disclosure. In view of the above description, those skilled in the art can fully understand how to implement the technical solutions disclosed herein. [Explanation of symbols]

[0290] 13 Voice recognition device 14 Voice recognition device 131 First Acquisition Module 132 First Decision Module 133 Second Decision Module 134 Third Decision Module 135 Settings Module 136 Fourth Decision Module 137 Fifth Decision Module 138 Training Module 141 Memory 142 processors 150 Computer Systems 1500 Bus 1510 Memory 1520 Processor 1530 Input / Output Interface 1540 Network Interface 1550 Storage Interface

Claims

1. obtaining a first linear spectrum corresponding to a training sound having a different sampling rate, the abscissa of the first linear spectrum being a spectral sequence serial number, the ordinate of the first linear spectrum being a frequency domain serial number, and a value of a coordinate point determined by the abscissa and the ordinate being an original amplitude value corresponding to the training sound; determining a maximum sampling rate and another sampling rate other than the maximum sampling rate among the different sampling rates; determining a maximum frequency domain serial number of the first linear spectrum corresponding to the other sampling rate as a first frequency domain serial number; determining a maximum frequency domain serial number of the first linear spectrum corresponding to the maximum sampling rate as a second frequency domain serial number; setting an amplitude value corresponding to each frequency domain serial number that is greater than the first frequency domain serial number and less than or equal to the second frequency domain serial number in the first linear spectrum corresponding to the other sampling rate to zero to obtain a second linear spectrum corresponding to the other sampling rate; determining a first speech feature of the training sound of the maximum sampling rate according to a first mel-spectral feature of the first linear spectrum corresponding to the maximum sampling rate; determining a second speech feature of the training sound of the other sampling rate according to a second mel-spectral feature of the second linear spectrum corresponding to the other sampling rate, and performing a local normalization process on the second mel-spectral feature to obtain the second speech feature; training a machine learning model by using the first speech feature and the second speech feature; Including, The local normalization process comprises: According to the maximum linear spectral frequency corresponding to the training sound of the other sampling rate, obtain a Mel-spectral frequency corresponding to the maximum linear spectral frequency; calculating a maximum Mel filter sequence number corresponding to the Mel spectral frequency; obtaining first amplitude values ​​corresponding to each other Mel filter serial number in the second mel spectral feature, the other Mel filter serial numbers being Mel filter serial numbers less than or equal to the maximum Mel filter serial number; calculating a mean and a standard deviation of all the first amplitude values ​​as a local mean and a local standard deviation, respectively; calculating a first difference between each of the first amplitude values ​​and the local average of the first amplitude values; calculating a ratio of each first difference to the local standard deviation as a normalized first amplitude value corresponding to each first amplitude value; replacing each first amplitude value in the second mel-spectral feature with the normalized first amplitude value corresponding to each first amplitude value; A speech recognition method comprising:

2. A step of obtaining a first linear spectrum corresponding to a training sound having a different sampling rate, wherein an abscissa of the first linear spectrum is a spectral sequence serial number, an ordinate of the first linear spectrum is a frequency domain serial number, and a value of a coordinate point determined by the abscissa and the ordinate is an original amplitude value corresponding to the training sound; determining a maximum sampling rate and another sampling rate other than the maximum sampling rate among the different sampling rates; determining a maximum frequency domain serial number of the first linear spectrum corresponding to the other sampling rate as a first frequency domain serial number; determining a maximum frequency domain serial number of the first linear spectrum corresponding to the maximum sampling rate as a second frequency domain serial number; setting an amplitude value corresponding to each frequency domain serial number that is greater than the first frequency domain serial number and less than or equal to the second frequency domain serial number in the first linear spectrum corresponding to the other sampling rate to zero to obtain a second linear spectrum corresponding to the other sampling rate; determining a first speech feature of the training sound of the maximum sampling rate according to a first mel-spectral feature of the first linear spectrum corresponding to the maximum sampling rate; determining a second speech feature of the training sound of the other sampling rate according to a second mel-spectrum feature of the second linear spectrum corresponding to the other sampling rate; training a machine learning model by using the first speech feature and the second speech feature; Including, The global normalization process comprises: obtaining second amplitude values ​​corresponding to each Mel filter sequence number in the first Mel spectral feature; calculating a mean and a standard deviation of all the second amplitude values ​​as a global mean and a global standard deviation; calculating a second difference between each of the second amplitude values ​​and the global average of the second amplitude values; calculating a ratio of each second difference to the global standard deviation as a normalized second amplitude value corresponding to each second amplitude value; replacing each second amplitude value in the first mel-spectral feature with the normalized second amplitude value corresponding to each second amplitude value; A speech recognition method comprising:

3. The step of obtaining first linear spectra corresponding to training sounds having different sampling rates includes: obtaining the first linear spectra corresponding to the training sounds at the different sampling rates by using a short-time Fourier transform, respectively; 3. The speech recognition method according to claim 1, further comprising:

4. The step of obtaining first linear spectra corresponding to training sounds having different sampling rates includes: acquiring oscillograms of the training sound signals at the different sampling rates; performing pre-emphasis processing on the speech signal oscillograms of the training sounds having the different sampling rates; obtaining the first linear spectrum corresponding to the training sound of the different sampling rates according to the audio signal oscillogram after the pre-emphasis processing; 3. The speech recognition method according to claim 1, further comprising:

5. performing a mel-filtering transformation on the first linear spectrum corresponding to the maximum sampling rate and the second linear spectrum corresponding to the other sampling rate by using a plurality of unit triangular filters to obtain the first mel-spectral feature and the second mel-spectral feature, respectively.

3. The speech recognition method according to claim 1, further comprising:

6. 3. The speech recognition method of claim 1, wherein the machine learning model comprises a deep neural network (DNN) model.

7. 3. The speech recognition method according to claim 1, wherein the different sampling rates include 16 kHz and 8 kHz.

8. obtaining a sound to be recognized; determining speech characteristics of the sound to be recognized; inputting the speech features of the sound to be recognized into the machine learning model to obtain a speech recognition result; 3. The speech recognition method according to claim 1, further comprising:

9. The step of determining speech characteristics of the sound to be recognized comprises: determining a maximum frequency domain serial number of the first linear spectrum of the sound to be recognized as a third frequency domain serial number; setting an amplitude value corresponding to each frequency domain serial number that is greater than the third frequency domain serial number and less than or equal to the second frequency domain serial number in the first linear spectrum of the sound to be recognized to zero to obtain a second linear spectrum of the sound to be recognized; determining the speech characteristics of the sound to be recognized according to the mel-spectral characteristics of the second linear spectrum of the sound to be recognized; 9. The speech recognition method of claim 8, comprising:

10. a first acquisition module configured to acquire a first linear spectrum corresponding to a training sound having a different sampling rate, the abscissa of the first linear spectrum being a spectrum sequence serial number, the ordinate of the first linear spectrum being a frequency domain serial number, and a value of a coordinate point determined by the abscissa and the ordinate being an original amplitude value corresponding to the training sound; a first determination module configured to determine, among the different sampling rates, a maximum sampling rate and another sampling rate other than the maximum sampling rate; a second determining module configured to determine a maximum frequency domain sequence number of the first linear spectrum corresponding to the other sampling rate as a first frequency domain sequence number; a third determination module configured to determine a maximum frequency domain sequence number of the first linear spectrum corresponding to the maximum sampling rate as a second frequency domain sequence number; a setting module configured to set an amplitude value corresponding to each frequency domain serial number that is greater than the first frequency domain serial number and less than or equal to the second frequency domain serial number in the first linear spectrum corresponding to the other sampling rate to zero to obtain a second linear spectrum corresponding to the other sampling rate; a fourth determination module configured to determine a first speech feature of the training sound of the maximum sampling rate according to a first mel-spectral feature of the first linear spectrum corresponding to the maximum sampling rate; a fifth determination module configured to determine a second speech feature of the training sound of the other sampling rate according to a second mel-spectral feature of the second linear spectrum corresponding to the other sampling rate, the fifth determination module performing a local normalization process on the second mel-spectral feature to obtain the second speech feature; a training module configured to train a machine learning model by using the first speech feature and the second speech feature; Equipped with The local normalization process comprises: According to the maximum linear spectral frequency corresponding to the training sound of the other sampling rate, obtain a Mel-spectral frequency corresponding to the maximum linear spectral frequency; calculating a maximum Mel filter sequence number corresponding to the Mel spectral frequency; obtaining first amplitude values ​​corresponding to each other Mel filter serial number in the second mel spectral feature, the other Mel filter serial numbers being Mel filter serial numbers less than or equal to the maximum Mel filter serial number; calculating a mean and a standard deviation of all the first amplitude values ​​as a local mean and a local standard deviation, respectively; calculating a first difference between each of the first amplitude values ​​and the local average of the first amplitude values; calculating a ratio of each first difference to the local standard deviation as a normalized first amplitude value corresponding to each first amplitude value; replacing each first amplitude value in the second mel-spectral feature with the normalized first amplitude value corresponding to each first amplitude value; A speech recognition device comprising:

11. A first acquisition module configured to acquire a first linear spectrum corresponding to a training sound having a different sampling rate, wherein an abscissa of the first linear spectrum is a spectral sequence serial number, an ordinate of the first linear spectrum is a frequency domain serial number, and a value of a coordinate point determined by the abscissa and the ordinate is an original amplitude value corresponding to the training sound; and a first determination module configured to determine, among the different sampling rates, a maximum sampling rate and another sampling rate other than the maximum sampling rate; a second determining module configured to determine a maximum frequency domain sequence number of the first linear spectrum corresponding to the other sampling rate as a first frequency domain sequence number; a third determination module configured to determine a maximum frequency domain sequence number of the first linear spectrum corresponding to the maximum sampling rate as a second frequency domain sequence number; a setting module configured to set an amplitude value corresponding to each frequency domain serial number that is greater than the first frequency domain serial number and less than or equal to the second frequency domain serial number in the first linear spectrum corresponding to the other sampling rate to zero to obtain a second linear spectrum corresponding to the other sampling rate; a fourth determination module configured to determine a first speech feature of the training sound of the maximum sampling rate according to a first mel-spectral feature of the first linear spectrum corresponding to the maximum sampling rate; a fifth determination module configured to determine a second speech feature of the training sound of the other sampling rate according to a second mel-spectrum feature of the second linear spectrum corresponding to the other sampling rate; a training module configured to train a machine learning model by using the first speech feature and the second speech feature; Equipped with The global normalization process comprises: obtaining second amplitude values ​​corresponding to each Mel filter sequence number in the first Mel spectral feature; calculating a mean and a standard deviation of all the second amplitude values ​​as a global mean and a global standard deviation; calculating a second difference between each of the second amplitude values ​​and the global average of the second amplitude values; calculating a ratio of each second difference to the global standard deviation as a normalized second amplitude value corresponding to each second amplitude value; replacing each second amplitude value in the first mel-spectral feature with the normalized second amplitude value corresponding to each second amplitude value; A speech recognition device comprising:

12. Memory, a processor coupled to said memory, said processor being configured to perform the speech recognition method according to any one of claims 1 to 9 based on computer program instructions stored in said memory. Speech recognition device.

13. A computer-storable medium having stored thereon computer program instructions which, when executed by a processor, implements the speech recognition method of any one of claims 1 to 9.

Images