A novel dynamic temporal recurrent neural network for dialect recognition

Abstract

The application discloses a novel dynamic time delay neural network for dialect recognition, and belongs to the field of acoustic model modeling; specifically, an existing time delay neural network is introduced into k parallel convolution kernels, attention weight parameters of the respective convolution kernels are calculated, and an optimal weight parameter convolution kernel is obtained through weighted average fusion; the optimal weight parameter convolution kernel is used to replace a conventional convolution kernel of the existing time delay neural network, so that an improved dynamic time delay neural network is obtained; according to different dialect inputs, each weight parameter in the optimal weight parameter convolution kernel is dynamically adjusted, deeper feature information of audio is extracted, comparison and analysis are conducted on the extracted deep features and a dialect acoustic template, and finally, a classifier is combined to determine a dialect category; and the application improves the accuracy of determining the dialect category to which the audio belongs.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of acoustic model modeling, and particularly relates to a novel dynamic time delay neural network for dialect recognition. BACKGROUND

[0002] With the development of emerging technologies such as machine learning and artificial intelligence, the speech recognition technology for Mandarin has been sufficient to meet the requirements of most application scenarios, however, in the cultural heritage and regional communication scenarios, dialects still occupy a dominant position, and the complex speech characteristics and lack of standardized data lead to serious challenges for existing recognition systems. Especially for dialects such as Minnan and Cantonese, which have rich tones and regional variations, the subtle differences in pronunciation cause the accuracy of general speech recognition models to drop dramatically.

[0003] In the current research on dialect recognition, there are mainly two methods:

[0004] One is the phased dialect recognition: first, identify which dialect the speaker uses, and then use the model specific to the dialect to determine the content of the speech; the advantage of this method is that it can take advantage of the specific characteristics of each dialect to improve the accuracy of recognition. In this way, the model can be optimized for the speech characteristics of different dialects.

[0005] The other is unified model training: directly use the data of all dialects to train a unified model, and then use this model to recognize the content of the dialect; this method only needs to maintain one model, rather than multiple dialect-specific models; simplifies the management and deployment of the model; however, this method may not perform as well as a specially trained model on the characteristics of some dialects.

[0006] Time delay neural networks are mainly used for processing one-dimensional time series tasks, and have better performance in acoustic modeling than traditional HMM time series models due to their fewer parameters; however, it is particularly important in dialect recognition to accurately determine the dialect category. The acoustic model built by the ordinary time delay neural network has limited representation ability, and cannot achieve high recognition rate for all dialect categories at the same time, which may lead to errors in selecting dialect acoustic models and thus convert to incorrect speech content. Based on this, the time delay neural network model first identifies the category of the dialect, and then further identifies the content of the speech for the specific dialect, which can improve the accuracy and precision of dialect content recognition. SUMMARY

[0007] Based on the above problems, the present application constructs a novel dynamic time delay neural network, which improves the accuracy of dialect category recognition, identifies the dialect category used by the speaker, and then uses the acoustic model specific to the dialect for phased dialect recognition to determine the content of the speech.

[0008] The new dynamic time delay neural network for dialect recognition has the following specific structure: the convolution kernel of the existing time delay neural network is updated: firstly, k parallel convolution kernels are introduced, the attention weight parameters of each are calculated, and then an optimal weight parameter convolution kernel is obtained by weighted average fusion; finally, the optimal weight parameter convolution kernel is replaced with the conventional convolution kernel of the existing time delay neural network to obtain an improved dynamic time delay neural network; when different dialects are input, the weight parameters in the optimal weight parameter convolution kernel are dynamically adjusted, deeper feature information of the audio is extracted, the extracted deep features are compared and analyzed with the dialect acoustic template, and finally the dialect category is determined in combination with the classifier;

[0009] The specific updating process is as follows:

[0010] Step one, the input audio features are globally averaged and pooled to obtain global spatial features with a length of C;

[0011] The input audio features are a HxWxC tensor;

[0012] The global average pooling operation is to calculate the mean value of each channel in the spatial dimension, and the output of the global average pooling of the cth channel is GlobalFeature c The formula is as follows:

[0013]

[0014] Where H is the height of the feature map, W is the width of the feature map, and C is the total number of channels. X i,j,c represents the feature value of the cth channel at position (i,j), c∈C.

[0015] Step two, the global spatial features are mapped to k dimensions in turn through two fully connected layers, and are normalized by softmax to obtain the weight coefficients of the k convolution kernels;

[0016] The formula is as follows:

[0017]

[0018] Where, Z k is the kth output of the second fully connected layer, and Softmax(Z k ) is the weight coefficient of the kth convolution kernel.

[0019] Step three, weight all parallel convolution kernel weights according to their weight coefficients, and fuse to obtain a new convolution kernel;

[0020] The weight of each convolution kernel is W conv,k ∈R N×N×C , where N is the size of the convolution kernel;

[0021] The new convolution kernel W is obtained through weighted average new , and the formula is:

[0022]

[0023] pi k is the weight coefficient of the kth convolution kernel, W conv,k is the original weight of the kth convolution kernel.

[0024] Step four, input the new convolution kernel into the time delay neural network model, so that the deep features in the input dialect can be adaptively captured.

[0025] The improved dynamic time delay neural network model, and the output formula is:

[0026]

[0027] C t ={X t-n , X t+n}

[0028]

[0029] Y t ' is the output feature, f is an activation function, W T and b are fixed weight matrix and bias vector respectively, C t is the input feature, X t-n is the number of context information frames input to the lower layer network at t time, and X t+n is the number of context information frames input to the lower layer network at t time, and n is greater than or equal to 1.

[0030] The improved time delay neural network model can adaptively select the optimal feature extraction mode according to the audio content of the input dialect: in the transient stage of dialect pronunciation, the short-time impulse response is strengthened, and in the steady state stage, the resonance peak tracking ability is enhanced.

[0031] Step five, through comparison and analysis of the extracted deep features and the dialect acoustic template, finally combined with the classifier to determine the dialect category.

[0032] The advantages of the present application are:

[0033] The novel dynamic time delay neural network for dialect recognition can dynamically adjust the weight coefficient of each convolution kernel according to different dialect audio inputs, obtain an optimal convolution kernel parameter through weighted average, extract deeper features of the dialect audio, and improve the accuracy of dialect recognition. This operation not only does not bring additional depth or width to the time delay neural network, but also greatly enhances the performance of the acoustic model of the time delay neural network. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of a novel dynamic time-delay neural network for dialect recognition according to the present invention.

[0035] Figure 2 This is a schematic diagram of a typical time-delay neural network structure.

[0036] Figure 3 A schematic diagram illustrating a specific implementation method for the dynamic adjustment of this invention.

[0037] Figure 4 A schematic diagram showing the structure of the dynamic time-delay neural network of this invention inserted into an existing time-delay neural network structure. Detailed Implementation

[0038] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail and in depth below with reference to the accompanying drawings.

[0039] In time-delay neural network acoustic modeling, the acoustic modeling effect is good. However, for different dialect audio inputs, time-delay neural networks with fixed parameters may not have a high recognition accuracy for certain dialect types. The acoustic model lacks a certain degree of generalization and flexibility, resulting in a low recognition rate for certain dialect types.

[0040] Compared to traditional time-delay neural network acoustic models, where all weight parameters are fixed once training is complete and all parameters are treated equally for any input, the acoustic model's ability to extract effective audio features is very limited. To improve the expressive power of time-delay neural network acoustic models and increase the flexibility of acoustic model feature representation, this invention improves the convolution method of time-delay neural networks in the architecture by replacing the single convolution kernel operation with multiple parallel convolution kernels. A set of convolution kernel coefficients is then obtained, with each convolution kernel having different convolution weight parameters. Different convolution kernel weight parameters are generated according to different inputs, and multiple convolution kernels with fixed parameters are fused into a dynamic kernel. This can capture deeper time-varying information in dialect audio and enhance the expressive power of the acoustic model for dialect recognition.

[0041] This invention proposes a novel dynamic time-delay neural network for dialect recognition, with the following specific structure:

[0042] The new dynamic time delay neural network for dialect recognition has the following specific structure: the convolution kernel of the existing time delay neural network is updated: first, introduce k parallel convolution kernels, calculate the attention weight parameters of each, and then weighted average fusion to obtain a convolution kernel with an optimal weight parameter; finally, replace the conventional convolution kernel of the existing time delay neural network with the convolution kernel with the optimal weight parameter to obtain an improved dynamic time delay neural network; when different dialects are input, the weight parameters in the convolution kernel with the optimal weight parameter are dynamically adjusted, the deeper feature information of the audio is extracted, the deep features extracted are compared and analyzed with the dialect acoustic template, and finally the dialect category is determined in combination with the classifier;

[0043] As shown in Figure 1 , the specific updating process is as follows:

[0044] Step one, the input audio feature x is globally averaged and pooled to obtain a global spatial feature with a length of C;

[0045] The input audio feature is an HxWxC tensor;

[0046] The global average pooling operation is to calculate the mean value of each channel in the spatial dimension, and the output of the global average pooling of the cth channel is GlobalFeature c The formula is as follows:

[0047]

[0048] Where H is the height of the feature map, W is the width of the feature map, and C is the total number of channels. X i,j,c represents the feature value of the cth channel at position (i,j), c∈C.

[0049] Step two, map the global spatial feature through two fully connected layers in sequence to k dimensions, and do softmax normalization to obtain the weight coefficients of the k convolution kernels;

[0050] The formula is as follows:

[0051]

[0052] Where, Z k is the kth output of the second fully connected layer, and Softmax(Z k ) is the weight coefficient of the kth convolution kernel.

[0053] Step three, weight average all parallel convolution kernel weights according to their weight coefficients to obtain a new convolution kernel;

[0054] k parallel convolution kernels, the weight of each convolution kernel is W conv,k ∈R N×N×Cwhere N is the size of the convolution kernel; the weight coefficients of each convolution kernel are weighted averaged to obtain a new convolution kernel W new , the formula is:

[0055]

[0056] π k is the weight coefficient of the kth convolution kernel, and the weight coefficient set corresponding to the k parallel convolution kernels is {π1, π2,..., π k}; W conv,k is the original weight of the kth convolution kernel.

[0057] Step four, input the new convolution kernel W new into the time delay neural network model for improvement, and dynamically change the analysis period to adaptively capture the deep features in the input dialect.

[0058] By dynamically changing the analysis period like adjusting the microscope focus: 'high magnification' fine observation when encountering short dialect explosion sound, 'zoom out' to grasp the overall trend when encountering long tone change.

[0059] In the structure of the ordinary time delay neural network, the output is calculated according to the following formula:

[0060] Y t =f(W T C t +b)

[0061] C t ={X t-n ,X t+n}

[0062] Where X t-n is the number of n input text information frames to the lower layer network at time t; X t+n is the number of n input context information frames to the lower layer network at time t, n is greater than or equal to 1; C t is the input feature, Y t is the output feature, W T and b are the fixed weight matrix and bias vector respectively, and f is the activation function.

[0063] The traditional method has obvious shortcomings in capturing dialect-specific features, especially in the context of time sequence features, and the weight matrix W T with fixed parameters cannot adapt to the phoneme change characteristics unique to each dialect. The improved dynamic time delay neural network, the output formula is:

[0064]

[0065]

[0066] k identical size convolution kernels are adopted, and weight coefficients π k are combined. It is worth noting that the weight coefficients π k here are not fixed; on the contrary, they vary according to the input data 'x', and are aggregated into an optimal convolution kernel through weighted averaging of these kernel coefficients. Since the convolution kernel parameters are not fixed according to the input, they tend to dynamically adjust to the current dialect category, and the features output by the acoustic model are more conducive to determining which dialect.

[0067] The improved time delay neural network model has non-fixed convolution kernel coefficient parameters, and can adaptively select an optimal feature extraction mode according to the audio content of the input dialect: in the transient stage of dialect pronunciation, short-time impulse response is strengthened, and in the steady stage, the ability of formant tracking is enhanced.

[0068] Step five, finally, the dialect category is determined by combining the classifier after comparing and analyzing the extracted deep features with the dialect acoustic template.

[0069] The traditional fixed convolution kernel adopts homogenization weight processing in time domain feature extraction, and is difficult to adapt to the time-varying characteristics of dialects. The convolution method proposed in the present application generates convolution kernel coefficients with content adaptability by real-time sensing of the time-frequency feature differences of the input audio, and realizes dynamic adjustment of parameters by using a weighted fusion strategy. This design makes each time window of the convolution kernel have feature sensitivity, and the model can adaptively select the optimal feature extraction mode according to the audio content: in the transient stage of dialect pronunciation (such as the end of throat stop in Minnan), the short-time impulse response is strengthened, and in the steady stage (such as the continuous voiced sound of Wu dialect), the formant tracking ability is enhanced. This dynamic weight adjustment mechanism effectively captures the deep time-varying information such as tone sliding and transition phonemes in dialect audio, so that the accuracy of the dialect recognition task can be effectively improved.

[0070] Compared with the existing time delay neural network model, the convolution method is improved: k parallel convolution kernels are used instead of a conventional single convolution kernel, each convolution kernel has different convolution weight parameters, and each convolution kernel coefficient is different. The parallel convolution kernel weight parameters are weighted and averaged according to the convolution kernel coefficients, and a new convolution kernel is obtained. Because the features of the input audio are different, the obtained set of convolution kernel coefficients is also not completely the same, and the convolution kernel weight parameters obtained by weighted averaging are also not completely the same. Through this way, the convolution can dynamically adjust the weight parameters according to the input, and the model can more flexibly select the most relevant feature extraction mode according to the input content, capture deeper time-varying information in dialect audio, and thus improve the accuracy of the recognition task.

[0071] As Figure 2The diagram shows a typical time-delay neural network structure. The input of each layer of the time-delay neural network is obtained through the context window of the lower layer. Therefore, it can describe the time relationship between nodes in the upper and lower layers. After each layer undergoes an affine transformation, like a DNN, there will be an activation function as the total output of that layer. This output is used to extract audio features and determine the dialect of the audio.

[0072] For the acoustic model of the time-delay neural network after training, the weight parameters of each node in each layer are fixed. The ability to judge audio may vary for different dialects. The time-delay neural network essentially uses one-dimensional convolution operations to process time series data. Each layer can contain multiple convolution kernels to extract local features.

[0073] This invention improves the way convolution kernels extract features, enabling dynamic adjustment of convolution parameters. The main improvement lies in replacing the current convolution kernel with multiple parallel convolution kernels. During model training, the parameters of these parallel kernels are not identical; finally, they are aggregated into an optimal convolution kernel through a weighted average of the different input weights.

[0074] like Figure 3 The diagram illustrates a specific implementation of a dynamically adjusted neural network. The k parameters are adjustable. When k is 1, it behaves similarly to a standard time-delay neural network, outputting after a convolutional computation. The dynamically adjusted time-delay neural network continuously adjusts the weight parameters based on the input audio, using attention weights to extract deeper information from the audio features.

[0075] During training, assuming k is 2, there are two parallel convolution weight parameters, and each convolution weight parameter has an attention weight coefficient π. k The first convolution has a weight coefficient of π1, and the second convolution has a weight coefficient of π2. π1 plus π2 equals 1. When training audio is input, the neural network continuously propagates forward and backward to update the weight parameters of the convolutions. Because there are two parallel convolution kernels, the parameters within the two kernels are not exactly the same during training. During training, assuming that the weight parameters of the first convolution are more suitable for extracting audio features of dialect A, the weight coefficient of π1 is increased and the weight coefficient of π2 is weakened. Then, the two convolution kernels are weighted and averaged. The extracted features after this operation are more conducive to identifying dialect A. Assuming that the weight parameters of the second convolution are more suitable for extracting audio features of dialect B, the weight coefficient of π2 is increased and the weight coefficient of π1 is weakened. The extracted features are more conducive to identifying dialect B.

[0076] Here will encounter a problem: the model in the training process can not also actively to determine the tendency to increase the coefficient of pi1 or pi2, the need to use attention mechanism to calculate pi k The process is shown in the dashed box in FIG. 8. When the audio feature is input, the input audio feature needs to be first subjected to a global average pooling operation to obtain a global spatial feature, and then mapped to a dimension of k through two FC layers, and subjected to a softmax normalization process to obtain k attention weights for the k convolution kernels of the layer. Figure 2

[0077] In the process of model training, the attention mechanism is also trained, so that the model can dynamically adjust the convolution kernel coefficients according to different inputs. Unlike other attention mechanisms, the attention mechanism in the present application is applied to the output feature mapping of the convolution, and the attention mechanism and the convolution run in parallel and share the input, so that the model can dynamically adjust different convolution kernel parameters according to different input audios. The neural network can continuously optimize the parameters in the convolution kernel 1 and the convolution kernel 2 through the mechanism of back propagation, and can also optimize the parameters of the FC in the attention mechanism. The process of model training is thus completed. However, this method is not limited to dynamic change according to different dialect audios, but different frames of audio in an audio can be used as the input of the time delay neural network for continuous dynamic adjustment and optimization of the convolution weight parameters. Figure 2 In the model inference stage, there are also two convolution kernels running in parallel and having fixed parameters, but due to the existence of the attention mechanism, the weight coefficients can dynamically change in the range of 0-1 according to different inputs, and then multiplied by the convolution kernel parameters to obtain a weighted average of the two convolution kernels. Thus, for different inputs, even if the two convolution kernels have fixed parameters, the convolution kernel parameters are not the same in each inference process, which improves the flexibility of the model and enhances the representation ability of the model.

[0078] In the final implementation process, as shown in FIG. 9, each convolution of the time delay neural network is replaced by this dynamically adjusted convolution kernel, and part of the nodes can also be selectively replaced according to the actual situation. In addition, k is also a parameter that can be adjusted. Here, k does not mean that the more k is, the stronger the representation ability of the model is. With the increase of k, the model parameters are likely to increase, and the model is prone to overfitting. The specific setting of how many is optimal needs to be further verified according to the actual situation.

[0079] Figure 4

[0080] ​​​The foregoing is merely a specific implementation of the present application, enabling those skilled in the art to understand or implement the present application. The present application improves the original time delay neural network, so that the network can dynamically adjust the acoustic model parameters according to different inputs, thereby improving the dialect recognition accuracy.

Claims

1. A novel dynamic temporal recurrent neural network for dialect recognition, characterized in that, The specific structure is as follows: the convolution kernel of the time delay neural network is updated: First, introduce k parallel convolution kernels, calculate the attention weight coefficients of each; K is an adjustable parameter, and each calculation node of the time delay neural network is replaced by a dynamically adjusted convolution weight; Then, a weighted average fusion is performed to obtain an optimized weight coefficient convolution kernel; the specific process is as follows: Step one, globally average pool the input audio features to obtain global spatial features with a length of C; Step two, map the global spatial features to k dimensions in turn through two fully connected layers, perform softmax normalization processing, and obtain the weight coefficients of the k convolution kernels; The formula is as follows: wherein Z k is the kth output of the second fully connected layer, and Softmax(Z k ) is the weight coefficient of the kth convolution kernel. Step three, weight average all parallel convolution kernel weights according to their weight coefficients to obtain a new convolution kernel; The weights of each convolution kernel are W conv,k ∈R N×N×C where N is the size of the convolution kernel; The new convolution kernel W is fused new The formula is: π k is a weight coefficient of the kth convolution kernel, W conv,k is the original weight of the kth convolution kernel; Finally, replace the conventional convolution kernel of the time delay neural network with the optimized weight coefficient convolution kernel to obtain an improved dynamic time delay neural network; when different dialects are input, the weight coefficients in the optimized weight coefficient convolution kernel are dynamically adjusted, audio deep features are extracted, the extracted deep features are compared with dialect acoustic templates, and finally the dialect category is determined in combination with a classifier; The improved time delay neural network model can adaptively select a feature extraction mode according to the audio content of the input dialect: in the transient stage of dialect pronunciation, the short-time impulse response is strengthened, and in the steady-state stage, the formant tracking ability is enhanced; The output formula of the improved dynamic time delay neural network model is: C t = {X t-n , X t+n} 0≤π k ≤1， Y t Y is the output deep feature, f is an activation function, W T and b are fixed weight matrix and bias vector respectively, C t is the input feature, X t-n is the number of previous information frames input to the lower layer network at time t, n; X t+n is the number of subsequent information frames input to the lower layer network at time t, n, n is greater than or equal to 1.

2. A novel dynamic temporal recurrent neural network for dialect recognition as claimed in claim 1 wherein, In the step one, the input audio features are a HxWxC tensor; The global average pooling operation is to calculate the mean value of each channel in the spatial dimension, and the output of the global average pooling of the cth channel is GlobalFeature c The formula is as follows: where H is the height of the feature map, W is the width of the feature map, X i,j,c denotes the feature value of the c-th channel at position (i,j), c e C.

Images