A training method for a dialect recognition model, a readable storage medium, and a terminal device

Abstract

The present application belongs to the field of computer technology, and in particular, relates to a training method for a dialect recognition model, a computer-readable storage medium, and a terminal device. In the method, the preset dialect recognition model determines the output dialect category corresponding to the speech sample according to the preset centroid, wherein each centroid is used to characterize the characteristics of a dialect category; the dialect recognition model adjusts the model parameters according to the output dialect category and the target dialect category corresponding to the speech sample, and continues to execute the step of determining the output dialect category corresponding to the speech sample according to the preset centroid until the preset training conditions are met to obtain a trained dialect recognition model. Since the preset centroids of various dialect categories are used in the training process, and the centroid of each dialect category can characterize the characteristics of the dialect category, a reliable basis is provided for the recognition of dialects.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and in particular, relates to a training method for a dialect recognition model, a computer-readable storage medium, and a terminal device. Background Art

[0002] With the development of science and technology, more and more smart devices have entered people's lives, and people's lives have become more and more convenient. At present, the most frequent voice interactions between people and smart devices are home appliances and car-mounted equipment. Voice interaction has liberated people's hands and limbs to a large extent, especially for the elderly and children who cannot operate complex electronic systems. Voice interaction can directly operate electronic systems according to people's spoken instructions, but many elderly people cannot speak Mandarin, and some Mandarin is mixed with local dialects. However, the voice interaction functions of current smart devices are all designed for Mandarin and cannot effectively recognize dialects. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a training method for a dialect recognition model, a computer-readable storage medium and a terminal device to solve the problem that the voice interaction functions of current smart devices are all designed for Mandarin and cannot effectively recognize dialects.

[0004] A first aspect of an embodiment of the present application provides a method for training a dialect recognition model, which may include:

[0005] The preset dialect recognition model determines the output dialect category corresponding to the speech sample according to the preset centroid, wherein each centroid is used to represent the characteristics of a dialect category;

[0006] The dialect recognition model adjusts model parameters according to the output dialect category and the target dialect category corresponding to the speech sample, and continues to execute the step of determining the output dialect category corresponding to the speech sample according to the preset centroid until the preset training conditions are met to obtain a trained dialect recognition model.

[0007] Furthermore, the dialect recognition model includes a word vector extraction module and a classification module;

[0008] The step of determining the output dialect category corresponding to the speech sample according to the preset centroid includes:

[0009] Inputting the frequency spectrum of the speech sample into the word vector extraction module to obtain the word vector of the speech sample;

[0010] The word vector of the speech sample is input into the classification module, and the classification module obtains an output dialect category corresponding to the speech sample according to the preset centroid and the word vector of the speech sample.

[0011] Furthermore, obtaining the output dialect category corresponding to the speech sample according to the preset centroid and the word vector of the speech sample includes:

[0012] Calculating the similarity between the word vector of the speech sample and the centroid of various preset dialect categories respectively;

[0013] The dialect category corresponding to when the similarity reaches a maximum value is determined as the output dialect category corresponding to the speech sample.

[0014] Furthermore, before respectively calculating the similarity between the word vector of the speech sample and the centroid of each preset dialect category, the method further includes:

[0015] For each dialect category, a speech sample set corresponding to the dialect category is obtained, wherein the speech sample set includes M speech samples, where M is a positive integer;

[0016] Calculate the word vector of each speech sample in the speech sample set respectively to obtain M word vectors;

[0017] An average value of the M word vectors is calculated, and the average value is determined as the centroid of the dialect category.

[0018] Furthermore, respectively calculating the similarity between the word vector of the speech sample and the centroid of various preset dialect categories includes:

[0019] Calculating the cosine similarity between the word vector of the speech sample and the centroid of various dialect categories respectively;

[0020] The similarities between the word vectors of the speech sample and the centroids of the various dialect categories are calculated according to the cosine similarities corresponding to the various dialect categories, as well as the preset weight coefficient and the preset bias coefficient.

[0021] Further, adjusting the model parameters according to the target dialect category and the output dialect category includes:

[0022] Calculating the training loss value of the dialect recognition model according to the similarity between the word vector of the speech sample and the centroid of various dialect categories;

[0023] The model parameters are adjusted according to the training loss value.

[0024] Furthermore, after obtaining the trained dialect recognition model, the method further includes:

[0025] Using preset test data to test the dialect recognition model, and counting the number of test successes and test failures respectively;

[0026] Calculating the recognition accuracy of the dialect recognition model according to the number of test successes and the number of test failures;

[0027] If the recognition accuracy is less than a preset accuracy threshold, continue training the dialect recognition model;

[0028] If the recognition accuracy is greater than or equal to the accuracy threshold, the test of the dialect recognition model is terminated.

[0029] A second aspect of the embodiments of the present application provides a dialect recognition method, which may include:

[0030] Obtaining the frequency spectrum of the speech to be recognized;

[0031] The frequency spectrum of the speech to be recognized is input into a trained dialect recognition model, and the dialect category corresponding to the speech to be recognized output by the dialect recognition model is obtained. The dialect recognition model is a dialect recognition model trained by any of the above-mentioned dialect recognition model training methods.

[0032] Furthermore, the dialect recognition model includes a word vector extraction module and a classification module;

[0033] The step of inputting the frequency spectrum of the speech to be recognized into a trained dialect recognition model, and obtaining the dialect category corresponding to the speech to be recognized output by the dialect recognition model, comprises:

[0034] Inputting the frequency spectrum of the speech to be recognized into the word vector extraction module to obtain the word vector of the speech to be recognized;

[0035] The word vector of the speech to be recognized is input into the classification module to obtain the dialect category corresponding to the speech to be recognized.

[0036] Furthermore, the step of inputting the word vector of the speech to be recognized into the classification module to obtain a dialect category corresponding to the speech to be recognized includes:

[0037] Calculating the similarity between the word vector of the speech to be recognized and the centroid of various preset dialect categories respectively;

[0038] The dialect category corresponding to when the similarity reaches a maximum value is determined as the dialect category corresponding to the speech to be recognized.

[0039] Furthermore, before respectively calculating the similarities between the word vectors of the speech to be recognized and the centroids of the preset dialect categories, the method further includes:

[0040] For each dialect category, a speech sample set corresponding to the dialect category is obtained, wherein the speech sample set includes M speech samples, where M is a positive integer;

[0041] Calculate the word vector of each speech sample in the speech sample set respectively to obtain M word vectors;

[0042] An average value of the M word vectors is calculated, and the average value is determined as the centroid of the dialect category.

[0043] Furthermore, the respectively calculating the similarity between the word vector of the speech to be recognized and the centroid of various preset dialect categories includes:

[0044] Calculating the cosine similarity between the word vector of the speech to be recognized and the centroid of various dialect categories respectively;

[0045] Based on the cosine similarities corresponding to the various dialect categories, as well as a preset weight coefficient and a preset bias coefficient, the similarities between the word vectors of the speech to be recognized and the centroids of the various dialect categories are calculated.

[0046] A third aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the training method of any of the above-mentioned dialect recognition models are implemented.

[0047] The fourth aspect of an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the training method of any one of the above-mentioned dialect recognition models are implemented.

[0048] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the dialect recognition model preset in the embodiments of the present application determines the output dialect category corresponding to the speech sample according to the preset centroid, wherein each centroid is used to characterize the characteristics of a dialect category; the dialect recognition model adjusts the model parameters according to the output dialect category and the target dialect category corresponding to the speech sample, and continues to perform the step of determining the output dialect category corresponding to the speech sample according to the preset centroid until the preset training conditions are met to obtain a trained dialect recognition model. Through such a training method, the dialect recognition model is continuously trained using training data. During the training process, the centroids corresponding to the preset various dialect categories are used, and the centroid of each dialect category can characterize the characteristics of the dialect category, thereby providing a reliable basis for the recognition of the dialect, and the model parameters are continuously adjusted according to the training results, so that a dialect recognition model that meets the training conditions can be finally obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0050] Figure 1 A flowchart of an embodiment of a method for training a dialect recognition model in an embodiment of the present application;

[0051] Figure 2 is a schematic flow chart of the preprocessing process of training data;

[0052] Figure 3 A schematic diagram of the data processing process of the dialect recognition model;

[0053] Figure 4 A schematic flow chart of processing the frequency spectrum of a speech sample for the dialect recognition model to obtain an output dialect category corresponding to the speech sample;

[0054] Figure 5 This is a flow chart of an embodiment of a dialect recognition method in an embodiment of the present application;

[0055] Figure 6 is a schematic flow chart of a preprocessing process of speech to be recognized;

[0056] Figure 7 A schematic flow chart of processing the frequency spectrum of the speech to be recognized for the dialect recognition model to obtain a dialect category corresponding to the speech to be recognized;

[0057] Figure 8 This is a schematic block diagram of a terminal device in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] See also Figure 1 In the embodiments of the present application, an embodiment of a method for training a dialect recognition model may include:

[0060] Step S101: A preset dialect recognition model determines an output dialect category corresponding to a speech sample according to a preset centroid.

[0061] Each centroid is used to characterize the characteristics of a dialect category.

[0062] The frequency spectrum of the speech sample can be processed by a pre-processing device (including but not limited to a terminal device with computing capability such as a computer and a server) by Figure 2 The preprocessing shown yields:

[0063] Step S201: Acquire a voice sample.

[0064] When the dialect recognition model is trained, a large amount of training data is generally required, wherein the training data may include multiple sets of training data, each set of training data includes the spectrum of a speech sample and the target dialect category corresponding to the speech sample. The specific number of the training data may be set according to actual conditions, for example, the dialect recognition model may obtain 1000 sets, 2000 sets, 3000 sets or other numbers of training data for training.

[0065] Generally, the voice sample can be obtained from a preset voice sample library, and the voice sample library can be established by collecting voice samples of a large number of users using different dialect categories, and all voice samples in the voice sample library are divided into various voice sample sets according to different dialect categories. Preferably, the voice sample set in any voice sample set can be collected from different regions using the corresponding dialect category. Exemplarily, if the voice sample set of Shaanxi dialect needs to collect 6,000 voice samples in total, it can be collected by collecting voice samples of 100 Shaanxi people, 60 sentences for each person; or by collecting voice samples of 6,000 Shaanxi people, 1 sentence for each person. And the objects of collecting voice samples come from different regions of Shaanxi, such as Baoji, Yan'an, Xi'an, Hanzhong, etc., to generate the voice sample set of Shaanxi dialect. The same operation is performed on dialects in other regions to obtain voice sample sets of each dialect. Any voice sample in the voice sample library has a corresponding dialect category, that is, the target dialect category. For example, if a certain voice sample belongs to the voice sample set of Shaanxi dialect, its target dialect category is Shaanxi dialect.

[0066] For the sake of simplicity, dialect labels can be used to represent various dialect categories. For example, if the speech sample library includes 5 dialect categories, dialect labels 0, 1, 2, 3, and 4 can be used to represent the 5 dialect categories respectively. For example, the dialect label of Minnan dialect is 0, the dialect label of Hakka dialect is 1, the dialect label of Sichuan dialect is 2, the dialect label of Shanghai dialect is 3, and the dialect label of Guizhou dialect is 4. It should be noted that the above is only an example. In actual applications, different forms of dialect labels can be set according to specific circumstances, and this embodiment will not be repeated.

[0067] Step S202: Process the speech sample to obtain a frequency spectrum of the speech sample.

[0068] Generally, the original data format of the voice sample is WAV audio format, which is the closest to lossless audio format, so its size is relatively large. In practical applications, in order to reduce the subsequent amount of calculation, the voice sample can be converted from WAV audio format to PCM audio format in advance. Preferably, considering that the voice sample may contain silent signals, these silent signals generally appear in the period before the user speaks, the period after the user speaks, and the period when the user pauses in the middle of speaking, and do not contain any useful information. Therefore, these silent signals can be removed from the voice sample to reduce interference with the final recognition result.

[0069] The speech sample is presented in the form of sound waves, and the height of the sound waves indicates the size of the sound. However, during speech recognition, the sound waves cannot well reflect the characteristics of speech, so the sound waves in the time domain must be converted into a spectrum that can better reflect the characteristics of speech. In this embodiment, the spectrum can be a Mel spectrum, which is a spectrum used to represent short-term audio information. Its principle is based on the logarithmic spectrum represented by the nonlinear Mel scale and its linear cosine transform. In a specific implementation, the speech sample can first be converted from the time domain to the frequency domain using Fourier transform, and then its logarithmic energy spectrum is convolved with a triangular filter group distributed according to the Mel scale, and finally the vector formed by the output of each filter is discretely cosine transformed to obtain its Mel spectrum.

[0070] In this embodiment, after obtaining the training sample, the dialect recognition model can Figure 3 The process shown processes the frequency spectrum of the speech sample, calculates and outputs the dialect category corresponding to the speech sample, that is, the output dialect category. The dialect recognition model includes a word vector extraction module and a classification module, the word vector extraction module is used to extract the word vector of the speech sample according to the frequency spectrum of the speech sample, and the classification module is used to determine the output dialect category corresponding to the speech sample according to the word vector of the speech sample.

[0071] Specifically, step S101 may include: Figure 4 The process shown:

[0072] Step S1011: input the frequency spectrum of the speech sample into the word vector extraction module to obtain the word vector of the speech sample.

[0073] The word vector extraction module can be any one of the existing networks, such as a convolutional neural network (CNN), a recurrent neural network (RNN), etc. Preferably, in this embodiment, a long short-term memory network (LSTM) can be used as the vector extraction module, and the spectrum of the speech sample is input into the LSTM network to obtain the word vector of the speech sample.

[0074] The LSTM network used in this embodiment can be composed of three layers of recurrent neural network and one fully connected layer. The spectrum of the speech sample is extracted through the features of each layer of recurrent neural network to obtain a two-dimensional feature map (such as Figure 3 As shown in step a), after further high purification by the fully connected layer, the two-dimensional feature map is converted into a one-dimensional vector, that is, the word vector of the speech sample (such as Figure 3 as shown in step b).

[0075] The number of features in the word vector (i.e., the length of the word vector) is determined by the number of nodes in the fully connected layer. For example, if the total number of dialect categories is 5, the number of nodes in the fully connected layer can also be set to 5, and the number of features of the word vector obtained after processing by the fully connected layer is also 5.

[0076] Preferably, in order to prevent the occurrence of overfitting, that is, the model has a very high recognition accuracy for training data, but a very low recognition accuracy for other data outside the training data, in this embodiment, after obtaining the word vector, it can be further regularized. Among them, regularization is a general term for a class of methods that introduce additional information into the model to prevent overfitting and improve the generalization performance of the model, including but not limited to L1 regularization and L2 regularization. In this embodiment, L2 regularization is preferably used to prevent overfitting.

[0077] Step S1012: input the word vector of the speech sample into the classification module, and the classification module obtains the output dialect category corresponding to the speech sample according to the preset centroid and the word vector of the speech sample.

[0078] Specifically, the similarities between the word vectors of the speech sample and the centroids of various preset dialect categories may be calculated first.

[0079] The centroids of various dialect categories can be calculated in advance. For each dialect category, a speech sample set corresponding to the dialect category is first obtained, wherein the speech sample set includes M speech samples, and M is a positive integer. Then, the word vectors of each speech sample in the speech sample set are calculated respectively to obtain M word vectors. Next, the average value of the M word vectors is calculated, and the average value is determined as the centroid of the dialect category.

[0080] Now, the calculation process of the centroid of the k-th dialect category (1≤k≤K, K is the total number of dialect categories) is taken as an example to illustrate. The centroid of the k-th dialect category can be calculated according to the following formula:

[0081]

[0082] Where m is the serial number of each speech sample in the speech sample set corresponding to the kth dialect category, 1≤m≤M, e km is the word vector of the mth speech sample in the speech sample set corresponding to the kth dialect category, c k is the centroid of the k-th dialect category.

[0083] For each dialect category, the above centroid calculation process is performed to obtain the centroid of various dialect categories. For example, if there are 5 dialect categories, 200 sentences of Minnan dialect, 200 sentences of Hakka dialect, 200 sentences of Sichuan dialect, 200 sentences of Shanghai dialect, and 200 sentences of Guizhou dialect are selected respectively. The word vectors of each sentence in the 200 Minnan dialect sentences are extracted through the LSTM network, and the word vectors of these 200 Minnan dialect sentences are averaged to calculate the centroid of Minnan dialect, and the centroid of Hakka dialect, Sichuan dialect, Shanghai dialect, and Guizhou dialect are obtained by analogy.

[0084] When respectively calculating the similarity between the word vector of the speech sample and the centroid of various preset dialect categories, the cosine similarity between the word vector of the speech sample and the centroid of various dialect categories can be calculated first, and then the similarity between the word vector of the speech sample and the centroid of various dialect categories can be calculated based on the cosine similarities corresponding to the various dialect categories, as well as the preset weight coefficient and the preset bias coefficient.

[0085] For example, the similarity between the word vector of the speech sample and the centroid of various dialect categories can be calculated according to the following formula:

[0086] S k =ω·cos(e,c k )+b;

[0087] Where e is the word vector of the speech sample, cos(e,c k ) is the cosine similarity between the word vector of the speech sample and the centroid of the kth dialect category, ω is the weight coefficient, and for different dialect categories, the weight coefficient is the same, b is the bias coefficient, and for different dialect categories, the bias coefficient is also the same, S k is the similarity between the word vector of the speech sample and the centroid of the k-th dialect category.

[0088] like Figure 3 As shown in step c, after calculating the similarity between the word vector of the speech sample and the centroids of various preset dialect categories, the similarity results will form the similarity matrix shown in the figure, and the dialect category corresponding to the maximum similarity can be determined as the output dialect category corresponding to the speech sample. For example, if the similarity between the word vector of the speech sample and the centroids of the five dialect categories of Minnan, Hakka, Sichuan, Shanghainese and Guizhou are: S 1 , S 2 , S 3 , S 4 , S 5 , where S 4The value of is the largest, then Shanghainese can be determined as the output dialect category corresponding to the speech sample.

[0089] Step S102: the dialect recognition model adjusts model parameters according to the target dialect category and the output dialect category, and continues to execute the step of determining the output dialect category corresponding to the speech sample according to the preset centroid until the preset training conditions are met to obtain a trained dialect recognition model.

[0090] Specifically, the target dialect category and the output dialect category may be compared first. If the two are inconsistent, it means that the output of the model is still inaccurate. The training loss value of the dialect recognition model may be calculated, and the model parameters of the dialect recognition model may be adjusted.

[0091] For example, the loss function tf.nn.sparse_softmax_cross_entropy_with_logits(labels=None,logits=None) provided by the TensorFlow system can be used to calculate the training loss value of the dialect recognition model. The loss function combines the softmax function and the cross entropy (cross_entropy) loss function to calculate the training loss value. Wherein, labels is the target dialect category, logits is the output dialect category y, and the output dialect category satisfies: y=w*x+b, where x is the input of the dialect recognition model, that is, the spectrum of the speech sample, w and b are both model parameters of the dialect recognition model, w is the weight parameter of the dialect recognition model, and b is the bias parameter of the dialect recognition model. The target dialect category (labels) and the output dialect category (logits) are used as the input parameters of the loss function, and the calculated output value is the training loss value of the dialect recognition model.

[0092] In this embodiment, the training loss value of the dialect recognition model can also be calculated based on the similarity between the word vector of the speech sample and the centroid of various dialect categories.

[0093] In a first specific implementation of this embodiment, the training loss value of the dialect recognition model can be calculated according to the following formula:

[0094]

[0095] Among them, S tg is the similarity between the word vector of the speech sample and the centroid of the target dialect category, L s is the training loss value of the dialect recognition model in the first specific implementation.

[0096] In a second specific implementation of this embodiment, the training loss value of the dialect recognition model may also be calculated according to the following formula:

[0097]

[0098] Where σ is the Sigmoid function, that is, σ(x) = 1 / (1+exp(-x)), L c is the training loss value of the dialect recognition model in the second specific implementation.

[0099] In a third specific implementation of this embodiment, the sum of the above two training loss values ​​may also be used as the training loss value of the dialect recognition model, that is:

[0100] L g = s + c ;

[0101] Among them, L g is the training loss value of the dialect recognition model in the third specific implementation.

[0102] After the training loss value of the dialect recognition model is calculated, the model parameters can be adjusted according to the training loss value.

[0103] In this embodiment, assuming that the model parameter of the dialect recognition model is W1, the training loss value is back-propagated to modify the model parameter W1 of the dialect recognition model to obtain the modified parameter W2. After modifying the parameters, the step of determining the output dialect category corresponding to the speech sample according to the preset centroid is continued, that is, the next training process is started. In this training process, the frequency spectrum of a new group of speech samples can be processed, and the training loss value corresponding to the speech sample can be calculated. The training loss value is back-propagated to modify the model parameter W2 of the dialect recognition model to obtain the modified parameter W3, ..., and so on. The above process is repeated continuously. Each training process can be trained for a new group of speech samples and the model parameters are modified until the preset training conditions are met. The training condition can be that the number of training times reaches a preset number threshold, and optionally, the number threshold can be 100,000 times; the training condition can also be that the dialect recognition model converges; since the number of training times may not reach the number threshold, but the dialect recognition model has converged, unnecessary work may be repeated; or the dialect recognition model cannot converge, which may lead to an infinite loop and the training process cannot be terminated. Based on the above two situations, the training condition can also be that the number of training times reaches the number threshold or the dialect recognition model converges. When the training conditions are met, a trained dialect recognition model can be obtained.

[0104] Furthermore, after obtaining the trained dialect recognition model, the following testing process can be performed:

[0105] First, the dialect recognition model is tested using preset test data, and the number of test successes and test failures are counted respectively.

[0106] The test data is similar to the training data, and also includes speech samples and target dialect categories corresponding to the speech samples. In a specific application, when establishing the speech sample library, it can be divided into two parts, one part is the training data, and the other part is the test data. For example, if each dialect category in the speech sample library has 6000 speech samples, 500 speech samples of each dialect category can be randomly selected as the test data of the dialect category, and the remaining 5500 speech samples can be used as the training data of the dialect category.

[0107] If the output dialect category of a speech sample of a test data is consistent with its target dialect category after being processed by the dialect recognition model, the test of the test data is successful. On the contrary, if the output dialect category is inconsistent with its target dialect category, the test of the test data fails. After using all the test data to test the dialect recognition model, the number of successful tests and the number of failed tests can be counted respectively.

[0108] Then, the recognition accuracy of the dialect recognition model is calculated according to the number of test successes and the number of test failures.

[0109] Specifically, the recognition accuracy of the dialect recognition model can be calculated according to the following formula:

[0110] AcRt=N1 / (N1+N2)

[0111] Among them, N1 is the number of successful tests, N2 is the number of failed tests, and AcRt is the recognition accuracy of the dialect recognition model.

[0112] If the recognition accuracy is less than the preset accuracy threshold, the dialect recognition model continues to be trained; if the recognition accuracy is greater than or equal to the accuracy threshold, the test of the dialect recognition model ends. The accuracy threshold can be set according to actual conditions, for example, it can be set to 90%, 95%, 98% or other values.

[0113] In a specific application, in order to facilitate the verification of the test results, the speech samples can be labeled according to their dialect categories, specifically, they can be labeled by dialect labels. For example, if 10 speech samples are used to test the dialect recognition model, and these 10 speech samples are all Shanghainese, then when the speech samples are input, their dialect labels are set to [3,3,3,3,3,3,3,3,3,3], and the output result of the dialect recognition model is obtained. If the output result is [0,2,1,4,3,1,2,0,2,3], it is obvious that the classification result is not good, so the model structure and parameters can be further adjusted to optimize the model until the final classification result is similar to [3,3,3,3,3,3,3,3,3,3].

[0114] In specific applications, in order to more accurately measure the test results, when testing the dialect recognition model, 500 speech samples of each dialect can be used at one time for testing, and the 500 speech samples of each dialect can be classified by the dialect recognition model. Assuming that there are 5 dialects, the final output result is between [0, 1, 2, 3, 4]. The output result is used to determine whether it meets the preset accuracy threshold. Specifically, by comparing the output result with the value of the corresponding position in the dialect label marked during input, the same number is counted, and the ratio of the same number to the total number is calculated to determine whether the ratio is greater than or equal to the accuracy threshold. Exemplarily, for 500 speech samples of Minnan dialect, the dialect label is marked as [0,0,0,...,0,0,0,...,0,0,0], if the test result is [0,0,0,....1,0,2,...,0,1,0], there are 500 results in total, the total number of the same corresponding position values ​​in the test results and the dialect label value is counted, represented by N, and then N / 500 is calculated, and the ratio is determined as the recognition accuracy of the dialect recognition model, and it is determined whether N / 500 is greater than or equal to the preset accuracy threshold. If so, the test result meets the requirements and the model building process is completed. If not, the test result does not meet the requirements, and the model parameters are adjusted and the model is trained again until the test result meets the requirements.

[0115] In summary, the dialect recognition model preset in the embodiment of the present application processes the frequency spectrum of the speech sample according to the preset centroids of various dialect categories to obtain the output dialect category corresponding to the speech sample, wherein each centroid is used to characterize the characteristics of a dialect category; the dialect recognition model adjusts the model parameters according to the output dialect category and the target dialect category corresponding to the speech sample, and continues to perform the step of determining the output dialect category corresponding to the speech sample according to the preset centroid until the preset training conditions are met to obtain a trained dialect recognition model. Through such a training method, the dialect recognition model is continuously trained using training data. During the training process, the centroids corresponding to the preset various dialect categories are used, and the centroid of each dialect category can characterize the characteristics of the dialect category, thereby providing a reliable basis for the recognition of the dialect, and the model parameters are continuously adjusted according to the training results, so that a dialect recognition model that meets the training conditions can be finally obtained.

[0116] See also Figure 5 , an embodiment of a dialect recognition method in the embodiment of the present application may include:

[0117] Step S501: Acquire the frequency spectrum of the speech to be recognized.

[0118] The frequency spectrum of the speech to be recognized can be pre-processed by a pre-processing device (including but not limited to a terminal device with computing capability such as a computer or a server) by Figure 6 The preprocessing shown yields:

[0119] Step S5011: Acquire the speech to be recognized.

[0120] The speech to be recognized can be the speech that the user collects instantly through the microphone of a terminal device such as a mobile phone or a tablet computer. In a specific usage scenario of this embodiment, when the user wants to perform dialect recognition instantly, the user can turn on the dialect recognition mode of the terminal device by clicking a specific physical button or a virtual button before collecting the speech to be recognized. In this mode, the terminal device can process each sentence of speech collected by the user according to the subsequent steps to obtain the dialect category corresponding to the speech. The specific processing process will be described in detail later.

[0121] The speech to be recognized may also be a speech originally stored in the terminal device, or a speech obtained by the terminal device from a cloud server or other terminal device through a network. In another specific usage scenario of this embodiment, when the user wants to perform dialect recognition on one or more existing speech to be recognized, the user can open the dialect recognition mode of the terminal device by clicking a specific physical button or a virtual button, and select these speech to be recognized (the order of clicking the button and selecting the speech can be interchanged, that is, the speech can be selected first, and then the dialect recognition mode of the terminal device can be opened). Then, the terminal device can process these speech to be recognized according to the subsequent steps to obtain the dialect categories corresponding to these speech. The specific processing process will be described in detail later.

[0122] Step S5012: Process the speech to be recognized to obtain a frequency spectrum of the speech to be recognized.

[0123] Generally, the original data format of the speech to be recognized is the WAV audio format. WAV is the audio format closest to lossless, so its size is relatively large. In practical applications, in order to reduce the subsequent amount of calculation, the speech to be recognized can be converted from the WAV audio format to the PCM audio format in advance. Preferably, considering that the speech to be recognized may contain silent signals, these silent signals generally appear in the period before the user speaks, the period after the user speaks, and the period when the user pauses in the middle of speaking, and do not contain any useful information. Therefore, these silent signals can be removed from the speech to be recognized to reduce interference with the final recognition result.

[0124] The speech to be recognized is presented in the form of sound waves, and the height of the sound waves indicates the size of the sound. However, during speech recognition, the sound waves cannot well reflect the characteristics of the speech, so the sound waves in the time domain must be converted into a spectrum that can better reflect the characteristics of the speech. In this embodiment, the spectrum can be a Mel spectrum, which is a spectrum used to represent short-term audio information. Its principle is based on the logarithmic spectrum represented by the nonlinear Mel scale and its linear cosine transform. In a specific implementation, the speech to be recognized can first be converted from the time domain to the frequency domain using Fourier transform, and then its logarithmic energy spectrum is convolved with a triangular filter group distributed according to the Mel scale, and finally the vector composed of the output of each filter is discretely cosine transformed to obtain its Mel spectrum.

[0125] Step S502: input the frequency spectrum of the speech to be recognized into a trained dialect recognition model, and obtain the dialect category corresponding to the speech to be recognized output by the dialect recognition model.

[0126] Among them, the dialect recognition model is a dialect recognition model trained by any of the above-mentioned dialect recognition model training methods.

[0127] In this embodiment, after acquiring the spectrum of the speech to be recognized, the dialect recognition model can process the spectrum of the speech to be recognized according to the centroids corresponding to various preset dialect categories, calculate and output the dialect category corresponding to the speech to be recognized, wherein each centroid is used to characterize the characteristics of a dialect category. The dialect recognition model includes a word vector extraction module and a classification module, wherein the word vector extraction module is used to extract the word vector of the speech to be recognized according to the spectrum of the speech to be recognized, and the classification module is used to determine the dialect category corresponding to the speech to be recognized according to the word vector of the speech to be recognized.

[0128] Specifically, step S502 may include: Figure 7 The process shown:

[0129] Step S5021: input the frequency spectrum of the speech to be recognized into the word vector extraction module to obtain the word vector of the speech to be recognized.

[0130] The word vector extraction module can be any one of the existing networks, such as a convolutional neural network (CNN), a recurrent neural network (RNN), etc. Preferably, in this embodiment, a long short-term memory network (LSTM) can be used as the vector extraction module, and the spectrum of the speech to be recognized is input into the LSTM network to obtain the word vector of the speech to be recognized.

[0131] Step S5022: input the word vector of the speech to be recognized into the classification module to obtain the dialect category corresponding to the speech to be recognized.

[0132] Specifically, the similarities between the word vectors of the speech to be recognized and the centroids of various preset dialect categories may be calculated first.

[0133] The centroids of various dialect categories can be calculated in advance. For each dialect category, a speech sample set corresponding to the dialect category is first obtained, wherein the speech sample set includes M speech samples, and M is a positive integer. Then, the word vectors of each speech sample in the speech sample set are calculated respectively to obtain M word vectors. Next, the average value of the M word vectors is calculated, and the average value is determined as the centroid of the dialect category.

[0134] For example, the centroid of the k-th dialect category can be calculated according to the following formula:

[0135]

[0136] Where m is the serial number of each speech sample in the speech sample set corresponding to the kth dialect category, 1≤m≤M, e km is the word vector of the mth speech sample in the speech sample set corresponding to the kth dialect category, c k is the centroid of the k-th dialect category.

[0137] For each dialect category, the above centroid calculation process is performed to obtain the centroid of various dialect categories. For example, if there are 5 dialect categories, 200 sentences of Minnan dialect, 200 sentences of Hakka dialect, 200 sentences of Sichuan dialect, 200 sentences of Shanghai dialect, and 200 sentences of Guizhou dialect are selected respectively. The word vectors of each sentence in the 200 Minnan dialect sentences are extracted through the LSTM network, and the word vectors of these 200 Minnan dialect sentences are averaged to calculate the centroid of Minnan dialect, and the centroid of Hakka dialect, Sichuan dialect, Shanghai dialect, and Guizhou dialect are obtained by analogy.

[0138] When respectively calculating the similarities between the word vectors of the language to be recognized and the centroids of various preset dialect categories, the cosine similarities between the word vectors of the language to be recognized and the centroids of various dialect categories can be calculated first, and then the similarities between the word vectors of the language to be recognized and the centroids of various dialect categories can be calculated based on the cosine similarities corresponding to the various dialect categories, as well as the preset weight coefficient and the preset bias coefficient.

[0139] For example, the similarity between the word vector of the speech to be recognized and the centroid of various dialect categories can be calculated according to the following formula:

[0140] S k =ω·cos(e,c k )+b;

[0141] Where e is the word vector of the speech to be recognized, cos(e,c k ) is the cosine similarity between the word vector of the speech to be recognized and the centroid of the kth dialect category, ω is the weight coefficient, and for different dialect categories, the weight coefficient is the same, b is the bias coefficient, and for different dialect categories, the bias coefficient is also the same, S k is the similarity between the word vector of the speech to be recognized and the centroid of the kth dialect category.

[0142] After calculating the similarity between the word vector of the speech to be recognized and the centroids of various preset dialect categories, the dialect category corresponding to the maximum value of the similarity can be determined as the dialect category corresponding to the speech to be recognized. For example, if the similarity between the word vector of the speech to be recognized and the centroids of five dialect categories, namely, Minnan dialect, Hakka dialect, Sichuan dialect, Shanghai dialect and Guizhou dialect, is respectively: S 1 , S 2 , S 3 , S 4 , S 5 , where S 4 The value of is the largest, then Shanghainese can be determined as the dialect category corresponding to the speech to be recognized.

[0143] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0144] Figure 8 A schematic block diagram of a terminal device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0145] like Figure 8 As shown, the terminal device 8 of this embodiment includes: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80. When the processor 80 executes the computer program 82, the steps in the training method embodiments of the above-mentioned dialect recognition model are implemented, for example Figure 1 Steps S101 to S102 are shown.

[0146] Exemplarily, the computer program 82 may be divided into one or more modules / units, which are stored in the memory 81 and executed by the processor 80 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 82 in the terminal device 8.

[0147] The terminal device 8 can be a computing device such as a mobile phone, a tablet computer, a desktop computer, a notebook, a PDA, a cloud server, etc. Those skilled in the art will understand that Figure 8 It is only an example of the terminal device 8 and does not constitute a limitation of the terminal device 8. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 8 may also include input and output devices, network access devices, buses, etc.

[0148] The processor 80 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0149] The memory 81 may be an internal storage unit of the terminal device 8, such as a hard disk or memory of the terminal device 8. The memory 81 may also be an external storage device of the terminal device 8, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 8. Further, the memory 81 may also include both an internal storage unit of the terminal device 8 and an external storage device. The memory 81 is used to store the computer program and other programs and data required by the terminal device 8. The memory 81 may also be used to temporarily store data that has been output or is to be output.

[0150] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0151] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0152] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0153] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0154] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0156] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0157] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for training a dialect recognition model, characterized in that: include: The preset dialect recognition model determines the output dialect category corresponding to the speech samples of different dialect categories in the speech sample library according to the preset centroid, wherein each centroid is used to characterize the characteristics of a dialect category, and each centroid is the average value of each word vector in the speech sample set corresponding to a dialect category, and each word vector is extracted by the LSTM network; The dialect recognition model adjusts model parameters according to the output dialect category and the target dialect category corresponding to the speech sample, and continues to perform the step of determining the output dialect category corresponding to the speech samples of different dialect categories in the speech sample library according to the preset centroid, until the preset training conditions are met, so as to obtain a trained dialect recognition model; The adjusting of the model parameters according to the target dialect category and the output dialect category includes: Calculating the training loss value of the dialect recognition model according to the similarity between the word vector of the speech sample and the centroid of various dialect categories; The training loss value of the dialect recognition model is: Among them, K is the total number of dialect categories, S k is the similarity between the word vector of the speech sample and the centroid of the k-th dialect category, S tg is the similarity between the word vector of the speech sample and the centroid of the target dialect category, L s is the training loss value of the dialect recognition model; Alternatively, the training loss value of the dialect recognition model is: Among them, σ is the Sigmoid function, L c is the training loss value of the dialect recognition model; Alternatively, the training loss value of the dialect recognition model is: L g =L s +L c ; Among them, L g is the training loss value of the dialect recognition model; The model parameters are adjusted according to the training loss value.

2. The method for training a dialect recognition model according to claim 1, characterized in that: The dialect recognition model includes a word vector extraction module and a classification module; The step of determining the output dialect category corresponding to the speech samples of different dialect categories in the speech sample library according to the preset centroid includes: Inputting the frequency spectrum of the speech sample into the word vector extraction module to obtain the word vector of the speech sample; The word vector of the speech sample is input into the classification module, and the classification module obtains an output dialect category corresponding to the speech sample according to the preset centroid and the word vector of the speech sample.

3. The method for training a dialect recognition model according to claim 2, characterized in that: The step of obtaining an output dialect category corresponding to the speech sample according to the preset centroid and the word vector of the speech sample includes: Calculating the similarity between the word vector of the speech sample and the centroid of various preset dialect categories respectively; The dialect category corresponding to when the similarity reaches a maximum value is determined as the output dialect category corresponding to the speech sample.

4. The method for training a dialect recognition model according to claim 3, characterized in that: Before respectively calculating the similarity between the word vector of the speech sample and the centroid of each preset dialect category, the method further includes: For each dialect category, a speech sample set corresponding to the dialect category is obtained, wherein the speech sample set includes M speech samples, where M is a positive integer; Calculate the word vector of each speech sample in the speech sample set respectively to obtain M word vectors; An average value of the M word vectors is calculated, and the average value is determined as the centroid of the dialect category.

5. The method for training a dialect recognition model according to claim 4, characterized in that: The respectively calculating the similarity between the word vector of the speech sample and the centroid of various preset dialect categories includes: Calculating the cosine similarity between the word vector of the speech sample and the centroid of various dialect categories respectively; The similarities between the word vectors of the speech sample and the centroids of the various dialect categories are calculated according to the cosine similarities corresponding to the various dialect categories, as well as the preset weight coefficient and the preset bias coefficient.

6. The method for training a dialect recognition model according to any one of claims 1 to 5, characterized in that: After obtaining the trained dialect recognition model, it also includes: Using preset test data to test the dialect recognition model, and counting the number of test successes and test failures respectively; Calculating the recognition accuracy of the dialect recognition model according to the number of test successes and the number of test failures; If the recognition accuracy is less than a preset accuracy threshold, continue training the dialect recognition model; If the recognition accuracy is greater than or equal to the accuracy threshold, the test of the dialect recognition model is terminated.

7. A dialect recognition method, characterized in that: include: Obtaining the frequency spectrum of the speech to be recognized; The frequency spectrum of the speech to be recognized is input into a trained dialect recognition model, and the dialect category corresponding to the speech to be recognized output by the dialect recognition model is obtained, wherein the dialect recognition model is the dialect recognition model described in any one of claims 1 to 6.

8. The dialect recognition method according to claim 7, characterized in that: The dialect recognition model includes a word vector extraction module and a classification module; The step of inputting the frequency spectrum of the speech to be recognized into a trained dialect recognition model, and obtaining the dialect category corresponding to the speech to be recognized output by the dialect recognition model, comprises: Inputting the frequency spectrum of the speech to be recognized into the word vector extraction module to obtain the word vector of the speech to be recognized; The word vector of the speech to be recognized is input into the classification module to obtain the dialect category corresponding to the speech to be recognized.

9. The dialect recognition method according to claim 8, characterized in that: The step of inputting the word vector of the speech to be recognized into the classification module to obtain a dialect category corresponding to the speech to be recognized includes: Calculating the similarity between the word vector of the speech to be recognized and the centroid of various preset dialect categories respectively; The dialect category corresponding to when the similarity reaches a maximum value is determined as the dialect category corresponding to the speech to be recognized.

10. The dialect recognition method according to claim 9, characterized in that: Before respectively calculating the similarity between the word vector of the speech to be recognized and the centroid of each preset dialect category, the method further includes: For each dialect category, a speech sample set corresponding to the dialect category is obtained, wherein the speech sample set includes M speech samples, where M is a positive integer; Calculate the word vector of each speech sample in the speech sample set respectively to obtain M word vectors; An average value of the M word vectors is calculated, and the average value is determined as the centroid of the dialect category.

11. The dialect recognition method according to claim 9, characterized in that: The respectively calculating the similarity between the word vector of the speech to be recognized and the centroid of various preset dialect categories includes: Calculating the cosine similarity between the word vector of the speech to be recognized and the centroid of various dialect categories respectively; Based on the cosine similarities corresponding to the various dialect categories, as well as a preset weight coefficient and a preset bias coefficient, the similarities between the word vectors of the speech to be recognized and the centroids of the various dialect categories are calculated.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

13. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images