Cross-dialect speech recognition method, system and equipment based on information decoupling

Abstract

The invention discloses a cross-dialect speech recognition method, system and device based on information decoupling, and belongs to the field of speech recognition. According to the cross-dialect speech recognition method based on information decoupling, speaker information is decoupled from dialect pronunciation embedding by constructing an information acquisition model, a dialect decoupling model, a dialect splicing model and a speech recognition model, and dialect pronunciation embedding irrelevant to a speaker is obtained, so that the speaker information can be decoupled from the dialect pronunciation embedding. Semantic information and dialect information are decoupled, so that accent embedding is prevented from being interwoven with other voice attributes, and more accurate dialect features can be obtained; meanwhile, in the dialect adaptation stage, the extracted dialect pronunciation embedding help model is used for learning dialect voice, so that the generalization ability of multiple dialects can be effectively improved, and the recognition accuracy is improved. Furthermore, accent embedding can be effectively extracted, information of any specific speaker is not needed, and therefore the performance of a cross-dialect speech recognition model can be effectively improved.

Description

Technical Field

[0001] The invention relates to a cross-dialect speech recognition method, system and device based on information decoupling, and belongs to the field of speech recognition. Background Art

[0002] A Chinese patent application (publication number CN117894295A) proposes a speech recognition method, device, electronic device and storage medium, the method comprising: performing feature extraction processing of a set level on a first speech feature of a speech to be recognized, to obtain a speech feature of the set level; the set level includes: frame level and / or segment level; based on the speech feature of the set level, predicting the attribute information of the speech to be recognized, the attribute information includes: at least one of dialect type information and speech distance information; based on the first speech feature, the speech feature of the set level and the attribute information of the speech to be recognized, predicting the speech recognition result of the speech to be recognized. By adopting the technical solution of the present application, the dialect type information and speech distance information of the speech can be predicted in advance, and speech recognition can be performed directly based on the predicted dialect type information and / or speech distance information, without manually switching the dialect package and the recognition module of the speech distance, thereby improving the convenience of speech recognition.

[0003] The above scheme does not take into account the impact of accent (pronunciation) on the speech recognition system, and the accent information embedding is usually intertwined with other speech attributes, which affects the performance of the speech recognition system and leads to low recognition accuracy of the speech recognition system. In particular, it is difficult for the existing speech recognition system to accurately recognize the accented speech of non-native speakers.

[0004] The information disclosed in this Background Art is only for understanding the background of the inventive concept and therefore it may include information that does not constitute the prior art. Summary of the invention

[0005] In response to the above problem or one of the above problems, an object of the present invention is to provide a cross-dialect speech recognition method, system and device based on information decoupling, which decouples speaker information from dialect pronunciation embedding and obtains speaker-independent dialect pronunciation embedding, so that speaker information, semantic information and dialect information can be decoupled, thereby effectively avoiding the interweaving of accent embedding and other speech attributes, thereby obtaining more accurate dialect features; at the same time, in the dialect adaptation stage, the extracted dialect pronunciation embedding is used to help the model learn dialect speech, thereby effectively improving the generalization ability of multiple dialects and improving the recognition accuracy.

[0006] In response to the above problem or one of the above problems, the second object of the present invention is to provide a cross-dialect speech recognition method, system and device based on information decoupling, which can effectively extract accent embedding without the need for any specific speaker information, thereby effectively improving the performance of the cross-dialect speech recognition model and having a lower word error rate.

[0007] To achieve one of the above purposes, the first technical solution of the present invention is:

[0008] A cross-dialect speech recognition method based on information decoupling comprises the following steps:

[0009] Step 1: Obtain speaker information through a pre-built information collection model;

[0010] Step 2: Use the pre-built dialect decoupling model to decouple the speaker information from the dialect pronunciation embedding to obtain the speaker-independent dialect pronunciation embedding;

[0011] Step 3: Use a pre-built dialect concatenation model to concatenate the dialect pronunciation embedding with the dialect sample to obtain dialect pronunciation feature data;

[0012] Step 4: Based on the pre-built speech recognition model, dialect adaptation is performed on the dialect pronunciation feature data to complete cross-dialect speech recognition based on information decoupling.

[0013] After continuous exploration and experimentation, the present invention decouples speaker information from dialect pronunciation embedding by constructing information collection models, dialect decoupling models, dialect splicing models, and speech recognition models, and obtains speaker-independent dialect pronunciation embedding, thereby decoupling speaker information, semantic information, and dialect information, thereby effectively avoiding the interweaving of accent embedding with other speech attributes, and thus obtaining more accurate dialect features; at the same time, in the dialect adaptation stage, the extracted dialect pronunciation embedding is used to help the model learn dialect speech. Compared with traditional dialect recognition methods, the features of dialect pronunciation extracted by the present invention can effectively improve the generalization ability of multiple dialects and improve recognition accuracy.

[0014] Furthermore, the present invention can effectively extract accent (pronunciation) embeddings without any specific speaker information, thereby effectively improving the performance of cross-dialect speech recognition models, and is particularly suitable for accented speech recognition of non-native speakers.

[0015] Furthermore, experiments using the present invention on the CommonVoice dataset show that the method proposed by the present invention has a lower word error rate than the original dialect speech recognition.

[0016] As the preferred technical measures:

[0017] Step 2: Use the pre-built dialect decoupling model to decouple the speaker information from the dialect pronunciation embedding. The method for obtaining the speaker-independent dialect pronunciation embedding is as follows:

[0018] Using the Mel filter, the speaker information is converted into a discrete signal, and the speech semantic information is retained to obtain the speech spectrum signal;

[0019] Through the speaker encoder created in advance, the neural network algorithm is used to extract feature information from the speech spectrum signal and generate a speaker feature vector;

[0020] Based on the gradient reversal layer, the gradient of the speaker feature vector is reversed to obtain the speaker reverse information;

[0021] Through the dialect encoder created in advance, the adversarial training algorithm is used to cluster the speech spectrum signals, and the dialect pronunciation embedding is obtained based on the speaker's reverse information.

[0022] As the preferred technical measures:

[0023] The speaker encoder is created as follows:

[0024] Construct a linear layer and a softmax layer to calculate each speaker embedding s i The probability distribution of

[0025] According to the probability distribution, establish the speaker classification loss function;

[0026]

[0027] Where logP(l i ∣s i ) is the speaker embedding s i Belongs to the speaker i probability;

[0028] Based on the speaker classification loss function, set the voiceprint model ECAPA-TDNN;

[0029] Use the voice dataset Voxceleb 2 to pre-train the voiceprint model ECAPA-TDNN and obtain weight information;

[0030] According to the weight information, the voiceprint model ECAPA-TDNN is initialized to obtain the speaker encoder.

[0031] As the preferred technical measures:

[0032] The dialect encoder is created as follows:

[0033] Use the GST model to create the architecture of the dialect encoder and set the number of basis vectors;

[0034] Perform k-means clustering on the speech spectrum signal, set the number of clusters, and obtain the cluster center;

[0035] Calculate the clustering value of each cluster center and get the clustering array;

[0036] Using the clustering array, each basis vector is initialized and the dialect encoder is trained to obtain the dialect feature vector;

[0037] The dialect feature vector is embedded as the dialect pronunciation to complete the creation of the dialect encoder.

[0038] As the preferred technical measures:

[0039] The training goal of the speaker encoder is to make the speaker encoding discriminable by the speaker classifier, and use the speaker classifier loss to optimize it;

[0040] Or / and, the dialect encoder training methods are MTL, xvectoremb, Ours-a, Ours-b, and Ours-a&b.

[0041] As the preferred technical measures:

[0042] The method of clustering speech spectrum signals using the adversarial training algorithm through the previously created dialect encoder and obtaining dialect pronunciation embedding based on the speaker's reverse information is as follows:

[0043] Based on dialect pronunciation variable a i By speaker y i The probability of being extracted from the speech spectrum signal is used to establish an adversarial loss function;

[0044] Minimize the adversarial loss function to remove dialect pronunciation variables in the speech spectrum signal;

[0045] Based on speaker embedding and dialect pronunciation variables a i , establish an orthogonal loss function to supervise the dialect pronunciation embedding and the speaker embedding to be orthogonal to each other, so that the dialect pronunciation embedding and the speaker embedding are independent of each other;

[0046] Establishing the overall optimization objective based on the adversarial loss function, the orthogonal loss function and the speaker classification loss function;

[0047] Using the adversarial training algorithm, and based on the overall optimization goal and speaker reverse information, the speech spectrum signal is clustered and trained to obtain the dialect pronunciation embedding.

[0048] As the preferred technical measures:

[0049] The calculation formula of the adversarial loss function is as follows:

[0050]

[0051] Where logP(y i ∣a i ) is the dialect pronunciation embedded in a i By speaker y i The probability of being extracted from the speech sample;

[0052] Or / and, the calculation formula of the orthogonal loss function is as follows:

[0053]

[0054] in is the Frobenius norm;

[0055] Or / and, the total optimization goal L dm The expression is as follows:

[0056] L dm =L sc +αL ort +βL adv

[0057] Among them, L sc is the speaker classification loss function; α and β are correlation coefficients; L ort is the orthogonal loss function; L adv is the adversarial loss function.

[0058] To achieve one of the above purposes, the second technical solution of the present invention is:

[0059] A cross-dialect speech recognition method based on information decoupling, comprising the following contents:

[0060] First, obtain the speaker information;

[0061] Then, the speaker information is decoupled from the dialect pronunciation embedding to obtain the speaker-independent dialect pronunciation embedding;

[0062] Then, the dialect pronunciation is embedded and integrated into the encoder, or the dialect pronunciation is embedded and integrated into the decoder, and dialect adaptation is performed to obtain dialect pronunciation feature data;

[0063] Finally, based on the dialect pronunciation feature data, the dialect speech to be recognized is recognized to obtain speech recognition information, thus completing cross-dialect speech recognition based on information decoupling.

[0064] After continuous exploration and experimentation, the present invention decouples speaker information from dialect pronunciation embedding, obtains speaker-independent dialect pronunciation embedding, and thus decouples speaker information, semantic information and dialect information, thereby effectively avoiding the interweaving of accent embedding with other speech attributes, thereby obtaining more accurate dialect features; at the same time, in the dialect adaptation stage, the extracted dialect pronunciation embedding is used to help the model learn dialect speech. Compared with traditional dialect recognition methods, the dialect pronunciation features extracted by the present invention can effectively improve the generalization ability of multiple dialects and improve the recognition accuracy.

[0065] Furthermore, the present invention can effectively extract accent embeddings without requiring any specific speaker information, thereby effectively improving the performance of cross-dialect speech recognition models.

[0066] To achieve one of the above purposes, the third technical solution of the present invention is:

[0067] A cross-dialect speech recognition system based on information decoupling, comprising an information collection module, a dialect decoupling module, a dialect splicing module and a speech recognition module;

[0068] An information collection module, used to obtain speaker information;

[0069] The dialect decoupling module is used to decouple the speaker information from the dialect pronunciation embedding to obtain the speaker-independent dialect pronunciation embedding;

[0070] A dialect concatenation module is used to concatenate dialect pronunciation embeddings with dialect samples to obtain dialect pronunciation feature data;

[0071] The speech recognition module is used to perform dialect adaptation on dialect pronunciation feature data and complete cross-dialect speech recognition based on information decoupling.

[0072] After continuous exploration and experimentation, the present invention decouples speaker information from dialect pronunciation embedding by setting an information collection module, a dialect decoupling module, a dialect splicing module, and a speech recognition module, and obtains speaker-independent dialect pronunciation embedding, thereby decoupling speaker information, semantic information, and dialect information, thereby effectively avoiding the interweaving of accent embedding with other speech attributes, and thus obtaining more accurate dialect features; at the same time, in the dialect adaptation stage, the extracted dialect pronunciation embedding is used to help the module learn dialect speech. Compared with the traditional dialect recognition method, the features of dialect pronunciation extracted by the present invention can effectively improve the generalization ability of multiple dialects and improve the recognition accuracy.

[0073] To achieve one of the above purposes, the fourth technical solution of the present invention is:

[0074] An electronic device comprising:

[0075] one or more processors;

[0076] A storage device for storing one or more programs;

[0077] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned cross-dialect speech recognition method based on information decoupling.

[0078] Compared with the prior art solutions, the present invention has the following beneficial effects:

[0079] After continuous exploration and experimentation, the present invention decouples speaker information from dialect pronunciation embedding and obtains speaker-independent dialect pronunciation embedding, thereby decoupling speaker information, semantic information and dialect information, thereby effectively avoiding the interweaving of accent embedding with other speech attributes, and thus obtaining more accurate dialect features; at the same time, in the dialect adaptation stage, the extracted dialect pronunciation embedding is used to help the model learn dialect speech.

[0080] Compared with traditional dialect recognition methods, the features of dialect pronunciation extracted by the present invention can effectively improve the generalization ability of multiple dialects and improve the recognition accuracy.

[0081] Furthermore, the present invention can effectively extract accent embeddings without requiring any specific speaker information, thereby effectively improving the performance of cross-dialect speech recognition models.

[0082] Furthermore, experiments using the present invention on the CommonVoice dataset show that the method proposed by the present invention has a lower word error rate than the original dialect speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 A schematic diagram of the cross-dialect speech recognition method of the present invention;

[0084] Figure 2 It is a schematic diagram of the overall architecture of the cross-dialect speech recognition method of the present invention;

[0085] Figure 3 It is a schematic diagram of a process of dialect adaptation performed by the present invention; DETAILED DESCRIPTION

[0086] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0087] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the following detailed description of the present invention. Those skilled in the art can fully understand the present invention without the description of these details.

[0088] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention pertains. The terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "or / and" as used herein includes any and all combinations of one or more of the associated listed items.

[0089] like Figure 1 As shown, the first specific embodiment of the cross-dialect speech recognition method based on information decoupling of the present invention is:

[0090] A cross-dialect speech recognition method based on information decoupling comprises the following steps:

[0091] Step 1: Obtain speaker information through a pre-built information collection model;

[0092] Step 2: Use the pre-built dialect decoupling model to decouple the speaker information from the dialect pronunciation embedding to obtain the speaker-independent dialect pronunciation embedding;

[0093] Step 3: Use a pre-built dialect concatenation model to concatenate the dialect pronunciation embedding with the dialect sample to obtain dialect pronunciation feature data;

[0094] Step 4: Based on the pre-built speech recognition model, dialect adaptation is performed on the dialect pronunciation feature data to complete cross-dialect speech recognition based on information decoupling.

[0095] A second specific embodiment of the cross-dialect speech recognition method based on information decoupling of the present invention:

[0096] A cross-dialect speech recognition method based on information decoupling, comprising the following contents:

[0097] First, obtain the speaker information;

[0098] Then, the speaker information is decoupled from the dialect pronunciation embedding to obtain the speaker-independent dialect pronunciation embedding;

[0099] Then, the dialect pronunciation is embedded and integrated into the encoder, or the dialect pronunciation is embedded and integrated into the decoder, and dialect adaptation is performed to obtain dialect pronunciation feature data;

[0100] Finally, based on the dialect pronunciation feature data, the dialect speech to be recognized is recognized to obtain speech recognition information, thus completing cross-dialect speech recognition based on information decoupling.

[0101] A third specific embodiment of the cross-dialect speech recognition method based on information decoupling of the present invention:

[0102] A cross-dialect speech recognition method based on feature fusion mainly includes two stages. The first stage is dialect feature extraction and the second stage is dialect adaptation.

[0103] The first stage is dialect feature extraction. In this stage, a dialect encoder needs to be trained first, and its output is the extracted dialect features. In order to achieve information decoupled dialect feature extraction, this embodiment introduces an adversarial learning method to train the dialect encoder. In adversarial training, this embodiment mainly uses speaker features for adversarial learning. To this end, this embodiment uses a linear layer and a softmax layer to calculate each speaker embedding s i The probability distribution of , the loss function of speaker classification can be defined as:

[0104]

[0105] The goal of using speaker information for adversarial training is to make the dialect pronunciation embedding obtained by the dialect encoder not contain speaker information. To achieve this goal, this embodiment uses a gradient reversal layer (GRL) added between the dialect classifier and the speaker classifier. The function of the gradient reversal layer is to reverse the gradient returned by the speaker classifier and then return it to the dialect encoding module. With the gradient reversal layer, this embodiment can reversely optimize the dialect encoder by minimizing the loss function of the speaker classifier, thereby removing the speaker information in the dialect encoding. Therefore, this adversarial loss can be defined as:

[0106]

[0107] In order to further ensure that the dialect pronunciation embedding and the speaker embedding are independent of each other, this embodiment introduces an orthogonal loss function to supervise that the dialect pronunciation embedding and the speaker embedding are orthogonal to each other. The orthogonal loss can be defined as:

[0108]

[0109] In summary, the overall optimization goal of the speaker information decoupling module can be defined as:

[0110] L dm =L sc +αL ort +βL adv

[0111] By optimizing the objective function, the present embodiment can obtain a speaker-independent dialect encoder to achieve dialect. Through the dialect encoder, the present embodiment can obtain the dialect features of each dialect sample.

[0112] In the dialect adaptation stage, this embodiment can realize the dialect adaptation function by splicing the dialect information extracted in the previous stage with the dialect sample and inputting it into the speech recognition model.

[0113] like Figure 2 As shown, the fourth specific embodiment of the cross-dialect speech recognition method based on information decoupling of the present invention is:

[0114] A cross-dialect speech recognition method based on information decoupling mainly includes two stages: the first stage is dialect feature extraction, and the second stage is dialect adaptation. Dialect features are dialect pronunciation features, such as the pronunciation differences between Mandarin and Northeastern dialect.

[0115] The purpose of the speaker decoupling module is to decouple the speaker information from the dialect pronunciation embedding, so as to obtain speaker-independent dialect pronunciation embedding. The speaker decoupling module mainly consists of two encoders, the dialect encoder and the speaker encoder. Both the speaker encoder and the dialect encoder take the reference Mel spectrum as input. The Mel spectrum is obtained by mathematical transformation through the Mel filter, which converts the continuous signal into a discrete signal and retains the speech semantic information.

[0116] The dialect encoder outputs the dialect pronunciation embedding, which needs to be used as a conditional input to the speech synthesis module. At the same time, it needs to be independent of the speaker embedding output by the speaker encoder. The dialect pronunciation embedding is a vector containing dialect pronunciation features, and the accent information embedding is a vector containing speaker information.

[0117] The speaker encoder uses the structure of xvector, and uses the pre-trained xvector model to initialize it during training. Initialization is assignment, and the pre-trained xvector model is used here as the optimization starting point of the speaker encoder in the present invention.

[0118] The training goal of the speaker encoder is to make the speaker encoding discriminated by the speaker classifier. This embodiment uses the speaker classifier loss to optimize it. The structure of xvector is an encoder structure, which is used to further extract the required features from the mel spectrum.

[0119] The speaker classifier loss is similar to the dialect classification loss. This example uses a linear layer and a softmax layer to calculate each speaker embedding s i The probability distribution of , the loss function of speaker classification can be defined as

[0120]

[0121] Where logP(l i ∣s i ) is the speaker embedding s i Belongs to the speaker i The probability of the speaker embedding s i is a vector or a partial vector. For example, the speaker embedding here can be equivalent to the speaker information feature vector.

[0122] This embodiment is a speaker identity separation accent modeling method that extracts accent information embedding without any specific speaker information and uses it to improve the performance of multi-accent speech recognition systems. Experiments conducted on the CommonVoice dataset show that the method proposed in this embodiment reduces the word error rate (WER) by 15% compared to the baseline method.

[0123] In this embodiment, the adversarial training in the speaker decoupling module includes the following contents:

[0124] The goal of adversarial training is to make the dialect pronunciation embedding obtained by the dialect encoder contain no speaker information. To achieve this goal, this embodiment uses a gradient reversal layer (GRL) added between the dialect classifier and the speaker classifier. The function of the gradient reversal layer is to reverse the gradient returned by the speaker classifier and then return it to the dialect encoding module. With the gradient reversal layer, this embodiment can reversely optimize the dialect encoder by minimizing the loss function of the speaker classifier, thereby removing the speaker information in the dialect encoding. Therefore, this adversarial loss can be defined as:

[0125]

[0126] Where logP(y i ∣a i ) is the dialect pronunciation embedded in a i By speaker y i The probability of extracting from the speech sample.

[0127] In order to further ensure that the dialect pronunciation embedding and the speaker embedding are independent of each other, this embodiment introduces an orthogonal loss function to supervise that the dialect pronunciation embedding and the speaker embedding are orthogonal to each other. The orthogonal loss can be defined as:

[0128]

[0129] in is the Frobenius norm.

[0130] In summary, the overall optimization goal of the speaker information decoupling module can be defined as:

[0131] L dm =L sc +αL ort +βL adv

[0132] In the dialect adaptation stage, the dialect information extracted in the previous stage is spliced ​​with the dialect sample and input into the speech recognition model (the specific model structure can be referred to the invention content part) to realize the dialect adaptation function.

[0133] The dialect encoder trained by the information decoupling method can be used to model dialect information and extract dialect pronunciation embedding, which is used to help the benchmark speech recognition model adapt to the dialect speech. The present invention also proposes two target dialect adaptation methods based on dialect pronunciation embedding. The first method integrates the dialect pronunciation embedding into the encoder; the second method integrates the dialect pronunciation embedding into the decoder. The specific architecture is shown in Figure 3 .

[0134] A specific embodiment of applying the method of the present invention to speech recognition:

[0135] This embodiment uses the Common Voice English dataset, and selects six national dialects with relatively sufficient resources, namely the United States, the United Kingdom, India, Australia, New Zealand, and Scotland. The dataset division and audio feature extraction follow the method of the present invention.

[0136] The auxiliary speech synthesis model used in this embodiment uses the architecture of Tacotron 2. The dimensions of dialect pronunciation embedding and speaker embedding accepted by the Tacotron 2 encoder are both 128.

[0137] The speaker encoder uses the current model ECAPA-TDNN with the best speaker recognition effect. This embodiment uses the SpeechBrain toolkit to pre-train it with Voxceleb 2, and then initializes the speaker encoder with the weights of the pre-trained model.

[0138] The dialect encoder in this embodiment uses the GST architecture, where the number of basis vectors is set to 10. This embodiment performs k-means clustering on the dialect pronunciation embedding extracted from the pre-trained model, sets the number of clusters to 10, and finally uses the value of each cluster center to initialize these vectors. K-means clustering is to use the clustering results as the initialization position of the dialect feature vector when GST sets 10 basis vectors.

[0139] The experimental results are shown in Table 1. The first four groups are the existing mainstream multi-dialect speech recognition methods, among which MTL multi-task learning achieves the best results. The adversarial learning method DAT has a higher word error rate (WER) than MTL. This is because the distance between the distribution classes of each dialect is large, and it is difficult to learn dialect-invariant features using domain adversarial training, so its effect lags behind multi-task learning MTL.

[0140] By comparing the two groups of experiments of Ours-a and xvector proposed in the present invention, it can be seen that Ours-a has a lower WER than xvector in all dialects, indicating that the decoupled dialect pronunciation embedding can model the dialect characteristics more accurately and significantly improve the adaptability of the ASR model to the dialect. In addition, it also reduces the influence of different speaker information on the dialect pronunciation embedding, making the dialect pronunciation embedding more generalizable. Among all the methods, Ours-a&b of the present invention achieved the best effect. Compared with the baseline experiment Joint, it achieved a 15.8\% decrease in WER, verifying the effectiveness of the proposed method.

[0141] Table 1 Test results

[0142] method Australia US UKEN India New Zealand Scotland SC Avg Joint 19.87 22.67 19.81 25.29 15.81 18.47 20.32 DAT 18.32 18.23 17.58 24.77 14.68 18.14 18.62 MTL 17.86 18.74 16.79 22.69 14.56 16.82 17.91 xvectoremb 18.64 18.91 17.21 23.09 14.42 17.96 18.37 Ours-a 18.64 18.12 16.24 22.12 13.76 16.23 17.52 Ours-b 17.86 18.54 16.53 22.52 13.95 16.66 17.68 Ours-a&b 17.27 17.63 15.86 22.75 13.23 15.77 17.09

[0143] An embodiment of a device applying the method of the present invention:

[0144] An electronic device comprising:

[0145] one or more processors;

[0146] A storage device for storing one or more programs;

[0147] When the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned cross-dialect speech recognition method based on information decoupling.

[0148] A computer medium embodiment using the method of the present invention:

[0149] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned cross-dialect speech recognition method based on information decoupling.

[0150] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, and computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0151] The present application is described by flowcharts or / and block diagrams of the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each process or / and block in the flowchart or / and block diagram and the combination of the processes or / and blocks in the flowchart or / and block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart or / and block diagram. Figure 1 Process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0152] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 Process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0153] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 Process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0154] The model in this application is an object that objectively describes the morphological structure with the help of physical or virtual representation. The object is not equal to the object and is not limited to physical and virtual. It can be a data processing function, software program, processing mode, usage method, operation method, workflow, application process, electronic hardware, circuit module, processing system, system imitation or simulation object.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field can still modify or replace the specific implementation methods of the present invention with equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A cross-dialect speech recognition method based on information decoupling, characterized by: The following steps are involved: Step 1: Obtain speaker information through a pre-built information collection model; Step 2: Use the pre-built dialect decoupling model to decouple the speaker information from the dialect pronunciation embedding to obtain the speaker-independent dialect pronunciation embedding; Step 3: Use a pre-built dialect concatenation model to concatenate the dialect pronunciation embedding with the dialect sample to obtain dialect pronunciation feature data; Step 4: Based on the pre-built speech recognition model, dialect adaptation is performed on the dialect pronunciation feature data to complete cross-dialect speech recognition based on information decoupling.

2. The method for cross-dialect speech recognition based on information decoupling as claimed in claim 1, characterized in that: Step 2: Use the pre-built dialect decoupling model to decouple the speaker information from the dialect pronunciation embedding. The method for obtaining the speaker-independent dialect pronunciation embedding is as follows: Using the Mel filter, the speaker information is converted into a discrete signal, and the speech semantic information is retained to obtain the speech spectrum signal; Through the speaker encoder created in advance, the neural network algorithm is used to extract feature information from the speech spectrum signal and generate a speaker feature vector; Based on the gradient reversal layer, the gradient of the speaker feature vector is reversed to obtain the speaker reverse information; Through the dialect encoder created in advance, the adversarial training algorithm is used to cluster the speech spectrum signals, and the dialect pronunciation embedding is obtained based on the speaker's reverse information.

3. The method for cross-dialect speech recognition based on information decoupling as claimed in claim 3, characterized in that: The speaker encoder is created as follows: Construct a linear layer and a softmax layer to calculate each speaker embedding s i The probability distribution of According to the probability distribution, establish the speaker classification loss function; Where logP(l i ∣s i ) is the speaker embedding s i Belongs to the speaker i probability; Based on the speaker classification loss function, set the voiceprint model ECAPA-TDNN; Use the voice dataset Voxceleb 2 to pre-train the voiceprint model ECAPA-TDNN and obtain weight information; According to the weight information, the voiceprint model ECAPA-TDNN is initialized to obtain the speaker encoder.

4. The method for cross-dialect speech recognition based on information decoupling as claimed in claim 3, characterized in that: The dialect encoder is created as follows: Use the GST model to create the architecture of the dialect encoder and set the number of basis vectors; Perform k-means clustering on the speech spectrum signal, set the number of clusters, and obtain the cluster center; Calculate the clustering value of each cluster center and get the clustering array; Using the clustering array, each basis vector is initialized and the dialect encoder is trained to obtain the dialect feature vector; The dialect feature vector is embedded as the dialect pronunciation to complete the creation of the dialect encoder.

5. The method for cross-dialect speech recognition based on information decoupling as claimed in claim 4, characterized in that: The training goal of the speaker encoder is to make the speaker encoding discriminable by the speaker classifier, and use the speaker classifier loss to optimize it; Or / and, the dialect encoder training methods are MTL, xvectoremb, Ours-a, Ours-b, and Ours-a&b.

6. The method for cross-dialect speech recognition based on information decoupling as claimed in claim 4, characterized in that: The method of clustering speech spectrum signals using the adversarial training algorithm through the previously created dialect encoder and obtaining dialect pronunciation embedding based on the speaker's reverse information is as follows: Based on dialect pronunciation variable a i By speaker y i The probability of being extracted from the speech spectrum signal is used to establish an adversarial loss function; Minimize the adversarial loss function to remove dialect pronunciation variables in the speech spectrum signal; Based on speaker embedding and dialect pronunciation variables a i , establish an orthogonal loss function to supervise the dialect pronunciation embedding and the speaker embedding to be orthogonal to each other, so that the dialect pronunciation embedding and the speaker embedding are independent of each other; Establishing the overall optimization objective based on the adversarial loss function, the orthogonal loss function and the speaker classification loss function; Using the adversarial training algorithm, and based on the overall optimization goal and speaker reverse information, the speech spectrum signal is clustered and trained to obtain the dialect pronunciation embedding.

7. The method for cross-dialect speech recognition based on information decoupling as claimed in claim 5, characterized in that: The calculation formula of the adversarial loss function is as follows: Where logP(y i ∣a i ) is the dialect pronunciation embedded in a i By speaker y i The probability of being extracted from the speech sample; Or / and, the calculation formula of the orthogonal loss function is as follows: in is the Frobenius norm; Or / and, the total optimization goal L dm The expression is as follows: L dm =L sc +αL ort +βL adv Among them, L sc is the speaker classification loss function; α and β are correlation coefficients; L ort is the orthogonal loss function; L adv is the adversarial loss function.

8. A cross-dialect speech recognition method based on information decoupling, characterized by: Includes the following: First, obtain the speaker information; Then, the speaker information is decoupled from the dialect pronunciation embedding to obtain the speaker-independent dialect pronunciation embedding; Then, the dialect pronunciation is embedded and integrated into the encoder, or the dialect pronunciation is embedded and integrated into the decoder, and dialect adaptation is performed to obtain dialect pronunciation feature data; Finally, based on the dialect pronunciation feature data, the dialect speech to be recognized is recognized to obtain speech recognition information, thus completing cross-dialect speech recognition based on information decoupling.

9. A cross-dialect speech recognition system based on information decoupling, characterized by: It includes information collection module, dialect decoupling module, dialect splicing module and speech recognition module; An information collection module, used to obtain speaker information; The dialect decoupling module is used to decouple the speaker information from the dialect pronunciation embedding to obtain the speaker-independent dialect pronunciation embedding; A dialect concatenation module is used to concatenate dialect pronunciation embeddings with dialect samples to obtain dialect pronunciation feature data; The speech recognition module is used to perform dialect adaptation on dialect pronunciation feature data and complete cross-dialect speech recognition based on information decoupling.

10. An electronic device, characterized in that: It includes: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement a cross-dialect speech recognition method based on information decoupling as described in any one of claims 1-8.

Images