A method for constructing a speech recognition model based on cross-domain alignment and intra-domain differentiation

Abstract

This invention belongs to the fields of speech recognition and domain adaptation, and more specifically relates to a method for constructing a speech recognition model based on cross-domain alignment and intra-domain differentiation. This method, in addition to aligning cross-domain features, adds an intra-domain differentiation module to mitigate the damage to the model's discriminative ability in the target domain caused by the alignment of cross-domain features. This method does not require real labels for the target domain data, saving data annotation costs and improving the model's accuracy in the target domain.

Description

Technical Field

[0001] This invention belongs to the fields of speech recognition and domain adaptation, and more specifically relates to a method for constructing a speech recognition model based on cross-domain alignment and intra-domain differentiation. Background Technology

[0002] With the rise of deep learning technology, end-to-end speech recognition technology, relying on large amounts of labeled data, has developed rapidly. Under the premise that training and test data are similarly distributed, current speech recognition models have achieved extremely high accuracy. However, in practical applications, the performance of speech recognition models degrades significantly due to data distribution shifts caused by accents, environment, and even the microphone used for recording. Recording and labeling audio for different scenarios requires substantial human and material resources. Therefore, enhancing the robustness of speech recognition systems and improving their performance in low-resource target domains such as dialects and noisy environments has become a hot research topic in recent years.

[0003] Research has revealed a discrepancy in data distribution between training and testing data, indicating a domain shift between the source and target domains. Mitigating this domain shift can effectively improve the accuracy of speech recognition systems. Based on this, many studies have focused on globally aligning features extracted from source and target domain data, such as domain adversarial learning techniques and generative adversarial networks (GANs). Other studies have employed finer-grained character-level feature alignment to further enhance model accuracy.

[0004] Nevertheless, current domain adaptation techniques are not yet fully capable of adapting to complex and ever-changing scenarios. How to further improve model performance has become a key challenge in current research on domain adaptation methods for speech recognition. Summary of the Invention

[0005] Existing methods simply align cross-domain features to enhance the model's transferability, but ignore the fact that the model's discriminative ability in the target domain is impaired by forced alignment.

[0006] To mitigate the impact of domain offset between source and target domain data on the performance of speech recognition systems, this invention proposes a speech recognition model construction method based on cross-domain alignment and intra-domain discrimination. This method, in addition to aligning cross-domain features, adds an intra-domain discrimination module to reduce the impairment of the model's discriminative ability in the target domain caused by the alignment of cross-domain features. This method does not require real labels for the target domain data, saving data annotation costs and improving the model's accuracy in the target domain.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A method for constructing a speech recognition model based on cross-domain alignment and intra-domain differentiation includes the following steps:

[0009] S101, Data Processing: Train a speech recognition model based on an encoder and a joint decoder using labeled source domain data to recognize preprocessed unlabeled target domain data and generate target domain data with pseudo-labels; perform data augmentation on the target domain data to obtain augmented target domain data. The labeled source domain data, pseudo-labeled target domain data, and augmented target domain data are now ready.

[0010] S102, Feature extraction: The labeled source domain data, the pseudo-labeled target domain data, and the augmented target domain data are fed into the feature encoder to extract the intermediate feature representations of each part.

[0011] S103, Speech Recognition: Input the intermediate feature representations corresponding to labeled source domain data and pseudo-labeled target domain data into the speech recognition joint decoder to calculate the speech recognition loss.

[0012] S104, label frame-level character pseudo-labels, using the decoder of the speech recognition model to label the extracted intermediate features of each frame with character pseudo-labels;

[0013] S105, cross-domain alignment: using the frame-level character pseudo-labels of the intermediate features of the source and target domains obtained in S104, the distribution of the intermediate features of each character in the source and target domains is obtained. The data distribution fitting method is then used to align the feature distributions of the source and target domains belonging to the same character, and the cross-domain alignment loss is calculated.

[0014] S106. Intra-domain differentiation: Using S104, obtain frame-level character pseudo-labels corresponding to the intermediate features of the target domain and the augmented target domain. Obtain the distribution of the intermediate features of each character in the target domain and the augmented target domain. With the feature centers of the same character as positive pairs and the feature centers of different characters as negative pairs, use contrastive learning techniques to make the feature centers of the same character in the target domain and the augmented target domain cluster together and the feature centers of different characters move away from each other, so as to simultaneously enhance the model's ability to differentiate within the target domain. Calculate the intra-domain differentiation loss.

[0015] S107, jointly optimize the model, jointly optimize the three loss functions of speech recognition, cross-domain alignment, and intra-domain discrimination until the loss converges, and a speech recognition model that performs well on the target domain can be obtained.

[0016] Further optimization of this technical solution includes the following processing methods for unlabeled target domain data in step S101: filtering and deleting abnormal data that is extremely long or short; and data augmentation methods include speed transformation, pitch transformation, adding reverberation, and time masking.

[0017] In a further optimization of this technical solution, the feature encoder in step S102 is a Transformer encoder.

[0018] A further optimization of this technical solution is that the speech recognition loss function is:

[0019] L ASR (X,Y)=λL CTC (X,Y)+(1-λ)L ATT (X,Y)

[0020] Where X represents source domain data and target domain data, Y represents source domain data labels and target domain data pseudo-labels, and L represents... CTC It is CTC loss, L ATT It is an attention loss, and the hyperparameter λ balances these two losses.

[0021] In a further optimization of this technical solution, the decoder in step S104 is a CTC decoder.

[0022] In a further optimization of this technical solution, step S104 utilizes argmax and the probability distribution of CTC output to label intermediate features with CTC pseudo-labels frame by frame, as shown in the following formula:

[0023]

[0024] In a further optimization of this technical solution, the cross-domain alignment loss in step 105 is calculated as follows:

[0025]

[0026] Where N is the total number of characters, S and T represent the source domain and destination domain respectively, and C i This represents the i-th character in symbol set C. Let be the regenerating kernel Hilbert space, and k be the Gaussian kernel function.

[0027] A further optimization of this technical solution is provided, with the following example of loss calculation in step 106:

[0028]

[0029] Among them: ψ(a, b)=exp(sim(f(a), f(b)) / τ)

[0030]

[0031] Where 1≤i,j≤N, T represents the target domain, aug represents the augmented target domain, and C i This represents the i-th character in the dictionary. Let sim(u,v) be the feature center, sim(u,v) represent the cosine similarity between u and v, f() represent the intermediate features extracted by the feature extraction module, τ be the temperature hyperparameter, and L be the final value. DI It is the average loss over all positive pairs.

[0032] In a further optimization of this technical solution, the loss function of the speech recognition model in step 107 is as follows:

[0033]

[0034] Among them, L ASR For speech recognition loss, L MA For cross-domain alignment loss, L DI The loss is used to distinguish within the domain, and α and β are hyperparameters that balance the two losses.

[0035] Unlike existing technologies, the above technical solution has the advantage that by applying the speech recognition domain adaptation method proposed in this invention, the model's cross-domain alignment capability at the character level and its discrimination capability at the target domain can be improved simultaneously, thereby enhancing the performance of the speech recognition model on cross-domain tasks such as different devices, different background sounds, and different accents. Attached Figure Description

[0036] Figure 1 A flowchart illustrating the method for building a speech recognition model. Detailed Implementation

[0037] To explain in detail the technical content, structural features, objectives, and effects of the technical solution, the following description is provided in conjunction with specific embodiments and accompanying drawings.

[0038] The technical solution adopted in this invention will be divided into two stages: data preparation and domain adaptation.

[0039] Data preparation phase: Given labeled source domain data and unlabeled target domain data, a speech recognition model is first trained using the source domain data, and then this model is used to generate pseudo-labels for the unlabeled target domain data. Next, data augmentation is performed on the target domain data to obtain augmented target domain data. The resulting dataset includes labeled source domain data, (pseudo)labeled target domain data, and augmented target domain data.

[0040] In the domain adaptation phase, the three parts of data mentioned above are fed into the feature encoder to extract feature representations, and pseudo-labels are applied to each extracted frame of feature representation. Then, two losses are calculated at the character level: cross-domain alignment and intra-domain discrimination. For the cross-domain alignment loss, a data distribution fitting method (such as Maximum Mean Difference MMD) is used to align the feature distributions of the same character belonging to the source and target domains, thereby enhancing the model's cross-domain alignment capability. For the intra-domain discrimination loss, contrastive learning techniques are used to cluster the feature centers of the same character in the target domain data features and the augmented target domain data features, while dispersing the feature centers of different characters, thereby simultaneously enhancing the model's discrimination capability within the target domain and improving the model's accuracy in the target domain.

[0041] Please see Figure 1 The diagram shown is a flowchart of a speech recognition model construction method. A preferred embodiment of the present invention includes the following steps:

[0042] S101. Data Processing. Given unlabeled target domain data, firstly, all data is filtered to remove outliers such as extremely long or short data. Secondly, a speech recognition model based on a Transformer encoder and a CTC-Attention (Connectionist Temporal Classification, CTC) joint decoder is trained using labeled source domain data to recognize the unlabeled target domain data and generate pseudo-labeled target domain data. Finally, the target domain data is augmented using techniques such as velocity transformation, pitch transformation, adding reverberation, and time masking to obtain augmented target domain data. At this point, the three parts of data—labeled source domain data, pseudo-labeled target domain data, and (pseudo-labeled) augmented target domain data—are ready.

[0043] S102. Feature Extraction. The above three parts of data are fed into the feature encoder, such as the Transformer encoder, to extract the intermediate feature representations corresponding to each part.

[0044] S103, Speech Recognition. The intermediate feature representations corresponding to the labeled source domain data and the pseudo-labeled target domain data are input into the CTC-Attention joint decoder for speech recognition, and the speech recognition loss is calculated using the following formula:

[0045] L ASR (X,Y)=λL CTC (X,Y)+(1-λ)L ATT (X,Y)

[0046] Where X represents the source domain data and the target domain data, and Y represents the source domain data label and the target domain data pseudo-label. L CTC It is CTC loss, LATT It's an attention loss. The hyperparameter λ balances these two losses.

[0047] S104. Label frame-level character pseudo-labels. Utilize the decoder of the speech recognition model (such as the CTC decoder) to label each extracted intermediate feature with character pseudo-labels, i.e., characters from the dictionary such as "a", "b", etc. Taking the CTC decoder as an example, the intermediate features can be labeled with CTC pseudo-labels frame by frame using argmax and the probability distribution of the CTC output, as shown in the following formula.

[0048]

[0049] Where P CTC Let X be the probability distribution output by the CTC decoder. n Let X be the input of the nth frame, and Y be the input of the nth frame. n Let X be the possible labels for the nth frame. It is to make P CTC The largest X n Frame-level pseudo-tags.

[0050] S105. Cross-domain alignment. Using the frame-level character pseudo-labels of the intermediate features between the source and target domains obtained in S104, the distribution of the intermediate features corresponding to each character in the source and target domains can be obtained. Further, a data distribution fitting method (such as Maximum Mean Difference (MMD)) is used to align the feature distributions of the source and target domains belonging to the same character, and the cross-domain alignment loss L is calculated. MA The formula is as follows:

[0051]

[0052] Where N is the total number of characters. S and T represent the source domain and destination domain, respectively, and C... i This represents the i-th character in the symbol set C. Let H be the regenerated kernel Hilbert space, and k be the Gaussian kernel function used in this invention.

[0053] S106. Intra-domain differentiation. Using S104 to obtain frame-level character pseudo-labels corresponding to the intermediate features of the target domain and the augmented target domain, the distribution of the intermediate features of each character in the target domain and the augmented target domain can be obtained. Since the number of frames extracted from the speech data is large, the time complexity of frame-level contrastive learning is too high. This invention averages the average intermediate features corresponding to each character to obtain the approximate feature center corresponding to each character. Using feature centers of the same character as positive pairs and feature centers of different characters as negative pairs, contrastive learning techniques (such as NT_XENT loss) are employed to cluster feature centers of the same character in the target domain and the augmented target domain, while distancing feature centers of different characters from each other. This simultaneously enhances the model's discriminative ability within the target domain, and the intra-domain discriminative loss L is calculated.DI An example of loss calculation is as follows:

[0054]

[0055] Among them: ψ(a, b)=exp(sim(f(a), f(b)) / τ)

[0056]

[0057] in Let C be the feature center, 1≤i,j≤N, T represent the target domain, aug represent the augmented target domain, and C be the feature center. i Let represent the i-th character in the dictionary, and sim(u,v) represent the cosine similarity between u and v. f() represents the intermediate features extracted by the feature extraction module, and τ is the temperature hyperparameter. The final L... DI It is the average loss over all positive pairs.

[0058] S107. Joint Optimization Model. Three loss functions—speech recognition, cross-domain alignment, and intra-domain discrimination—are jointly optimized until convergence, resulting in a speech recognition model that performs well in the target domain. The losses are as follows:

[0059]

[0060] α and β are hyperparameters that balance the two losses.

[0061] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Unless otherwise specified, an element defined by the phrase "comprising..." or "including..." does not exclude the presence of additional elements in the process, method, article, or terminal device that includes said element. Additionally, in this document, "greater than," "less than," "exceeding," etc., are understood to exclude the stated number; "above," "below," "within," etc., are understood to include the stated number.

[0062] Although the above embodiments have been described, those skilled in the art, once they understand the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the above descriptions are merely embodiments of the present invention and do not limit the scope of patent protection of the present invention. Any equivalent structural or procedural transformations made using the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1.A method for constructing a speech recognition model based on cross-domain alignment and intra-domain differentiation, characterized in that, The method comprises the following steps: S101, data processing, using a labeled source domain data to train an encoder and joint decoder based speech recognition model to identify preprocessed unlabeled target domain data, generate target domain data with pseudo labels; data augmentation is performed on the target domain data to obtain augmented target domain data; the labeled source domain data, the target domain data with pseudo labels, and the augmented target domain data are prepared; S102, feature extraction, the labeled source domain data, the target domain data with pseudo labels, and the augmented target domain data are transmitted into a feature encoder to extract intermediate feature representations of each part; S103, speech recognition, the intermediate feature representations corresponding to the labeled source domain data and the target domain data with pseudo labels are input into a speech recognition joint decoder to calculate a speech recognition loss; S104, frame-level character pseudo label annotation, a decoder of the speech recognition model is used to mark a character pseudo label for each frame of intermediate feature extracted from each part of data; S105, cross-domain alignment, frame-level character pseudo labels of intermediate features of the source domain and the target domain obtained in S104 are used to obtain distributions of intermediate features corresponding to each character in the source domain and the target domain, and a data distribution fitting method is further used to align the source domain and the target domain feature distributions belonging to the same character to calculate a cross-domain alignment loss; S106, intra-domain differentiation, frame-level character pseudo labels corresponding to intermediate features of the target domain and the augmented target domain obtained in S104 are used to obtain distributions of intermediate features corresponding to each character in the target domain and the augmented target domain, and a contrast learning technology is used to make the feature centers of the same characters in the target domain and the augmented target domain gather and the feature centers of different characters move away from each other to simultaneously enhance the intra-domain differentiation ability of the model in the target domain, and an intra-domain differentiation loss is calculated; S107, jointly optimize the model, the speech recognition, the cross-domain alignment, and the intra-domain differentiation three loss functions are jointly optimized until the loss converges, and a speech recognition model with good performance in the target domain is obtained. 2.The method for constructing a speech recognition model based on cross-domain alignment and intra-domain distinction according to claim 1, wherein, The processing mode of the unlabeled target domain data in the step S101 comprises screening and deleting extreme long and short abnormal data; the data augmentation mode comprises speed transformation, tone transformation, adding reverberation, and time masking. 3.The method of constructing a speech recognition model based on cross-domain alignment and intra-domain distinction according to claim 1, wherein, The feature encoder in the step S102 is a Transformer encoder. 4.The method for constructing a speech recognition model based on cross-domain alignment and intra-domain distinction according to claim 1, wherein, The speech recognition loss function is: wherein is source domain data, target domain data, is source domain data label, target domain data pseudo label, is CTC loss, is attention loss, hyperparameter balances the two losses. 5.The method for constructing a speech recognition model based on cross-domain alignment and intra-domain distinction according to claim 1, wherein, The decoder in the step S104 is a CTC decoder. 6.The method for constructing a speech recognition model based on cross-domain alignment and intra-domain distinction according to claim 1, wherein, The step S104 labels the intermediate feature frame by frame with the CTC pseudo label by using argmax and the probability distribution output by CTC, and the formula is as follows: wherein is the probability distribution output by the CTC decoder, is the first frame input, is the possible label of the first frame, is the frame-level pseudo label that makes the maximum . 7.The method for constructing a speech recognition model based on cross-domain alignment and intra-domain distinction according to claim 1, wherein, The cross-domain alignment loss in step 105 is as follows: Wherein is the total number of characters, and respectively represent the source domain, the target domain, represent the first character of the symbol set , is a reproducing kernel Hilbert space, is a Gaussian kernel function. 8.The method for constructing a speech recognition model based on cross-domain alignment and intra-domain distinction according to claim 1, wherein, The loss calculation in step 106 is as follows: wherein: wherein is the feature center, , represents the target domain, represents the augmented target domain, represents the i-th character in the dictionary, represents represents and the cosine similarity of and represents the intermediate feature extracted by the feature extraction module, is the temperature hyperparameter, and the final is the average value of the loss for all positive pairs. 9.The method for constructing a speech recognition model based on cross-domain alignment and intra-domain distinction according to claim 1, wherein, The loss function of the speech recognition model in the step 107 is as follows: wherein, is the speech recognition loss, is the cross-domain alignment loss, is the intra-domain discrimination loss, and is a hyperparameter balancing the two losses.

Images