A speech recognition model training method, device, apparatus and storage medium

Abstract

The application discloses a speech recognition model training method and device, equipment and a storage medium, relates to the technical field of speech recognition, and comprises the following steps: processing an initial audio signal by using a preset audio processing method, and obtaining initial speech training data based on an acoustic feature consistency constraint condition, an identification accuracy constraint condition and a teacher-student comparison constraint condition and a target feature sequence obtained by processing; processing the initial speech training data by using a speech synthesis center, processing obtained to-be-processed speech training data by using a preset acoustic scene simulation system, and obtaining to-be-enhanced speech training data; processing the to-be-enhanced speech training data based on a word level by using an attention enhancement mechanism to obtain word-enhanced speech training data, processing the word-enhanced speech training data based on a sentence level by using a comparison learning framework, training an initial speech recognition model by using obtained target speech training data, and obtaining a target speech recognition model. In this way, the efficiency of training the speech recognition model can be improved.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a speech recognition model training method, apparatus, device, and storage medium. Background Technology

[0002] Currently, mainstream methods in the field of speech recognition are primarily based on deep neural networks, including Transformer (a neural network architecture based on self-attention mechanisms) and Conformer (Convolution-augmented Transformer). However, while these models possess strong feature extraction and sequence modeling capabilities, they consume significant computational resources, have high deployment costs, and are difficult to run on resource-constrained terminal devices. Furthermore, due to limitations in training data, the recognition accuracy of these models is often less than ideal when processing proper nouns and keywords.

[0003] Furthermore, traditional knowledge distillation methods primarily focus on the consistency of the output distributions of the teacher and student models, typically using KL (Kullback-Leibler Divergence, i.e., relative entropy) divergence or mean squared error as distillation loss. However, this approach has limited effectiveness when dealing with black-box teacher models and cannot fully utilize the deep knowledge within them. In addition, existing data augmentation methods mostly rely on simple signal processing and lack specific mechanisms for generating training data for specialized terminology.

[0004] As can be seen from the above, how to improve the efficiency of training speech recognition models is an urgent problem to be solved. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a speech recognition model training method, apparatus, device, and storage medium, which can improve the efficiency of training speech recognition models during the training process. The specific solution is as follows:

[0006] Firstly, this application provides a method for training a speech recognition model, including:

[0007] The initial audio signal is processed using a preset frequency domain modulation method and a preset time domain stretching method to obtain a target feature sequence. This target feature sequence is then processed based on acoustic feature consistency constraints, recognition accuracy constraints, and teacher-student comparison constraints to obtain initial speech training data. The acoustic feature consistency constraints are used to constrain the consistency of the acoustic features of the target feature sequence. The teacher-student comparison constraints are used to constrain the process of training the initial speech recognition model based on the target teacher-student comparison results. The target teacher-student comparison results are obtained by comparing the outputs of the teacher model and the student model.

[0008] A speech synthesis center is established based on several TTS models. The initial speech training data is processed using the speech synthesis center to obtain speech training data to be processed. The speech training data to be processed is then processed using a preset acoustic scene simulation system to obtain speech training data to be enhanced.

[0009] The attention enhancement mechanism based on multi-granularity features is used to perform keyword recognition enhancement processing on the speech training data to be enhanced at the word level to obtain word-enhanced speech training data. Then, the contrastive learning framework is used to perform sentence recognition enhancement processing on the word-enhanced speech training data at the sentence level to obtain target speech training data.

[0010] The initial speech recognition model is trained using the acoustic feature consistency loss function and the contrastive learning loss function, based on the target speech training data, to obtain the target speech recognition model.

[0011] Optionally, the step of processing the initial audio signal using a preset frequency domain modulation method and a preset time domain stretching method to obtain the target feature sequence includes:

[0012] Establish a comparative consistency distillation framework that includes several dimensional constraint mechanisms;

[0013] The initial audio signal is frequency-domain converted using the contrast-consistent distillation framework and based on short-time Fourier transform technology to obtain a first audio signal to be processed. The first audio signal to be processed is then processed using phase vocoder technology and a preset pitch preservation rule to obtain a second audio signal to be processed.

[0014] The second audio signal to be processed is used to extract features based on the contrast consistency distillation framework and a preset window overlap rate and several time window sizes to obtain corresponding audio features, and a feature sequence to be processed is established based on each audio feature; the feature sequence to be processed includes local speech information and long-term acoustic structure information.

[0015] A preset attention mechanism is used to perform time alignment operation on the dynamic time warping algorithm to obtain a target time warping algorithm. The target time warping algorithm is then used to process the feature sequence to be processed to obtain a target feature sequence.

[0016] Optionally, the initial speech training data is obtained by processing the target feature sequence based on acoustic feature consistency constraints, recognition accuracy constraints, and teacher-student contrast constraints, including:

[0017] Using preset feature extractor establishment rules, feature extractors are established for the teacher model and the student model respectively, resulting in a first feature extractor and a second feature extractor.

[0018] The first feature extractor, the second feature extractor, and the dynamic time warping algorithm are used, and the acoustic feature consistency constraint is determined based on a preset balance factor. The target feature sequence is then processed using the acoustic feature consistency constraint to obtain the first feature sequence.

[0019] The first feature sequence is input into the teacher model and the student model respectively to obtain the first feature extraction result corresponding to the teacher model and the second feature extraction result corresponding to the student model, and the first feature extraction result and the second feature extraction result are set as positive sample pairs;

[0020] The student model is processed using the teacher-student comparison constraints and based on the first feature extraction result and the second feature extraction result to obtain an initial speech recognition model;

[0021] The similarity of each of the first feature sequences is calculated based on the cosine similarity function to obtain the corresponding similarity results. Audio signals with a similarity of not less than a preset similarity threshold and corresponding to different audio types in the similarity results are identified as hard negative samples.

[0022] The second feature sequence is determined using the cosine similarity function and based on the positive sample pairs, the hard negative samples, and the preset temperature parameters.

[0023] The second feature sequence is processed using an attention decoder to obtain cross-entropy loss. Then, the second feature sequence is processed using the CTC loss function based on a preset regularization term, a preset weight coefficient, and the cross-entropy loss to obtain initial speech training data.

[0024] Optionally, the step of establishing a speech synthesis center based on several TTS models to process the initial speech training data using the speech synthesis center to obtain speech training data to be processed includes:

[0025] A speech synthesis center is established based on the FastSpeech2 model with a non-autoregressive architecture, the Tacotron2 model with a sequence-to-sequence architecture, and the VITS model with an end-to-end conditional variational autoencoder structure.

[0026] Phoneme analysis is performed on the initial speech training data to obtain phoneme analysis results, and the phoneme analysis results are processed using a preset audio-text alignment tool to obtain phoneme boundary information;

[0027] Based on the acoustic features corresponding to the phoneme boundary information, a speech synthesis model is determined from the speech synthesis center, and the initial speech training data is synthesized using the speech synthesis model to obtain the speech training data to be processed; the acoustic features include vowels, consonants, phoneme duration, and pitch contours.

[0028] Optionally, the step of processing the speech training data to be processed using a preset acoustic scene simulation system to obtain speech training data to be enhanced includes:

[0029] An acoustic feature library is established based on acoustic parameter description information corresponding to several acoustic scenarios; the acoustic parameter description information includes room impulse response information, signal-to-noise ratio distribution information, and reverberation time.

[0030] Determine whether the current ambient noise is transient noise. If the current ambient noise is transient noise, then use a preset acoustic physical model and an energy envelope-based triggering mechanism, and perform time-frequency feature adjustment on the speech training data to be processed based on the corresponding room impulse response and time-varying mixing coefficient to obtain the speech training data to be enhanced.

[0031] If the current ambient noise is not transient noise, then the preset acoustic physical model and preset mixing strategy are used, and the time-frequency feature adjustment operation is performed on the speech training data to be processed based on the current ambient noise, the room impulse response and the time-varying mixing coefficient to obtain the speech training data to be enhanced.

[0032] Optionally, the attention enhancement mechanism based on multi-granularity features performs keyword recognition enhancement processing on the speech training data to be enhanced at the word level to obtain word-enhanced speech training data. Then, a contrastive learning framework is used to perform sentence recognition enhancement processing on the word-enhanced speech training data at the sentence level to obtain target speech training data, including:

[0033] An attention enhancement mechanism is established based on multi-granularity features. The attention enhancement mechanism is then used to perform keyword recognition enhancement processing on the speech training data to be enhanced based on word level, so as to obtain word-enhanced speech training data.

[0034] The context dependencies are determined by using a preset multi-head self-attention mechanism and based on a preset positional encoding to obtain the context dependencies of the word-enhanced speech training data.

[0035] The target speech training data is obtained by using a contrastive learning framework and performing sentence recognition enhancement processing on the word-enhanced speech training data based on sentence level and the context dependency relationship.

[0036] Optionally, the step of using the acoustic feature consistency loss function and the contrastive learning loss function, and training the initial speech recognition model based on the target speech training data to obtain the target speech recognition model, includes:

[0037] A data scoring system is established based on phoneme complexity, acoustic variability, and environmental complexity. The phoneme complexity includes the conditional entropy of the phoneme combination corresponding to the target speech training data. The acoustic variability is the dynamic feature of the audio segment obtained after processing the target speech training data based on the dynamic time warping algorithm. The environmental complexity includes the signal-to-noise ratio and reverberation time.

[0038] A contrastive learning loss function is determined based on the phoneme similarity to the target speech training data. Then, the target speech training data is scored using the data scoring system to obtain a scoring result, and it is determined whether the scoring result is greater than a preset scoring threshold.

[0039] If the score result is greater than the preset score threshold, the initial speech recognition model is trained using the acoustic feature consistency loss function, the contrastive learning loss function, and the joint loss function, based on the data corresponding to the score result, to obtain the target speech recognition model.

[0040] Secondly, this application provides a speech recognition model training device, comprising:

[0041] The feature sequence determination module processes the initial audio signal using a preset frequency domain modulation method and a preset time domain stretching method to obtain a target feature sequence. It then processes the target feature sequence based on acoustic feature consistency constraints, recognition accuracy constraints, and teacher-student comparison constraints to obtain initial speech training data. The acoustic feature consistency constraints constrain the consistency of the acoustic features of the target feature sequence. The teacher-student comparison constraints constrain the process of training the initial speech recognition model based on the target teacher-student comparison results, where the target teacher-student comparison results are obtained by comparing the outputs of the teacher model and the student model.

[0042] The training data processing module is used to establish a speech synthesis center based on several TTS models, and to use the speech synthesis center to process the initial speech training data to obtain speech training data to be processed, and to use a preset acoustic scene simulation system to process the speech training data to be processed to obtain speech training data to be enhanced.

[0043] The training data determination module is used to perform keyword recognition enhancement processing on the speech training data to be enhanced at the word level using an attention enhancement mechanism based on multi-granularity features to obtain word-enhanced speech training data. Then, it uses a contrastive learning framework to perform sentence recognition enhancement processing on the word-enhanced speech training data at the sentence level to obtain target speech training data.

[0044] The speech recognition model training module is used to train the initial speech recognition model based on the target speech training data using the acoustic feature consistency loss function and the contrastive learning loss function to obtain the target speech recognition model.

[0045] Thirdly, this application provides an electronic device, comprising:

[0046] Memory, used to store computer programs;

[0047] A processor is used to execute the computer program to implement the aforementioned speech recognition model training method.

[0048] Fourthly, this application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned speech recognition model training method.

[0049] As can be seen from the above, before training the speech recognition model, this application needs to process the initial audio signal using a preset frequency domain modulation method and a preset time domain stretching method to obtain the target feature sequence. Then, based on acoustic feature consistency constraints, teacher-student contrast constraints, and recognition accuracy constraints, the target feature sequence is further processed to obtain initial speech training data. A speech synthesis center is established based on several TTS models to process the initial speech training data, obtaining speech training data to be processed. A preset acoustic scene simulation system is then used to process the speech training data to be processed, obtaining speech training data to be enhanced. An attention enhancement mechanism based on multi-granularity features is used to perform keyword recognition enhancement processing on the speech training data to be enhanced at the word level, obtaining word-enhanced speech training data. Then, a contrastive learning framework is used to perform sentence recognition enhancement processing on the word-enhanced speech training data at the sentence level, obtaining the target speech training data. Finally, the initial speech recognition model is trained using the acoustic feature consistency loss function, the contrastive learning loss function, and the joint loss function, based on the target speech training data, to obtain the target speech recognition model.

[0050] Therefore, this application first processes the initial audio signal using a preset frequency domain modulation method and a preset time domain stretching method to obtain a target feature sequence. Then, based on acoustic feature consistency constraints, teacher-student contrast constraints, and recognition accuracy constraints, the target feature sequence is further processed to obtain initial speech training data. Subsequently, a speech synthesis center is established based on several TTS models to process the initial speech training data, obtaining speech training data to be processed. This data is then processed using a preset acoustic scene simulation system to obtain speech training data to be enhanced. Next, an attention enhancement mechanism based on multi-granularity features is used to perform keyword recognition enhancement processing on the speech training data to be enhanced at the word level, obtaining word-enhanced speech training data. Then, a contrastive learning framework is used to perform sentence recognition enhancement processing on the word-enhanced speech training data at the sentence level, obtaining the target speech training data. Finally, the initial speech recognition model is trained using the acoustic feature consistency loss function, the contrastive learning loss function, and the joint loss function, based on the target speech training data, to obtain the target speech recognition model. This improves the efficiency of training the speech recognition model, thereby increasing the accuracy of speech recognition. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0052] Figure 1 This is a flowchart of a speech recognition model training method disclosed in this application;

[0053] Figure 2 This is a flowchart of a specific speech recognition model training method disclosed in this application;

[0054] Figure 3 This is a schematic diagram illustrating a specific process for constructing a contrastive learning framework as disclosed in this application;

[0055] Figure 4 This is a schematic diagram showing the results obtained from testing using test data and corresponding to several evaluation indicators, as disclosed in this application.

[0056] Figure 5 This is a schematic diagram illustrating the accuracy of a recognition test using proper nouns, as disclosed in this application, corresponding to several scene types.

[0057] Figure 6 This is a schematic diagram of the structure of a speech recognition model training device disclosed in this application;

[0058] Figure 7 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] Currently, in the field of speech recognition, mainstream methods are mainly based on deep neural networks. However, although these models have strong feature extraction and sequence modeling capabilities, they consume large amounts of computational resources, have high deployment costs, and are difficult to run on resource-constrained terminal devices. Furthermore, due to limitations in training data, the recognition accuracy of these models when processing proper nouns and keywords is often less than ideal. In addition, traditional knowledge distillation methods mainly focus on the consistency of the output distributions of the teacher and student models, typically using KL divergence or mean squared error as distillation loss. However, this method has limited effectiveness when dealing with black-box teacher models and cannot fully utilize the deep knowledge within the teacher model. Moreover, existing data augmentation methods are mostly based on simple signal processing and lack targeted mechanisms for generating training data for proper nouns. Therefore, this application provides a speech recognition model training method that can improve the efficiency of training speech recognition models during the training process.

[0061] See Figure 1 As shown in the figure, an embodiment of the present invention discloses a speech recognition model training method, including:

[0062] Step S11: Process the initial audio signal using a preset frequency domain modulation method and a preset time domain stretching method to obtain a target feature sequence. Then, process the target feature sequence based on acoustic feature consistency constraints, recognition accuracy constraints, and teacher-student comparison constraints to obtain initial speech training data. The acoustic feature consistency constraints are used to constrain the consistency of the acoustic features of the target feature sequence. The teacher-student comparison constraints are used to constrain the process of training the initial speech recognition model based on the target teacher-student comparison results. The target teacher-student comparison results are the results obtained by comparing the outputs of the teacher model and the student model.

[0063] In this embodiment, the speech recognition model training process went through three stages, as follows: Figure 2As shown in the diagram, in the first stage, a constraint mechanism including acoustic feature consistency constraints, teacher-student contrast constraints, and recognition accuracy constraints is introduced to establish a contrast consistency distillation method. This method overcomes the limitation of traditional knowledge distillation techniques that only focus on output distribution alignment. Through the synergistic effect of various constraint mechanisms, deep knowledge transfer from the teacher model to the student model is achieved. In the second stage, a TTS (Text-to-Speech Synthesis Model) data augmentation method is designed. This method generates high-quality and diverse training data by employing a multi-model collaborative speech synthesis strategy and cooperating with multiple TTS models of different architectures. Simultaneously, an acoustic scene simulation system is introduced to contextualize the generated data using a physical acoustic model, significantly improving the realism and practicality of the synthesized data. In the third stage, a keyword recognition enhancement method is developed. This method utilizes a multi-granularity feature attention enhancement mechanism and a hierarchical contrastive learning strategy to perform multi-level feature extraction and optimization operations, effectively improving the model's ability to recognize proper nouns and keywords.

[0064] Furthermore, in the first stage of the contrastive consistency distillation operation, this embodiment first needs to conduct an in-depth analysis of the knowledge distillation mechanism. Traditional knowledge distillation techniques mainly rely on Kullback-Leibler divergence to align the output distributions of the teacher and student models. This embodiment, however, designs a contrastive learning distillation framework. In this framework, the input audio signal x is first transformed and enhanced, and variants are generated through frequency domain modulation and time domain stretching. In one specific implementation, for frequency domain modulation, this embodiment first uses short-time Fourier transform technology to convert the signal to the frequency domain, and then adjusts the spectral envelope to achieve continuous pitch variation, controlling the variation range within ±20%. For time domain stretching, this embodiment uses phase vocoder technology to process the signal to achieve continuous variation at 0.8-1.2 times the speed while maintaining the pitch, thus preserving the basic semantic information of the speech while providing rich acoustic variations.

[0065] Specifically, the initial audio signal is processed using a preset frequency domain modulation method and a preset time domain stretching method to obtain a target feature sequence. This process may include: establishing a contrast-consistency distillation framework with several dimensional constraint mechanisms; using the contrast-consistency distillation framework and based on short-time Fourier transform technology to perform frequency domain transformation on the initial audio signal to obtain a first audio signal to be processed; processing the first audio signal to be processed using phase vocoder technology and a preset pitch preservation rule to obtain a second audio signal to be processed; using the contrast-consistency distillation framework and based on a preset window overlap rate and several time window sizes to extract features from the second audio signal to obtain corresponding audio features; and establishing a feature sequence to be processed based on each audio feature. The feature sequence to be processed includes local speech information and long-term acoustic structure information; performing time alignment operation on the dynamic time warping algorithm using a preset attention mechanism to obtain a target time warping algorithm; and processing the feature sequence to be processed using the target time warping algorithm to obtain the target feature sequence.

[0066] Furthermore, in the feature extraction process, this application embodiment uses multi-scale Log-Mel spectrograms as acoustic features. In one specific implementation, three different time window sizes (10ms, 25ms, 50ms) are used to perform feature extraction on the signal. The window overlap rate is set to 50%, which realizes multi-scale feature fusion operation, capturing both local speech details and preserving long-term acoustic structure information.

[0067] In this embodiment, after obtaining the fused feature sequence, the present application embodiment needs to use the attention-enhanced DTW algorithm to perform temporal alignment operation on the feature sequence. The above algorithm introduces a self-attention mechanism so as to better handle the alignment problem of long temporal speech.

[0068] It is worth mentioning that the first constraint condition in the triple constraint mechanism used in this application embodiment is the acoustic feature consistency constraint, and its expression is as follows:

[0069] L_acoustic = KL(F_t(x), F_s(x)) + λ_temporal * L_DTW(F_t(x), F_s(x));

[0070] Where F_t and F_s represent the feature extractors of the teacher model and student model, respectively, L_DTW represents the temporal alignment loss based on DTW (Dynamic Time Warping), and λ_temporal is the balancing factor with an initial value of 0.5, which can be dynamically adjusted as the training process progresses.

[0071] The second constraint is teacher-student knowledge transfer based on contrastive learning. At the individual audio level, this embodiment uses the outputs of the teacher model and the student model after processing the same audio as positive sample pairs. Furthermore, the selection of negative samples is based on acoustic similarity for hard negative sample mining. In one specific implementation, cosine similarity is used to measure the similarity of different audio features, and samples with a similarity greater than a preset similarity threshold but not belonging to the same category are selected as hard negative samples. The expression for the contrastive loss function is as follows:

[0072] L_contrast = -log(exp(sim(z_t, z_s) / τ) / Σexp(sim(z_t, z_n) / τ));

[0073] Where z_t and z_s are the feature representations of the teacher model and the student model, respectively, z_n is the negative sample feature, sim() is the cosine similarity function, and τ is the temperature parameter, which is determined by grid search on the validation set.

[0074] The third constraint is the recognition accuracy constraint, which uses the improved joint CTC / Attention loss function (Connectionist Temporal Classification Loss Function) for accuracy constraint, and the expression is as follows:

[0075] L_ce = α * L_ctc + (1-α) * L_att + β * L_reg;

[0076] Where L_ctc is the CTC loss, L_att is the cross-entropy loss of the attention decoder, and L_reg is the regularization term used to prevent overfitting. α and β are weight coefficients, dynamically adjusted using the Bayesian (Bayesian Optimization Method) optimization method.

[0077] Specifically, the initial speech training data is obtained by processing the target feature sequence based on acoustic feature consistency constraints, recognition accuracy constraints, and teacher-student contrast constraints. This process may include: establishing feature extractors for the teacher model and student model respectively using preset feature extractor establishment rules to obtain corresponding first and second feature extractors; using the first and second feature extractors with a dynamic time warping algorithm, and determining the acoustic feature consistency constraints based on a preset balance factor, and processing the target feature sequence using the acoustic feature consistency constraints to obtain the first feature sequence; inputting the first feature sequence into the teacher model and student model respectively to obtain the first feature extraction result corresponding to the teacher model and the second feature extraction result corresponding to the student model, and then processing the first feature sequence. The results and the second feature extraction results are set as positive sample pairs; the student model is processed using teacher-student comparison constraints and based on the first and second feature extraction results to obtain the initial speech recognition model; the similarity of each first feature sequence is calculated based on the cosine similarity function to obtain the corresponding similarity results, and audio signals with similarity values ​​not less than a preset similarity threshold and corresponding to different audio types are identified as hard negative samples; the second feature sequence is determined using the cosine similarity function and based on the positive sample pairs, hard negative samples, and preset temperature parameters; the second feature sequence is processed using the attention decoder to obtain the cross-entropy loss, and then the second feature sequence is processed using the CTC loss function based on the preset regularization term, preset weight coefficient, and cross-entropy loss to obtain the initial speech training data.

[0078] Step S12: Establish a speech synthesis center based on several TTS models, and use the speech synthesis center to process the initial speech training data to obtain speech training data to be processed. Then, use a preset acoustic scene simulation system to process the speech training data to be processed to obtain speech training data to be enhanced.

[0079] In this embodiment, during the second-stage TTS data augmentation process, this application embodiment designs a high-quality speech synthesis system based on multi-model collaboration. In one specific implementation, this application embodiment first constructs a high-quality speech synthesis system integrating models including FastSpeech2 (i.e., a Transformer-based non-autoregressive text-to-speech synthesis model), Tacotron2 (i.e., a sequence-to-sequence and attention-based autoregressive text-to-speech synthesis model), and VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech, i.e., a variational inference-based end-to-end text-to-speech synthesis model). These models each have their own characteristics: the FastSpeech2 model corresponds to a non-autoregressive architecture, which has fast inference capabilities; the Tacotron2 model corresponds to a sequence-to-sequence architecture, which performs well in prosodic modeling; and the VITS model corresponds to an end-to-end conditional variational autoencoder structure, which can generate more natural speech. Specifically, a speech synthesis center is established based on several TTS models to process the initial speech training data and obtain the speech training data to be processed. This can include: establishing a speech synthesis center based on a FastSpeech2 model with a non-autoregressive architecture, a Tacotron2 model with a sequence-to-sequence architecture, and a VITS model with an end-to-end conditional variational autoencoder structure; performing phoneme analysis on the initial speech training data to obtain phoneme analysis results, and processing the phoneme analysis results using a preset audio-text alignment tool to obtain phoneme boundary information; determining a speech synthesis model from the speech synthesis center based on the acoustic features corresponding to the phoneme boundary information, and using the speech synthesis model to synthesize the initial speech training data to obtain the speech training data to be processed; the acoustic features include vowels, consonants, phoneme duration, and pitch contours.

[0080] It is worth mentioning that, in the process of speech synthesis, this application proposes an adaptive synthesis strategy guided by acoustic features. This strategy first performs phoneme-level analysis on the input text. In one specific implementation, this application employs an improved Montreal-Forced-Aligner (MFA, a tool that forces alignment between audio and text) for phoneme alignment, thereby obtaining accurate phoneme boundary information. Then, based on the acoustic features of the phonemes, such as vowels, consonants, phoneme duration, and pitch contours, the most suitable synthesis model is dynamically selected. Specifically, for vowel segments with longer durations, the Tacotron2 model is preferentially used for synthesis to obtain a more natural prosodic expression; for consonants and short phonemes, the FastSpeech2 model is preferred for synthesis to ensure pronunciation accuracy.

[0081] In this embodiment, regarding environmental noise simulation, this application embodiment constructs a multi-level acoustic scene simulation system to establish an acoustic feature library containing various typical scenarios using the above system, and uses the preset acoustic scene simulation system to process the speech training data to be processed to obtain the speech training data to be enhanced. Specifically, the speech training data to be processed is processed using a preset acoustic scene simulation system to obtain speech training data to be enhanced. This process may include: establishing an acoustic feature library based on acoustic parameter description information corresponding to several acoustic scenes; the acoustic parameter description information includes room impulse response information, signal-to-noise ratio distribution information, and reverberation time; determining whether the current ambient noise is transient noise; if the current ambient noise is transient noise, then using a preset acoustic physical model and an energy envelope-based triggering mechanism, and based on the corresponding room impulse response and time-varying mixing coefficients, performing time-frequency feature adjustment operations on the speech training data to be processed to obtain speech training data to be enhanced; if the current ambient noise is not transient noise, then using a preset acoustic physical model and a preset mixing strategy, and based on the current ambient noise, room impulse response, and time-varying mixing coefficients, performing time-frequency feature adjustment operations on the speech training data to be processed to obtain speech training data to be enhanced.

[0082] Each scenario includes detailed acoustic parameter descriptions, such as room impulse response, signal-to-noise ratio distribution, and reverberation time. In one specific implementation, during the noise addition process, this embodiment employs a dynamic mixing strategy based on an acoustic physics model, as shown in the following expression:

[0083] L_mixed(t) = α(t)·S(t) + β(t)·[h(t) * N(t)];

[0084] Where S(t) is the original speech signal, N(t) is the ambient noise, h(t) is the room impulse response, and α(t) and β(t) are time-varying mixing coefficients, which are dynamically adjusted by analyzing the time-frequency characteristics of the signal. In particular, for transient noise, an energy envelope-based triggering mechanism is adopted to ensure the reasonable timing of noise addition.

[0085] Step S13: Using an attention enhancement mechanism based on multi-granularity features, keyword recognition enhancement processing is performed on the speech training data to be enhanced at the word level to obtain word-enhanced speech training data. Then, using a contrastive learning framework, sentence recognition enhancement processing is performed on the word-enhanced speech training data at the sentence level to obtain target speech training data.

[0086] In this embodiment, during the third stage of keyword recognition enhancement, this application proposes an attention enhancement mechanism based on multi-granularity features. This mechanism constructs a Transformer-based feature extractor at the word level and captures the contextual dependencies of keywords through a multi-head self-attention mechanism. Specifically, the attention enhancement mechanism based on multi-granularity features performs keyword recognition enhancement processing on the word-level speech training data to be enhanced, obtaining word-enhanced speech training data. Then, a contrastive learning framework is used to perform sentence recognition enhancement processing on the word-enhanced speech training data at the sentence level, obtaining target speech training data. This process can include: establishing an attention enhancement mechanism based on multi-granularity features; using this mechanism to perform keyword recognition enhancement processing on the word-level speech training data to obtain word-enhanced speech training data; using a preset multi-head self-attention mechanism and based on a preset positional encoding to determine the contextual dependencies of the word-enhanced speech training data; and using a contrastive learning framework and based on the sentence level and contextual dependencies to perform sentence recognition enhancement processing on the word-enhanced speech training data to obtain target speech training data.

[0087] In one specific implementation, the expression for the attention calculation formula is as follows:

[0088] Attention(Q,K,V) = softmax(QK^T / sqrt(d_k))V + PE;

[0089] PE stands for positional encoding, employing an improved relative positional encoding scheme that better handles variable-length sequences. Specifically, for keyword positions, an additional positional marker vector is introduced to enhance the representation of positional information.

[0090] Step S14: Using the acoustic feature consistency loss function and the contrastive learning loss function, and based on the target speech training data, train the initial speech recognition model to obtain the target speech recognition model.

[0091] In this embodiment, when assessing the difficulty of samples, this application embodiment designs a complexity scoring system based on multi-dimensional features. In constructing the complexity scoring system based on multi-dimensional features, this application embodiment needs to comprehensively consider the following factors: phoneme complexity: based on the conditional entropy of phoneme combinations; acoustic variation: using DTW (Dynamic Time Warping, an algorithm for measuring the similarity between two time series) distance to measure the dynamic features of the audio segment; environmental complexity: quantified through acoustic parameters such as signal-to-noise ratio and reverberation time.

[0092] Specifically, the initial speech recognition model is trained using the acoustic feature consistency loss function and the contrastive learning loss function, based on the target speech training data, to obtain the target speech recognition model. This process may include: establishing a data scoring system based on phoneme complexity, acoustic variability, and environmental complexity; phoneme complexity includes the conditional entropy of phoneme combinations corresponding to the target speech training data; acoustic variability is the dynamic feature of the audio segment obtained after processing the target speech training data using a dynamic time warping algorithm; environmental complexity includes signal-to-noise ratio and reverberation time; determining the contrastive learning loss function based on the phoneme similarity corresponding to the target speech training data, and then using the data scoring system to score the target speech training data, obtaining a scoring result, and determining whether the scoring result is greater than a preset scoring threshold; if the scoring result is greater than the preset scoring threshold, the initial speech recognition model is trained using the acoustic feature consistency loss function, the contrastive learning loss function, and the joint loss function, based on the data corresponding to the scoring result, to obtain the target speech recognition model.

[0093] It is worth mentioning that the process of building a contrastive learning framework is as follows: Figure 3 As shown in the embodiments of this application, a hierarchical feature comparison strategy is proposed. At the phoneme level, a contrastive learning loss based on phoneme similarity is constructed; at the word level, a semantically guided negative sample selection mechanism is introduced; and at the sentence level, a contrastive learning objective based on context consistency is designed. This multi-level contrastive learning framework can more comprehensively capture all levels of speech features, thereby improving the robustness and generalization ability of the model.

[0094] It is worth mentioning that the embodiments of this application not only achieve effective knowledge transfer, but also significantly improve the model's performance in keyword recognition. Among them, Figure 4 This is a diagram showing the results obtained from testing using test data and corresponding to several evaluation indicators. Figure 5 This is a diagram illustrating the accuracy of recognition tests using proper nouns, corresponding to several scene types.

[0095] As can be seen from the above, in the process of training the speech recognition model in this embodiment, the initial audio signal is first processed using a preset frequency domain modulation method and a preset time domain stretching method to obtain the target feature sequence. Then, the target feature sequence is processed based on acoustic feature consistency constraints, teacher-student contrast constraints, and recognition accuracy constraints to obtain initial speech training data. Subsequently, a speech synthesis center is established based on several TTS models to process the initial speech training data, obtaining speech training data to be processed. A preset acoustic scene simulation system is then used to process the speech training data to be processed, obtaining speech training data to be enhanced. Next, an attention enhancement mechanism based on multi-granularity features is used to perform keyword recognition enhancement processing on the speech training data to be enhanced at the word level, obtaining word-enhanced speech training data. Then, a contrastive learning framework is used to perform sentence recognition enhancement processing on the word-enhanced speech training data at the sentence level, obtaining the target speech training data. Finally, the initial speech recognition model is trained using the acoustic feature consistency loss function, the contrastive learning loss function, and the joint loss function, based on the target speech training data, to obtain the target speech recognition model. This improves the efficiency of training the speech recognition model, thereby increasing the accuracy of speech recognition.

[0096] Accordingly, see Figure 6 As shown, this application also provides a speech recognition model training device, including:

[0097] The feature sequence determination module 11 is used to process the initial audio signal using a preset frequency domain modulation method and a preset time domain stretching method to obtain a target feature sequence, and to process the target feature sequence based on acoustic feature consistency constraints, recognition accuracy constraints, and teacher-student comparison constraints to obtain initial speech training data; the acoustic feature consistency constraints are used to constrain the consistency of the acoustic features of the target feature sequence; the teacher-student comparison constraints are used to constrain the process of training the initial speech recognition model based on the target teacher-student comparison results, and the target teacher-student comparison results are the results obtained by comparing the outputs of the teacher model and the student model;

[0098] The training data processing module 12 is used to establish a speech synthesis center based on several TTS models, and to process the initial speech training data using the speech synthesis center to obtain speech training data to be processed, and to process the speech training data to be processed using a preset acoustic scene simulation system to obtain speech training data to be enhanced.

[0099] The training data determination module 13 is used to perform keyword recognition enhancement processing on the speech training data to be enhanced at the word level using an attention enhancement mechanism based on multi-granularity features to obtain word-enhanced speech training data. Then, it uses a contrastive learning framework to perform sentence recognition enhancement processing on the word-enhanced speech training data at the sentence level to obtain target speech training data.

[0100] The speech recognition model training module 14 is used to train the initial speech recognition model based on the target speech training data using the acoustic feature consistency loss function and the contrastive learning loss function to obtain the target speech recognition model.

[0101] As can be seen from the above, in the process of training the speech recognition model in this embodiment, the initial audio signal is first processed using a preset frequency domain modulation method and a preset time domain stretching method to obtain the target feature sequence. Then, the target feature sequence is processed based on acoustic feature consistency constraints, teacher-student contrast constraints, and recognition accuracy constraints to obtain initial speech training data. Subsequently, a speech synthesis center is established based on several TTS models to process the initial speech training data, obtaining speech training data to be processed. A preset acoustic scene simulation system is then used to process the speech training data to be processed, obtaining speech training data to be enhanced. Next, an attention enhancement mechanism based on multi-granularity features is used to perform keyword recognition enhancement processing on the speech training data to be enhanced at the word level, obtaining word-enhanced speech training data. Then, a contrastive learning framework is used to perform sentence recognition enhancement processing on the word-enhanced speech training data at the sentence level, obtaining the target speech training data. Finally, the initial speech recognition model is trained using the acoustic feature consistency loss function, the contrastive learning loss function, and the joint loss function, based on the target speech training data, to obtain the target speech recognition model. This improves the efficiency of training the speech recognition model, thereby increasing the accuracy of speech recognition.

[0102] In some specific embodiments, the feature sequence determination module 11 may specifically include:

[0103] The first audio signal processing unit is used to establish a contrast-consistent distillation framework that includes several dimensional constraint mechanisms.

[0104] The second audio signal processing utilizes the contrast-consistency distillation framework and performs frequency domain conversion on the initial audio signal based on short-time Fourier transform technology to obtain a first audio signal to be processed. The first audio signal to be processed is then processed using phase vocoder technology and a preset pitch preservation rule to obtain a second audio signal to be processed.

[0105] The first feature sequence determination unit is used to extract features from the second audio signal to be processed using the contrast consistency distillation framework and based on a preset window overlap rate and several time window sizes to obtain corresponding audio features, and to establish a feature sequence to be processed based on each audio feature; the feature sequence to be processed includes local speech information and long-term acoustic structure information.

[0106] The second feature sequence determination unit is used to perform time alignment operation on the dynamic time warping algorithm using a preset attention mechanism to obtain the target time warping algorithm, and to process the feature sequence to be processed using the target time warping algorithm to obtain the target feature sequence.

[0107] In some specific embodiments, the feature sequence determination module 11 may specifically include:

[0108] The feature extractor building unit is used to build feature extractors for the teacher model and the student model respectively using preset feature extractor building rules, so as to obtain the corresponding first feature extractor and second feature extractor.

[0109] The first feature sequence processing unit is used to utilize the first feature extractor, the second feature extractor and the dynamic time warping algorithm, and determine the acoustic feature consistency constraint based on a preset balance factor, and process the target feature sequence using the acoustic feature consistency constraint to obtain the first feature sequence;

[0110] The positive sample pair determination unit is used to input the first feature sequence into the teacher model and the student model respectively, to obtain the first feature extraction result corresponding to the teacher model and the second feature extraction result corresponding to the student model, and to set the first feature extraction result and the second feature extraction result as a positive sample pair;

[0111] The student model processing unit is used to process the student model using the teacher-student comparison constraints and based on the first feature extraction result and the second feature extraction result to obtain an initial speech recognition model.

[0112] The difficult-to-bear sample determination unit is used to calculate the similarity of each of the first feature sequences based on the cosine similarity function, obtain the corresponding similarity results, and determine the audio signals in the similarity results that are not less than a preset similarity threshold and correspond to different audio types as difficult-to-bear samples;

[0113] The second feature sequence processing unit is used to determine the second feature sequence by utilizing the cosine similarity function and based on the positive sample pairs, the hard negative samples and the preset temperature parameters.

[0114] The third feature sequence processing unit is used to process the second feature sequence using the attention decoder to obtain the cross-entropy loss, and then use the CTC loss function and based on the preset regularization term, preset weight coefficient and the cross-entropy loss to process the second feature sequence to obtain the initial speech training data.

[0115] In some specific embodiments, the training data processing module 12 may specifically include:

[0116] The speech synthesis center establishment unit is used to establish a speech synthesis center based on the FastSpeech2 model with a non-autoregressive architecture, the Tacotron2 model with a sequence-to-sequence architecture, and the VITS model with an end-to-end conditional variational autoencoder structure.

[0117] The phoneme boundary information determination unit is used to perform phoneme analysis on the initial speech training data, obtain phoneme analysis results, and process the phoneme analysis results using a preset audio text alignment tool to obtain phoneme boundary information.

[0118] The training data synthesis unit is used to determine a speech synthesis model from the speech synthesis center based on the acoustic features corresponding to the phoneme boundary information, so as to use the speech synthesis model to synthesize the initial speech training data to obtain the speech training data to be processed; the acoustic features include vowels, consonants, phoneme duration and pitch contour.

[0119] In some specific embodiments, the training data processing module 12 may specifically include:

[0120] An acoustic feature library establishment unit is used to establish an acoustic feature library based on acoustic parameter description information corresponding to several acoustic scenarios; the acoustic parameter description information includes room impulse response information, signal-to-noise ratio distribution information, and reverberation time;

[0121] The first feature adjustment unit is used to determine whether the current ambient noise is transient noise. If the current ambient noise is transient noise, the preset acoustic physical model and the triggering mechanism based on energy envelope are used to perform time-frequency feature adjustment operation on the speech training data to be processed based on the corresponding room impulse response and time-varying mixing coefficient to obtain the speech training data to be enhanced.

[0122] The second feature adjustment unit is used to adjust the time-frequency features of the speech training data to be processed based on the current ambient noise, the room impulse response, and the time-varying mixing coefficients, using the preset acoustic physical model and preset mixing strategy if the current ambient noise is not transient noise, to obtain the speech training data to be enhanced.

[0123] In some specific embodiments, the training data determination module 13 may specifically include:

[0124] The first recognition and enhancement processing unit is used to establish an attention enhancement mechanism based on multi-granularity features, and to use the attention enhancement mechanism to perform keyword recognition enhancement processing on the speech training data to be enhanced based on word level to obtain word-enhanced speech training data.

[0125] The dependency determination unit is used to determine the context dependency of the word-enhanced speech training data by using a preset multi-head self-attention mechanism and based on a preset position encoding, so as to obtain the context dependency.

[0126] The second recognition enhancement processing unit is used to perform sentence recognition enhancement processing on the word-enhanced speech training data using a contrastive learning framework and based on the sentence level and the contextual dependency relationship, to obtain the target speech training data.

[0127] In some specific embodiments, the speech recognition model training module 14 may specifically include:

[0128] A data scoring system establishment unit is used to establish a data scoring system based on phoneme complexity, acoustic variability, and environmental complexity; the phoneme complexity includes the conditional entropy of the phoneme combination corresponding to the target speech training data; the acoustic variability is the dynamic feature of the audio segment obtained after processing the target speech training data based on the dynamic time warping algorithm; the environmental complexity includes signal-to-noise ratio and reverberation time.

[0129] The scoring result judgment unit is used to determine the contrastive learning loss function based on the phoneme similarity corresponding to the target speech training data, then use the data scoring system to score the target speech training data, obtain the scoring result, and determine whether the scoring result is greater than a preset scoring threshold.

[0130] The speech recognition model training subunit is used to train the initial speech recognition model based on the data corresponding to the score result if the score result is greater than the preset score threshold, using the acoustic feature consistency loss function, the contrastive learning loss function and the joint loss function, to obtain the target speech recognition model.

[0131] Furthermore, embodiments of this application also disclose an electronic device, Figure 7This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the speech recognition model training method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0132] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0133] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0134] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the speech recognition model training method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0135] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed speech recognition model training method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0136] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0137] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0138] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0139] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0140] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for training a speech recognition model, characterized in that, include: The initial audio signal is processed using a preset frequency domain modulation method and a preset time domain stretching method to obtain a target feature sequence. This target feature sequence is then processed based on acoustic feature consistency constraints, recognition accuracy constraints, and teacher-student comparison constraints to obtain initial speech training data. The acoustic feature consistency constraints are used to constrain the consistency of the acoustic features of the target feature sequence. The teacher-student comparison constraints are used to constrain the process of training the initial speech recognition model based on the target teacher-student comparison results. The target teacher-student comparison results are obtained by comparing the outputs of the teacher model and the student model. A speech synthesis center is established based on several TTS models. The initial speech training data is processed using the speech synthesis center to obtain speech training data to be processed. The speech training data to be processed is then processed using a preset acoustic scene simulation system to obtain speech training data to be enhanced. The attention enhancement mechanism based on multi-granularity features is used to perform keyword recognition enhancement processing on the speech training data to be enhanced at the word level to obtain word-enhanced speech training data. Then, the contrastive learning framework is used to perform sentence recognition enhancement processing on the word-enhanced speech training data at the sentence level to obtain target speech training data. The initial speech recognition model is trained using the acoustic feature consistency loss function and the contrastive learning loss function, based on the target speech training data, to obtain the target speech recognition model.

2. The speech recognition model training method according to claim 1, characterized in that, The process of processing the initial audio signal using a preset frequency domain modulation method and a preset time domain stretching method to obtain the target feature sequence includes: Establish a comparative consistency distillation framework that includes several dimensional constraint mechanisms; The initial audio signal is frequency-domain converted using the contrast-consistent distillation framework and based on short-time Fourier transform technology to obtain a first audio signal to be processed. The first audio signal to be processed is then processed using phase vocoder technology and a preset pitch preservation rule to obtain a second audio signal to be processed. The second audio signal to be processed is used to extract features based on the contrast consistency distillation framework and a preset window overlap rate and several time window sizes to obtain corresponding audio features, and a feature sequence to be processed is established based on each audio feature; the feature sequence to be processed includes local speech information and long-term acoustic structure information. A preset attention mechanism is used to perform time alignment operation on the dynamic time warping algorithm to obtain a target time warping algorithm. The target time warping algorithm is then used to process the feature sequence to be processed to obtain a target feature sequence.

3. The speech recognition model training method according to claim 2, characterized in that, The target feature sequence is processed based on acoustic feature consistency constraints, recognition accuracy constraints, and teacher-student comparison constraints to obtain initial speech training data, including: Using preset feature extractor establishment rules, feature extractors are established for the teacher model and the student model respectively, resulting in a first feature extractor and a second feature extractor. The first feature extractor, the second feature extractor, and the dynamic time warping algorithm are used, and the acoustic feature consistency constraint is determined based on a preset balance factor. The target feature sequence is then processed using the acoustic feature consistency constraint to obtain the first feature sequence. The first feature sequence is input into the teacher model and the student model respectively to obtain the first feature extraction result corresponding to the teacher model and the second feature extraction result corresponding to the student model, and the first feature extraction result and the second feature extraction result are set as positive sample pairs; The student model is processed using the teacher-student comparison constraints and based on the first feature extraction result and the second feature extraction result to obtain an initial speech recognition model; The similarity of each of the first feature sequences is calculated based on the cosine similarity function to obtain the corresponding similarity results. Audio signals with a similarity of not less than a preset similarity threshold and corresponding to different audio types in the similarity results are identified as hard negative samples. The second feature sequence is determined using the cosine similarity function and based on the positive sample pairs, the hard negative samples, and the preset temperature parameters. The second feature sequence is processed using an attention decoder to obtain cross-entropy loss. Then, the second feature sequence is processed using the CTC loss function based on a preset regularization term, a preset weight coefficient, and the cross-entropy loss to obtain initial speech training data.

4. The speech recognition model training method according to claim 1, characterized in that, The establishment of a speech synthesis center based on several TTS models, and the use of the speech synthesis center to process the initial speech training data to obtain the speech training data to be processed, includes: A speech synthesis center is established based on the FastSpeech2 model with a non-autoregressive architecture, the Tacotron2 model with a sequence-to-sequence architecture, and the VITS model with an end-to-end conditional variational autoencoder structure. Phoneme analysis is performed on the initial speech training data to obtain phoneme analysis results, and the phoneme analysis results are processed using a preset audio-text alignment tool to obtain phoneme boundary information; Based on the acoustic features corresponding to the phoneme boundary information, a speech synthesis model is determined from the speech synthesis center, and the initial speech training data is synthesized using the speech synthesis model to obtain the speech training data to be processed; the acoustic features include vowels, consonants, phoneme duration, and pitch contours.

5. The speech recognition model training method according to claim 1, characterized in that, The process of using a preset acoustic scene simulation system to process the speech training data to be processed to obtain speech training data to be enhanced includes: An acoustic feature library is established based on acoustic parameter description information corresponding to several acoustic scenarios; the acoustic parameter description information includes room impulse response information, signal-to-noise ratio distribution information, and reverberation time. Determine whether the current ambient noise is transient noise. If the current ambient noise is transient noise, then use a preset acoustic physical model and an energy envelope-based triggering mechanism, and perform time-frequency feature adjustment on the speech training data to be processed based on the corresponding room impulse response and time-varying mixing coefficient to obtain the speech training data to be enhanced. If the current ambient noise is not transient noise, then the preset acoustic physical model and preset mixing strategy are used, and the time-frequency feature adjustment operation is performed on the speech training data to be processed based on the current ambient noise, the room impulse response and the time-varying mixing coefficient to obtain the speech training data to be enhanced.

6. The speech recognition model training method according to claim 1, characterized in that, The attention enhancement mechanism based on multi-granularity features performs keyword recognition enhancement processing on the speech training data to be enhanced at the word level to obtain word-enhanced speech training data. Then, a contrastive learning framework is used to perform sentence recognition enhancement processing on the word-enhanced speech training data at the sentence level to obtain target speech training data, including: An attention enhancement mechanism is established based on multi-granularity features. The attention enhancement mechanism is then used to perform keyword recognition enhancement processing on the speech training data to be enhanced based on word level, so as to obtain word-enhanced speech training data. The context dependencies are determined by using a preset multi-head self-attention mechanism and based on a preset positional encoding to obtain the context dependencies of the word-enhanced speech training data. The target speech training data is obtained by using a contrastive learning framework and performing sentence recognition enhancement processing on the word-enhanced speech training data based on sentence level and the context dependency relationship.

7. The speech recognition model training method according to any one of claims 1 to 6, characterized in that, The step of using an acoustic feature consistency loss function and a contrastive learning loss function, and training the initial speech recognition model based on the target speech training data to obtain the target speech recognition model includes: A data scoring system is established based on phoneme complexity, acoustic variability, and environmental complexity. The phoneme complexity includes the conditional entropy of the phoneme combination corresponding to the target speech training data. The acoustic variability is the dynamic feature of the audio segment obtained after processing the target speech training data based on the dynamic time warping algorithm. The environmental complexity includes the signal-to-noise ratio and reverberation time. A contrastive learning loss function is determined based on the phoneme similarity to the target speech training data. Then, the target speech training data is scored using the data scoring system to obtain a scoring result, and it is determined whether the scoring result is greater than a preset scoring threshold. If the score result is greater than the preset score threshold, the initial speech recognition model is trained using the acoustic feature consistency loss function, the contrastive learning loss function, and the joint loss function, based on the data corresponding to the score result, to obtain the target speech recognition model.

8. A speech recognition model training device, characterized in that, include: The feature sequence determination module processes the initial audio signal using a preset frequency domain modulation method and a preset time domain stretching method to obtain a target feature sequence. It then processes the target feature sequence based on acoustic feature consistency constraints, recognition accuracy constraints, and teacher-student comparison constraints to obtain initial speech training data. The acoustic feature consistency constraints constrain the consistency of the acoustic features of the target feature sequence. The teacher-student comparison constraints constrain the process of training the initial speech recognition model based on the target teacher-student comparison results, where the target teacher-student comparison results are obtained by comparing the outputs of the teacher model and the student model. The training data processing module is used to establish a speech synthesis center based on several TTS models, and to use the speech synthesis center to process the initial speech training data to obtain speech training data to be processed, and to use a preset acoustic scene simulation system to process the speech training data to be processed to obtain speech training data to be enhanced. The training data determination module is used to perform keyword recognition enhancement processing on the speech training data to be enhanced at the word level using an attention enhancement mechanism based on multi-granularity features to obtain word-enhanced speech training data. Then, it uses a contrastive learning framework to perform sentence recognition enhancement processing on the word-enhanced speech training data at the sentence level to obtain target speech training data. The speech recognition model training module is used to train the initial speech recognition model based on the target speech training data using the acoustic feature consistency loss function and the contrastive learning loss function to obtain the target speech recognition model.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the speech recognition model training method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the speech recognition model training method as described in any one of claims 1 to 7.

Images