Model training method and device, speech recognition method and device, equipment and storage medium

Abstract

The invention discloses a model training method and device, a speech recognition method and device, equipment and a storage medium, and the method comprises the steps: obtaining a synthetic speech data set and a real speech data set, and constructing a speech recognition model which comprises an encoder, an adaption module and a decoder; pre-training an encoder and a decoder by using the synthesized voice data set; and freezing the pre-trained decoder parameters, and finely adjusting the encoder parameters and the adaptive module parameters by using the real voice data set. The method effectively solves the problem of domain offset between the synthetic speech and the real speech, remarkably improves the model performance under the condition of a small amount of real speech data, prevents catastrophic forgetting by freezing the decoder, maintains the universality of the model, is high in generalization ability, and is suitable for speech recognition application in a low-resource scene.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a model training method, speech recognition method, device, equipment and storage medium. Background Technology

[0002] With the development of deep learning technology, end-to-end speech recognition models have achieved remarkable results in various general scenarios. However, in specific professional fields (such as medicine, law, and industry), obtaining large amounts of high-quality labeled real speech data is costly and time-consuming, which seriously restricts the application of speech recognition technology in these fields.

[0003] The current main solution is to use text-to-speech (TTS) technology to generate synthetic speech data to expand the training set. However, there are significant differences in acoustic characteristics between synthetic speech and real speech (known as domain shift), which leads to a decrease in the model's recognition performance in real-world scenarios. While traditional fine-tuning methods can partially alleviate this problem, they often cause the model to forget the general knowledge learned from the original data when adapting to new domains; this phenomenon is known as catastrophic forgetting.

[0004] While there are some domain-adaptive methods in the existing technology, most of them require a large amount of target domain data or cannot effectively solve the problem of simultaneous domain shift and catastrophic forgetting. Summary of the Invention

[0005] In view of this, this application provides a model training method, a speech recognition method, an apparatus, a device, and a storage medium to address the problem of how to effectively utilize synthetic speech data and overcome domain shift while preventing catastrophic forgetting when real labeled speech data is scarce.

[0006] To address the aforementioned technical problems, this application provides a model training method comprising: acquiring a synthetic speech dataset and a real speech dataset, and constructing a speech recognition model, the speech recognition model including an encoder, an adaptation module, and a decoder; pre-training the encoder and decoder using the synthetic speech dataset; freezing the parameters of the pre-trained decoder; fine-tuning the parameters of the pre-trained encoder and the adaptation module using the real speech dataset, the adaptation module being used to transform the features extracted by the encoder so that the transformed feature distribution is compatible with the input feature distribution expected by the pre-trained decoder.

[0007] As a further improvement to this application, after obtaining the synthesized speech dataset, the method further includes: performing acoustic enhancement processing on the speech data in the synthesized speech dataset.

[0008] As a further improvement to this application, the adaptation module includes an adapter, a domain discriminator, and a gradient inversion layer; fine-tuning the parameters of the pre-trained encoder and the adaptation module using real speech datasets includes: constructing a sample dataset using synthetic speech datasets and real speech datasets; and performing adversarial training on the encoder, adapter, and domain discriminator using the sample datasets.

[0009] As a further improvement of this application, adversarial training is performed on the encoder, adapter, and domain discriminator using a sample dataset, including: inputting sample data from the sample dataset into the encoder for feature extraction to obtain initial features; inputting the initial features into the adapter for feature transformation to obtain target features, and simultaneously inputting the initial features into the domain discriminator, which is used to classify whether the initial features originate from real speech data or synthetic speech data, and calculates the adversarial loss function value based on the classification result; inputting the target features into the decoder for decoding to obtain sample recognition results, and calculating the speech recognition loss function value based on the sample recognition results; inputting the adversarial loss function value into the gradient inversion layer for inversion, and then combining it with the speech recognition loss function value to calculate the total loss function value; updating the encoder and adapter through backpropagation based on the total loss function value, and updating the domain discriminator through backpropagation based on the adversarial loss function value.

[0010] As a further improvement to this application, the total loss function is expressed as: ; in, This represents the total loss function value. This represents the value of the speech recognition loss function. This represents the adversarial loss function value. This represents the weighting coefficient.

[0011] As a further improvement of this application, the weight coefficients are adaptively adjusted based on the classification accuracy of the domain discriminator, and their calculation formula is expressed as follows: ; in, Time indicates the first Iterative training, Indicates basic parameters, Indicates the first The classification accuracy of the domain discriminator during the next iteration of training.

[0012] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide a speech recognition method, which includes: receiving real speech data to be recognized; inputting the real speech data to be recognized into a speech recognition model trained by one of the above-mentioned model training methods for recognition, and obtaining real speech recognition text.

[0013] To address the aforementioned technical problems, another technical solution adopted in this application is: providing a model training device, comprising: an acquisition module for acquiring a synthetic speech dataset and a real speech dataset, and constructing a speech recognition model, the speech recognition model including an encoder, an adaptation module, and a decoder; a pre-training module for pre-training the encoder and decoder using the synthetic speech dataset; and a fine-tuning module for freezing the parameters of the pre-trained decoder and fine-tuning the parameters of the pre-trained encoder and the adaptation module using the real speech dataset, the adaptation module being used to transform the features extracted by the encoder so that the transformed feature distribution is compatible with the input feature distribution expected by the pre-trained decoder.

[0014] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide a speech recognition device, which includes: a receiving module for receiving real speech data to be recognized; and a recognition module for inputting the real speech data to be recognized into a speech recognition model trained by one of the above-mentioned model training methods for recognition, so as to obtain real speech recognition text.

[0015] To solve the above-mentioned technical problems, another technical solution adopted in this application is: to provide a computer device, the computer device including a processor and a memory coupled to the processor, the memory storing program instructions, and when the program instructions are executed by the processor, causing the processor to perform the steps of the model training method or the steps of the speech recognition method described above.

[0016] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a storage medium storing program instructions capable of implementing the model training method or the speech recognition method described above.

[0017] The beneficial effects of this application are as follows: The model training method of this application introduces an adaptation module for adversarial training, forcing the encoder to learn to extract domain-invariant features, thereby aligning the distribution of real speech features with the synthetic distribution expected by the decoder during pre-training. This enables the decoder to accurately identify features from real speech data, improving recognition accuracy. Furthermore, by freezing the decoder and using real speech data to fine-tune the encoder and adaptation module, the method effectively prevents catastrophic forgetting of the decoder, protects the language model knowledge acquired by the decoder during pre-training, and ensures that the general performance of the model does not degrade when adapting to new domains. Moreover, the pre-training process of the encoder and decoder can use a large amount of synthetic speech data, while only a small amount of real speech data is needed to fine-tune the encoder and adaptation module to complete the model training process while ensuring the performance of the speech recognition model, greatly reducing the demand for real speech data. Attached Figure Description

[0018] Figure 1 This is a schematic flowchart of a model training method according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the structural principle of the speech recognition model in the model training method of this invention. Figure 3 This is a flowchart illustrating a speech recognition method according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the functional modules of the model training device according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the functional modules of the speech recognition device according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of the storage medium according to an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0020] The terms "first," "second," and "third" in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationships and movements between components in a specific orientation (as shown in the figures). If the specific orientation changes, the directional indications also change accordingly. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0021] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0022] Figure 1 This is a flowchart illustrating the model training method according to an embodiment of the present invention. It should be noted that if substantially the same result is obtained, the method of the present invention is not necessarily identical. Figure 1 The illustrated process sequence is limited. For example... Figure 1 As shown, the model training method includes the following steps: Step S101: Obtain the synthetic speech dataset and the real speech dataset, and construct a speech recognition model, which includes an encoder, an adapter module and a decoder.

[0023] In this embodiment, the synthesized speech dataset can be obtained in the following ways: (1) Text corpus preparation: Collect text data from professional documents, manuals, and dialogue records in the target field (such as medical, legal, and industrial fields). The text corpus should cover the professional terminology and common expressions in the field; clean and standardize the text, including: removing special characters, correcting spelling errors, and unifying the format of numbers and abbreviations (such as unifying "Chapter 1" as "Chapter 1"), to ensure text quality. (2) Speech synthesis (TTS): Use advanced neural network TTS systems (such as VALL-E and FastSpeech 2) for speech synthesis. To enhance the diversity of the data, different acoustic parameters need to be set during synthesis. For example: Voiceprint diversity: Use more than 200 different speaker voiceprint parameters, speech rate variation: Set a random speech rate between 0.8 and 1.2 times, pitch variation: The fundamental frequency (pitch) fluctuates randomly within the range of ±20% of the standard value. The output audio specifications are unified, such as: sampling rate 16kHz, mono, 16-bit quantization precision. (3) Data annotation: The text annotation of the synthesized speech is the original input text. Its alignment information can be automatically generated by the TTS system to form perfectly annotated training data pairs.

[0024] In this embodiment, the acquisition of real speech datasets needs to be targeted at the application scenario and can be obtained in the following ways: (1) Data collection: Record real speech data in the target environment (such as hospital clinic, factory workshop). Use a variety of recording devices (such as close-range microphones, conference microphones, mobile phones, etc.) to simulate different sound qualities. The recording content needs to be related to the text domain of the synthesized corpus, but should be natural dialogue or reading aloud, rather than mechanically reading the text used for synthesis, in order to ensure the authenticity of the data. (2) Data annotation and cleaning: Manually transcribe or semi-automatically (first identify through the ASR system, then manually proofread) annotate the recorded speech to generate corresponding text tags. Perform quality checks on the audio and remove samples with too low signal-to-noise ratio, unclear speech or too much background noise.

[0025] In this embodiment, the speech recognition model adopts an encoder-decoder framework and reserves interfaces for adaptation modules. The encoder maps the input audio sequence X = (x1, x2, ..., xT) to a high-level acoustic feature sequence H_enc = (h1, h2, ..., hT'). This encoder preferably uses a Conformer structure, which combines the self-attention mechanism of the Transformer with the local feature extraction capabilities of CNNs, enabling more effective modeling of long-term dependencies and local patterns in audio. A typical configuration includes 12 Conformer layers, a model dimension of 256, and 4 attention heads. The decoder converts the feature sequence H_enc output by the encoder into a text sequence Y = (y1, y2, ..., yL). This decoder can be an autoregressive Transformer decoder or a CTC-based classifier. The Transformer decoder typically contains 6 layers and uses masked self-attention to ensure that prediction relies only on the generated output. The adapter module connects between the encoder output and the decoder input. Its core function is to transform the features H_enc extracted by the encoder to generate the target features H_adapt, so as to bridge the domain offset between the synthesized speech data and the real speech data, and make the distribution of H_adapt compatible with the input distribution expected by the pre-trained decoder.

[0026] Furthermore, in order to broaden the acoustic distribution of the synthesized speech data and enable it to cover a wider range of situations that may be encountered in the real world, in some embodiments, after the step of obtaining the synthesized speech dataset, the method further includes: performing acoustic enhancement processing on the speech data in the synthesized speech dataset.

[0027] Specifically, the speech generated by the original TTS is often acoustically overly idealized, lacking the various perturbations of the real environment, resulting in a significant difference in its acoustic distribution compared to real speech data (i.e., "domain offset"). Therefore, in this embodiment, after obtaining the synthesized speech dataset, acoustic enhancement of the speech data in the synthesized speech dataset is also required. This can be achieved through background noise injection, spatial reverberation simulation, volume and dynamic changes, etc. For example: using a volume mixing method, the noise signal is superimposed on the clean speech with a specific signal-to-noise ratio (SNR); using convolution operations, the clean speech is convolved with the room impulse response (RIR); all audio peaks are first normalized to -3dBFS, and then a random gain factor (such as [-6dB, +6dB]) is applied to simulate the recording level differences of different devices.

[0028] Preferably, in this embodiment, random enhancements are performed on batches of the loaded synthetic speech data in real time within each training set during the pre-training phase. This means that the same speech will appear with different enhancement variants in different epochs, greatly improving data utilization efficiency and diversity.

[0029] This embodiment simulates the acoustic characteristics of a real environment, artificially creating a diverse and highly comprehensive acoustic environment set. This makes the training samples of the pre-trained model more comprehensive, thereby learning more robust feature representations and reducing the adaptation difficulty of subsequent adaptation modules.

[0030] Step S102: Pre-train the encoder and decoder using the synthetic speech dataset.

[0031] Specifically, after obtaining the synthetic speech dataset, this embodiment adopts an end-to-end supervised learning approach to jointly train the encoder and decoder, optimizing the function of the speech recognition model itself. The goal is to train a powerful speech recognition model (Base Model) using massive, high-quality, and perfectly labeled synthetic speech data. This model needs to learn to extract meaningful acoustic features from audio signals and accurately map them into text sequences. In the pre-training stage, there is no need to consider the domain offset problem. Therefore, the pre-training stage can be trained entirely using the synthetic speech dataset, or it can be trained using a dataset that mixes synthetic speech datasets and real speech datasets.

[0032] Step S103: Freeze the parameters of the pre-trained decoder, and fine-tune the parameters of the pre-trained encoder and the adaptation module using a real speech dataset. The adaptation module is used to transform the features extracted by the encoder so that the transformed feature distribution is compatible with the input feature distribution expected by the pre-trained decoder.

[0033] It's important to understand that during the pre-training phase, a large amount of synthetic speech data is used as sample data to train the speech synthesis model. Therefore, the pre-trained model possesses strong recognition capabilities for synthesized speech, but lacks training on real speech data. Therefore, this embodiment freezes the decoder parameters after the pre-training phase and then fine-tunes the encoder and adaptation module parameters using a real speech dataset. This allows the model to adapt to the acoustic characteristics of the target domain while preserving the general knowledge of the pre-trained model (especially the decoder) to the maximum extent possible, using a small amount of real speech data. Specifically, freezing the decoder prevents catastrophic forgetting. The decoder carries the language model and acoustic-language mapping knowledge learned from massive amounts of synthesized speech data; freezing the decoder parameters completely protects this knowledge from being overwritten or destroyed by a small amount of real speech data. Fine-tuning the encoder allows it to learn how to extract more effective features from audio in the target domain. The core function of the adaptation module is to perform feature distribution transformation. By fine-tuning and learning a mapping function, it maps the feature distribution extracted by the encoder after fine-tuning the real speech data to a space compatible with the feature distribution seen by the pre-trained decoder on the synthesized speech data. The frozen decoder can efficiently process features from real speech data without any modification, thus achieving high-performance domain adaptation.

[0034] For further details, please refer to the following: Figure 2 The adaptation module includes an adapter, a neighborhood discriminator, and a gradient inversion layer.

[0035] Specifically, the adapter is responsible for performing nonlinear transformations on the features output by the encoder, aiming to "translate" the feature distribution of the real speech data to the distribution space desired by the pre-trained decoder. In this embodiment, the adapter adopts a bottleneck architecture, which includes: a dimensionality reduction layer: a linear layer or a one-dimensional convolution (Conv1D) that projects the high-dimensional features of the input to a low-dimensional space; a nonlinear layer: using the ReLU activation function to introduce nonlinear transformation capability; a dimensionality increase layer: another linear layer or a transposed one-dimensional convolution that restores the feature dimension from the low-dimensional features to the high-dimensional features; and a residual connection: the final output of the adapter is residually connected to the original encoder output features to ensure that even in the worst case, the adapter can degenerate into an identity mapping, guaranteeing training stability and preventing performance degradation. The Gradient Reversal Layer (GRL) is a key component for implementing adversarial training. Its behavior is asymmetric in forward and backward propagation: in forward propagation, the GRL acts as an identity mapping, GRL(X) = X, directly passing the encoder's output features to the downstream domain discriminator without modification; in backward propagation, it acts as a gradient inverter, multiplying the gradients fed back from the domain discriminator by a negative scaling factor before passing them to the encoder and adapter. The Domain Discriminator is a binary classifier used to determine whether input features originate from real or synthetic speech data. Its structure is a lightweight neural network, for example, consisting of two to three fully connected layers, using ReLU activation functions and Dropout layers to prevent overfitting. Finally, it outputs a scalar probability value D(x) ∈ [0, 1] (where 0 represents synthetic speech data and 1 represents real speech data) through a Sigmoid activation function, representing the confidence that the input features originate from real speech data.

[0036] Step S103, which involves fine-tuning the parameters of the pre-trained encoder and the adaptation module using a real speech dataset, specifically includes: 1. Construct a sample dataset using a synthetic speech dataset and a real speech dataset.

[0037] Specifically, a mixed batch sample dataset is constructed by sampling from both synthetic and real speech datasets. For example, a batch size of 64 contains 32 real speech samples and 32 synthetic speech samples. This provides a balanced mix of positive and negative samples for the domain discriminator while continuing to utilize synthetic speech data during the fine-tuning stage to prevent the encoder from forgetting the general features learned during pre-training.

[0038] 2. Use the sample dataset to perform adversarial training on the encoder, adapter, and domain discriminator.

[0039] Specifically, by using sample datasets to perform adversarial training on the encoder, adapter, and domain discriminator, the encoder and adapter are optimized to produce features that cannot be distinguished by the domain discriminator, thereby effectively bridging the domain offset between synthesized and real speech data, enabling the frozen pre-trained decoder to seamlessly process real speech data.

[0040] Furthermore, adversarial training is performed on the encoder, adapter, and domain discriminator using the sample dataset, including: 2.1 Input the sample data from the sample dataset into the encoder for feature extraction to obtain the initial features.

[0041] Specifically, a sample dataset containing real and synthesized speech data is input into a pre-trained encoder. The encoder encodes the audio sequences in the sample dataset and outputs an initial feature H_enc. This initial feature contains the semantic information of the audio and also includes the domain characteristics of its source (synthetic or real speech).

[0042] 2.2 Input the initial features into the adapter for feature transformation to obtain the target features. At the same time, input the initial features into the domain discriminator. The domain discriminator is used to classify whether the initial features come from real speech data or synthetic speech data, and calculate the adversarial loss function value based on the classification result.

[0043] Specifically, after obtaining the initial features, these features are simultaneously input to the adapter and the neighborhood discriminator. The adapter transforms the initial features H_enc to generate the target features H_adapt, bridging the domain offset between synthesized and real speech data and ensuring that the distribution of H_adapt is compatible with the input distribution desired by the pre-trained decoder. The process of inputting the initial features to the neighborhood discriminator involves feeding them into a gradient inversion layer, which then feeds them back into the neighborhood discriminator. The neighborhood discriminator determines the source of the input features and outputs a classification result. Then, the adversarial loss function is calculated based on the classification result. This adversarial loss function is calculated using the binary cross-entropy loss function, where the value is calculated for the first element in a batch. The calculation process for the adversarial loss function value of each sample is expressed as follows: ; in, Indicates the first The adversarial loss function value for each sample. This represents the real domain label of the sample (1 for real speech data and 0 for synthesized speech data). This represents the predicted probability value of the neighborhood discriminator. Indicates the first The initial features of each training sample are used to average the loss across all samples, yielding the total adversarial loss for that batch. .

[0044] 2.3 Input the target features into the decoder for decoding to obtain the sample recognition result, and calculate the speech recognition loss function value based on the sample recognition result.

[0045] Specifically, the decoder's predictions are compared with the sample's true text labels. For autoregressive decoders, the cross-entropy loss function is typically used: ; in, This represents the average speech recognition loss value for a training batch. N represents the size of the current training batch, which is N (audio, text) sample pairs. It is the sample index, which iterates through each sample in the batch (from 1 to N). Indicates the first The length of the real text label sequence of each sample (in tokens); It is a time step index that iterates through each token in the target text sequence (from 1 to ...). ); Indicates the first Each sample at time step The true target token (i.e., the first one in the real text) (a word or character) Indicates the first Each sample at time step All previously generated target token sequences; Indicates the first The feature representation obtained after the audio data of each sample is transformed by the encoder and adapter; This indicates that the decoder model, given the historical output, and input features Under the given conditions, predict the correct token. The probability of.

[0046] 2.4. After the adversarial loss function value is input into the gradient inversion layer for inversion, the total loss function value is calculated by combining it with the speech recognition loss function value.

[0047] The total loss function is expressed as: ; in, This represents the total loss function value. This represents the value of the speech recognition loss function. This represents the adversarial loss function value. This represents the weighting coefficient.

[0048] Furthermore, in some embodiments, fixed weight coefficients may not adapt to training dynamics. Therefore, in this embodiment, the weight coefficients are adaptively adjusted based on the classification accuracy of the domain discriminator, and the calculation formula is expressed as follows: ; in, Time indicates the first Weight coefficients during the next iteration of training Indicates basic parameters, Indicates the first The classification accuracy of the domain discriminator during the next iteration of training.

[0049] Specifically, the iterative process of the weighting coefficients is as follows: 1. Accuracy of the domain discriminator: in the first... After each iteration of training, calculate the classification accuracy of the domain discriminator in the current batch of sample data. For example, if the discriminator correctly classifies 80% of the samples in a batch, then = 0.8.

[0050] 2. Apply the adjustment formula: Calculate the λ value for the new round based on the weighting coefficient calculation formula: ; These are the base weight coefficients, which are preset hyperparameters (e.g., ...). = 0.5), This represents the adaptive adjustment factor.

[0051] when When the factor is 0.5 (the domain discriminator equals random guessing), the domain offset is 0, meaning the domain offset has been largely eliminated, and the adversarial task can be paused. = 0; when When the factor is 1.0 (the neighborhood discriminator is completely correct), the factor is 1, which means the domain offset is very severe and requires maximum strength ( = Conduct competitive training; when When the factor is between 0.5 and 1.0, it increases linearly from 0 to 1. It also increases linearly.

[0052] This embodiment forms a negative feedback loop through the adaptive adjustment strategy of the weight coefficients. If the domain discriminator is strong (high accuracy), λ is increased to strengthen the adversarial training intensity, forcing the encoder and adapter to work harder to learn domain-invariant features. As the capabilities of the encoder and adapter improve, the domain discriminator task becomes more difficult, the accuracy decreases, and λ decreases accordingly. Eventually, a balance is reached, the domain discriminator accuracy stabilizes at around 50%, λ approaches 0, and the training focus returns to the main speech recognition task, thus realizing the automated and intelligent adjustment of the adversarial intensity.

[0053] 2.5. The encoder and adapter are updated by backpropagation based on the total loss function value, and the domain discriminator is updated by backpropagation based on the adversarial loss function value.

[0054] The model training method in this embodiment introduces an adaptation module for adversarial training, forcing the encoder to learn and extract domain-invariant features. This aligns the distribution of real speech features with the synthetic distribution expected by the decoder during pre-training, enabling the decoder to accurately identify features from real speech data and improving recognition accuracy. Furthermore, by freezing the decoder and using real speech data to fine-tune the encoder and adaptation module, catastrophic forgetting by the decoder is effectively prevented, protecting the language model knowledge acquired by the decoder during pre-training and ensuring that the model's general performance does not degrade when adapting to new domains. Moreover, the pre-training process of the encoder and decoder can use a large amount of synthetic speech data, while only a small amount of real speech data is needed to fine-tune the encoder and adaptation module to complete the model training process while maintaining the performance of the speech recognition model, greatly reducing the need for real speech data.

[0055] Figure 3 This is a flowchart illustrating the speech recognition method according to an embodiment of the present invention. It should be noted that if substantially the same result is obtained, the method of the present invention is not necessarily identical. Figure 3 The illustrated process sequence is limited. For example... Figure 3 As shown, the speech recognition method includes the following steps: Step S201: Receive the real voice data to be recognized.

[0056] Step S202: Input the real speech data to be recognized into the speech recognition model trained by the model training method of one of the above embodiments for recognition, and obtain the real speech recognition text.

[0057] The speech recognition method in this embodiment uses the speech recognition model trained in the above embodiment to perform speech recognition, which can reduce the domain-specific information in the features, making the feature distribution extracted from real speech compatible with the feature distribution expected by the decoder, thereby enabling the decoder to process real speech features seamlessly and efficiently and accurately recognize real speech.

[0058] Figure 4 This is a schematic diagram of the functional modules of the model training device according to an embodiment of the present invention. Figure 4 As shown, the model training device 10 includes: an acquisition module 11, a pre-training module 12, and a fine-tuning module 13.

[0059] The acquisition module 11 is used to acquire synthetic speech datasets and real speech datasets, and to build a speech recognition model, which includes an encoder, an adaptation module and a decoder. Pre-training module 12 is used to pre-train the encoder and decoder using a synthetic speech dataset; The fine-tuning module 13 is used to freeze the parameters of the pre-trained decoder and fine-tune the parameters of the pre-trained encoder and the parameters of the adaptation module using a real speech dataset. The adaptation module is used to transform the features extracted by the encoder so that the transformed feature distribution is compatible with the input feature distribution expected by the pre-trained decoder.

[0060] Optionally, after the acquisition module 11 performs the step of acquiring the synthesized speech dataset, it is further configured to: perform acoustic enhancement processing on the speech data in the synthesized speech dataset.

[0061] Optionally, the adaptation module includes an adapter, a domain discriminator, and a gradient inversion layer; the fine-tuning module 13 performs the operation of fine-tuning the parameters of the pre-trained encoder and the parameters of the adaptation module using a real speech dataset, specifically including: constructing a sample dataset using a synthetic speech dataset and a real speech dataset; and performing adversarial training on the encoder, adapter, and domain discriminator using the sample dataset.

[0062] Optionally, the fine-tuning module 13 performs adversarial training on the encoder, adapter, and domain discriminator using a sample dataset. Specifically, this includes: inputting sample data from the sample dataset into the encoder for feature extraction to obtain initial features; inputting the initial features into the adapter for feature transformation to obtain target features, and simultaneously inputting the initial features into the domain discriminator, which classifies whether the initial features originate from real or synthesized speech data, and calculates the adversarial loss function value based on the classification result; inputting the target features into the decoder for decoding to obtain sample recognition results, and calculating the speech recognition loss function value based on the sample recognition results; inputting the adversarial loss function value into the gradient inversion layer for inversion, and then combining it with the speech recognition loss function value to calculate the total loss function value; updating the encoder and adapter through backpropagation based on the total loss function value, and updating the domain discriminator through backpropagation based on the adversarial loss function value.

[0063] Alternatively, the total loss function can be expressed as: ; in, This represents the total loss function value. This represents the value of the speech recognition loss function. This represents the adversarial loss function value. This represents the weighting coefficient.

[0064] Optionally, the weight coefficients are adaptively adjusted based on the classification accuracy of the domain discriminator, and their calculation formula is expressed as: ; in, Time indicates the first iterations of training, Indicates basic parameters, Indicates the first The classification accuracy of the domain discriminator during the next iteration of training.

[0065] For other details regarding the implementation techniques of each module in the model training device of the above embodiments, please refer to the description in the model training method of the above embodiments, which will not be repeated here.

[0066] Figure 5 This is a schematic diagram of the functional modules of the speech recognition device according to an embodiment of the present invention. Figure 5 As shown, the voice recognition device 20 includes a receiving module 21 and a recognition module 22.

[0067] The receiving module 21 is used to receive real voice data to be recognized.

[0068] The recognition module 22 is used to input the real speech data to be recognized into the speech recognition model trained by the model training method of one of the above embodiments for recognition, so as to obtain real speech recognition text.

[0069] For other details regarding the implementation techniques of each module in the speech recognition device of the above embodiments, please refer to the description of the speech recognition method in the above embodiments, which will not be repeated here.

[0070] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0071] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Figure 6 As shown, the computer device 30 includes a processor 31 and a memory 32 coupled to the processor 31. The memory 32 stores program instructions. When the program instructions are executed by the processor 31, the processor 31 performs the steps of the model training method or the speech recognition method described in any of the above embodiments.

[0072] The processor 31 can also be referred to as a Central Processing Unit (CPU). The processor 31 may be an integrated circuit chip with signal processing capabilities. The processor 31 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor.

[0073] See Figure 7 , Figure 7This is a schematic diagram of the structure of the storage medium according to an embodiment of the present invention. The storage medium of this embodiment stores program instructions 41 capable of implementing the above-described model training method or speech recognition method. These program instructions 41 can be stored in the storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or computer devices such as computers, servers, mobile phones, and tablets.

[0074] In the several embodiments provided in this application, it should be understood that the disclosed computer devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0075] Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A model training method, characterized in that, It includes: Acquire synthetic speech datasets and real speech datasets, and construct a speech recognition model, which includes an encoder, an adaptation module, and a decoder; The encoder and decoder are pre-trained using the synthesized speech dataset; The parameters of the pre-trained decoder are frozen, and the parameters of the pre-trained encoder and the adaptation module are fine-tuned using the real speech dataset. The adaptation module is used to transform the features extracted by the encoder so that the transformed feature distribution is compatible with the input feature distribution expected by the pre-trained decoder.

2. The model training method according to claim 1, characterized in that, After obtaining the synthesized speech dataset, the process also includes: The speech data in the synthesized speech dataset is subjected to acoustic enhancement processing.

3. The model training method according to claim 1, characterized in that, The adaptation module includes an adapter, a neighborhood discriminator, and a gradient inversion layer; The step of fine-tuning the parameters of the pre-trained encoder and the parameters of the adaptation module using the real speech dataset includes: A sample dataset is constructed using the synthesized speech dataset and the real speech dataset; The encoder, the adapter, and the domain discriminator are trained adversarially using the sample dataset.

4. The model training method according to claim 3, characterized in that, The adversarial training of the encoder, the adapter, and the domain discriminator using the sample dataset includes: The sample data in the sample dataset is input into the encoder for feature extraction to obtain initial features; The initial features are input into the adapter for feature transformation to obtain target features. At the same time, the initial features are input into the domain discriminator, which is used to classify whether the initial features are from real speech data or synthetic speech data, and calculate the adversarial loss function value based on the classification result. The target features are input into the decoder for decoding to obtain the sample recognition result, and the speech recognition loss function value is calculated based on the sample recognition result; After the adversarial loss function value is input into the gradient inversion layer for inversion, the total loss function value is calculated by combining it with the speech recognition loss function value. The encoder and the adapter are updated via backpropagation based on the total loss function value, and the domain discriminator is updated via backpropagation based on the adversarial loss function value.

5. The model training method according to claim 4, characterized in that, The total loss function is expressed as: ； in, This represents the total loss function value. This represents the value of the speech recognition loss function. This represents the adversarial loss function value. This represents the weighting coefficient.

6. The model training method according to claim 5, characterized in that, The weighting coefficients are adaptively adjusted based on the classification accuracy of the domain discriminator, and their calculation formula is expressed as follows: ； in, Time indicates the first iterations of training, Indicates basic parameters, Indicates the first The classification accuracy of the domain discriminator during the next iteration of training.

7. A speech recognition method, characterized in that, It includes: Receive real voice data to be recognized; The real speech data to be recognized is input into the speech recognition model trained by the model training method described in any one of claims 1-6 for recognition, and the real speech recognition text is obtained.

8. A model training device, characterized in that, It includes: An acquisition module is used to acquire synthetic speech datasets and real speech datasets, and to construct a speech recognition model, which includes an encoder, an adaptation module, and a decoder. A pre-training module is used to pre-train the encoder and the decoder using the synthesized speech dataset; The fine-tuning module is used to freeze the parameters of the pre-trained decoder and fine-tune the parameters of the pre-trained encoder and the parameters of the adaptation module using the real speech dataset. The adaptation module is used to transform the features extracted by the encoder so that the transformed feature distribution is compatible with the input feature distribution expected by the pre-trained decoder.

9. A voice recognition device, characterized in that, It includes: The receiving module is used to receive the real voice data to be recognized; The recognition module is used to input the real speech data to be recognized into the speech recognition model trained by the model training method according to any one of claims 1-6 for recognition, so as to obtain real speech recognition text.

10. A computer device, characterized in that, The computer device includes a processor and a memory coupled to the processor, the memory storing program instructions that, when executed by the processor, cause the processor to perform the steps of the model training method as described in any one of claims 1-6 or the steps of the speech recognition method as described in claim 7.

11. A storage medium, characterized in that, The system stores program instructions capable of implementing the model training method as described in any one of claims 1-6 or the speech recognition method as described in claim 7.

Images