Speech recognition model training method and speech recognition method

Abstract

The invention provides a training method of a speech recognition model and a speech recognition method, and the training method comprises the steps: extracting the speech features of label sample speech based on a speech coding network, decoding the speech features based on a decoding network, and obtaining the alignment information of the label sample speech; the coding module is used for extracting text features of label texts of the label sample voice based on the text coding network, performing feature alignment on the text features based on the alignment information to obtain aligned text features, and sharing parts of the voice coding network and the text coding network; performing parameter iteration on the voice coding network, the text coding network and the decoding network based on the voice features and the aligned text features; and constructing a speech recognition model based on the speech coding network and the speech decoding network which are completed by parameter iteration. According to the method provided by the invention, the generalization ability of the speech recognition model can be effectively improved through the application of the multi-modal training data and the coding module shared by the speech coding network and the text coding network.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a training method for a speech recognition model and a speech recognition method. Background Art

[0002] Unsupervised pre-training technology has been successfully introduced and applied to the field of speech recognition due to its ability to extract useful features from large amounts of unlabeled data, and has shown great potential in speech recognition in low-resource scenarios.

[0003] Unsupervised pre-training technology can use massive amounts of unlabeled speech data to train models to learn deep feature information in speech signals in an unsupervised manner. Subsequently, in the supervised training phase, a small amount of speech-labeled text pairs can be applied to fine-tune the pre-trained model to adapt to specific speech recognition tasks.

[0004] During the pre-training stage, feature extraction and model optimization are performed only through data from a single modality, namely speech. As a result, the speech recognition model finds it difficult to learn sufficient feature representations from the limited annotated text during fine-tuning, makes insufficient use of text information, has low language constraint capabilities, and suffers from serious generalization problems in low-resource scenarios. Summary of the invention

[0005] The present invention provides a speech recognition model training method and a speech recognition method, which are used to solve the defect of poor generalization of the speech recognition model in the related art.

[0006] The present invention provides a method for training a speech recognition model, comprising: Extracting speech features of the labeled sample speech based on a speech coding network, and decoding the speech features based on a decoding network to obtain alignment information of the labeled sample speech; Based on the text encoding network, extract the text features of the label text of the label sample speech, and perform feature alignment on the text features based on the alignment information to obtain aligned text features, and the speech encoding network and the text encoding network share a portion of the encoding module; Based on the speech features and the aligned text features, performing parameter iteration on the speech encoding network, the text encoding network and the decoding network; The speech recognition model is constructed based on the speech encoding network and the decoding network completed by parameter iteration.

[0007] According to a method for training a speech recognition model provided by the present invention, the method of performing parameter iteration on the speech encoding network, the text encoding network and the decoding network based on the speech features and the aligned text features comprises: Based on the difference between the speech feature and the aligned text feature, and the difference between the label text and the decoded text obtained by decoding the speech feature, parameter iteration is performed on the speech encoding network, the text encoding network and the decoding network.

[0008] According to a training method for a speech recognition model provided by the present invention, the method of performing parameter iteration on the speech encoding network, the text encoding network and the decoding network based on the speech features and the aligned text features further comprises: Extracting masked speech features of unlabeled sample speech based on the speech coding network, and extracting unmasked speech features of the unlabeled sample speech based on a speech backbone network in the speech coding network, wherein the speech backbone network is the part of the speech coding network other than the coding module; Based on the masked speech features and the unmasked speech features, parameter iteration is performed on the speech coding network.

[0009] According to a training method for a speech recognition model provided by the present invention, the method of performing parameter iteration on the speech encoding network, the text encoding network and the decoding network based on the speech features and the aligned text features further comprises: Extracting unlabeled text features of unlabeled sample text based on the text encoding network, and decoding the unlabeled text features based on the decoding network to obtain restored text; Based on the unlabeled sample text and the restored text, parameters of the text encoding network and the decoding network are iterated.

[0010] According to a method for training a speech recognition model provided by the present invention, the determination of the speech coding network includes: Get a pre-trained speech backbone network; The speech backbone network is cascaded with the coding module to obtain the speech coding network.

[0011] According to a method for training a speech recognition model provided by the present invention, the determination of the speech backbone network includes: Identify faculty and student networks; Extracting teacher speech features of unsupervised speech based on the teacher network, and extracting student speech features of masked speech based on the student network, wherein the masked speech is obtained by masking the unsupervised speech; Based on the teacher speech feature and the student speech feature, performing parameter iteration on the teacher network and the student network; The student network after parameter iteration is used as the speech backbone network.

[0012] According to a method for training a speech recognition model provided by the present invention, the step of performing parameter iteration on the teacher network and the student network based on the teacher speech feature and the student speech feature comprises: Based on the feature decoding network, the student speech feature is decoded to obtain the masked speech feature; Based on the difference between the masked speech feature and the teacher speech feature, parameter iteration is performed on the teacher network and the student network.

[0013] A method for training a speech recognition model according to the present invention further includes: The speech recognition model is fine-tuned based on the scene speech in the target scene and the label text of the scene speech.

[0014] The present invention also provides a speech recognition method, comprising: Get the speech to be recognized; The speech to be recognized is recognized based on a speech recognition model, wherein the speech recognition model is obtained based on the training method of the speech recognition model as described above.

[0015] The present invention also provides a training device for a speech recognition model, comprising: A speech encoding unit, configured to extract speech features of the labeled sample speech based on a speech encoding network, and decode the speech features based on a decoding network to obtain alignment information of the labeled sample speech; A text encoding unit, used for extracting text features of the label text of the label sample speech based on a text encoding network, and aligning the text features based on the alignment information to obtain aligned text features, and an encoding module of a shared part of the speech encoding network and the text encoding network; A parameter iteration unit, configured to perform parameter iteration on the speech encoding network, the text encoding network, and the decoding network based on the speech feature and the aligned text feature; The model building unit is used to build the speech recognition model based on the speech encoding network and decoding network completed by parameter iteration.

[0016] The present invention also provides a speech recognition device, comprising: An acquisition unit, used for acquiring speech to be recognized; The recognition unit is used to recognize the speech to be recognized based on a speech recognition model, wherein the speech recognition model is obtained based on the training method of the speech recognition model as described above.

[0017] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the training method of the speech recognition model or the speech recognition method as described above is implemented.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a training method for a speech recognition model or a speech recognition method as described above.

[0019] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned speech recognition model training methods or speech recognition methods.

[0020] The training method of the speech recognition model and the speech recognition method provided by the present invention apply labeled sample speech and labeled text of the labeled sample speech as multimodal training data. Through the application of multimodal training data and the encoding module shared by the speech encoding network and the text encoding network, the joint feature representation of the two modalities of speech and text can be learned during the training process, thereby improving the feature representation ability of the speech recognition model, and then effectively improving the generalization ability of the speech recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 It is a flow chart of the training method of the speech recognition model provided by the present invention.

[0023] Figure 2 It is a structural schematic diagram of the feature extraction network provided by the present invention.

[0024] Figure 3 It is a training diagram of the speech backbone network provided by the present invention.

[0025] Figure 4 It is a flowchart of the first pre-training stage of the speech recognition model provided by the present invention.

[0026] Figure 5 It is a flowchart of the second pre-training stage of the speech recognition model provided by the present invention.

[0027] Figure 6It is a flow chart of the speech recognition method provided by the present invention.

[0028] Figure 7 It is a structural schematic diagram of the training device of the speech recognition model provided by the present invention.

[0029] Figure 8 It is a structural schematic diagram of the speech recognition device provided by the present invention.

[0030] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] The problem of speech recognition in multilingual and low-resource scenarios is an important challenge facing current speech recognition technology. Given the ability of unsupervised pre-training technology in the text field to extract useful features from a large amount of unlabeled data, people are trying to introduce and apply unsupervised pre-training technology to the field of speech recognition to solve the problem of low-resource speech recognition.

[0033] Unsupervised pre-training methods mainly rely on constructing representation learning tasks, and use massive amounts of unlabeled speech data to train models in an unsupervised manner to learn deep features in speech signals. This method does not rely on any form of annotation in the pre-training stage, and only uses data from a single modality, speech, for feature extraction and model optimization. Subsequently, in the supervised training stage, a small amount of speech-labeled text paired data is used to fine-tune the pre-trained model to adapt to specific speech recognition tasks. This two-stage training strategy has achieved significant performance improvements for low-resource speech recognition tasks under experimental conditions, demonstrating the effectiveness of unsupervised pre-training technology in solving low-resource problems.

[0034] Among them, when learning deep features in speech signals through unsupervised pre-training methods, it can be achieved by constructing a contrastive learning task. For example, in the specific implementation, the pre-training framework may include a feature extraction module, a quantization module, and a context representation module. After the speech signal passes through the feature extraction module, it is divided into two paths, one of which is frame information masked and input into the context representation module based on the Transformer structure, and the other is passed through the quantization module to discretize the continuous input features and then transform them into the codebook space to obtain a representation vector based on the codebook. Finally, combining the output of the context representation module and the output of the quantization module, the true quantized hidden variable representation is distinguished from other negative examples by constructing a contrast task. The pre-training framework thus implemented has achieved good performance in low-resource speech recognition tasks under experimental conditions.

[0035] For example, under the pre-training framework, the training of speech recognition models can include two stages, namely the unsupervised pre-training stage and the supervised learning stage. In the unsupervised pre-training stage, unsupervised data is used to construct a contrastive learning task, and the loss function of this stage is the contrastive prediction loss; in the supervised learning stage, annotated speech data is used to minimize the CTC (Connectionist Temporal Classification) loss training of the pre-trained model, and finally a speech recognition system with good performance under high signal-to-noise conditions is obtained.

[0036] The specific training process is as follows: 1. Data construction: Pre-training data for the unsupervised pre-training stage and fine-tuning data for the supervised learning stage are constructed respectively, where the pre-training data is unsupervised data and the fine-tuning data is labeled data.

[0037] 2. Pre-training stage: The network framework for pre-training may include a feature extraction module, a quantization module, and a context representation module, wherein the feature extraction module may be a 7-layer convolutional network, and the context representation module may be composed of a multi-layer Transformer network.

[0038] The unsupervised data used for pre-training is extracted through a 7-layer convolutional network and input into the context representation module and the quantization module respectively. The context representation module will mask some time frames of the input features, while the quantization module will not perform masking. The goal of pre-training is to learn to distinguish the correct quantization from the negative examples for all the masked time steps in the context representation module. The loss function of pre-training can be the fusion loss, which includes the contrast loss. and diversity loss (This loss is used to average the usage of each codebook) and is defined as follows:

[0039] in, represents the cosine similarity, is the output of the context representation module, is the quantitative representation of the positive sample, is the average of the softmax distribution of the g-th group of codebooks, is the average of the softmax distribution of codebook v in the g-th group. , are model parameters, is the set of quantized representations of positive samples, is the number of codebook groups, is the number of codebooks in a set of codebooks.

[0040] 3. Supervised learning stage: Use labeled data to perform supervised learning on the pre-trained model. The current mainstream approach is to add a randomly initialized linear layer to the pre-trained framework during the fine-tuning phase, and the CTC loss function is used as the loss function.

[0041] However, when customizing the pre-training scheme for a certain language based on the speech recognition model obtained by the above training process, there are still several key issues that need to be resolved. First, the poor recognition effect of low-frequency words is a significant problem. Since low-frequency words appear very rarely in the training data, it is difficult for the model to learn sufficient feature representations from the limited annotated data during fine-tuning, resulting in a significant drop in performance when recognizing these words. Secondly, the lack of generalization of the model is also a major challenge. In the pre-training stage, feature extraction and model optimization are performed only through data from a single modality, namely speech. As a result, the speech recognition model is difficult to learn sufficient feature representations from the limited annotated text during fine-tuning, and the use of text information is insufficient. The language constraint ability is low, and due to the lack of language constraints, self-created vocabulary is very likely to appear in the output results of the speech recognition model.

[0042] In view of the model generalization problems and poor recognition effect of the current mainstream pre-training technology in low-resource scenarios, an embodiment of the present invention provides a method for training a speech recognition model.

[0043] Figure 1 It is a flow chart of the training method of the speech recognition model provided by the present invention, such as Figure 1 As shown, the method includes: Step 110: extracting speech features of the labeled sample speech based on a speech coding network, and decoding the speech features based on a decoding network to obtain alignment information of the labeled sample speech.

[0044] Here, the speech coding network is an important component structure in the field of speech recognition and processing. The speech coding network is used to extract language-related feature information from speech signals, that is, to extract speech features, so as to facilitate subsequent tasks such as speech recognition, speech synthesis or speech classification. In the embodiments of the present invention, the network structure of the speech coding network is not specifically limited.

[0045] In the process of training the speech recognition model, the speech coding network can be regarded as a part of the speech recognition model, and the speech coding network can be used to extract speech features from the labeled sample speech. Here, the labeled sample speech is the labeled speech data. In the speech recognition scenario, the labeled sample speech is the sample speech with corresponding labeled text. The labeled text here is the label of the labeled sample speech during speech recognition, which can reflect the real text corresponding to the labeled sample speech.

[0046] In addition, the decoding network can map speech features to corresponding text representations, thereby converting the speech features into text sequences. For example, the decoding network can be an RNN-T (Recurrent Neural Network Transducer) decoder, or can be a CTC decoder, etc. The network structure of the decoding network is not specifically limited in the embodiments of the present invention.

[0047] In the process of training the speech recognition model, the decoding network can also be regarded as a part of the speech recognition model, and the decoding network can be used to decode the speech features output by the speech encoding network. It can be understood that by decoding the speech features through the decoding network, the text sequence corresponding to the speech features can be obtained, thereby achieving the alignment relationship between the labeled sample speech and the decoded text sequence, that is, obtaining the alignment information of the labeled sample speech.

[0048] Here, the alignment information of the labeled sample speech is represented as the alignment relationship between speech and text, which can be specifically reflected as the time period corresponding to each character, word or sentence in the labeled sample speech.

[0049] Step 120, based on the text encoding network, extract the text features of the label text of the label sample speech, and align the text features based on the alignment information to obtain aligned text features. The speech encoding network and the text encoding network share some encoding modules.

[0050] Specifically, the text encoding network is used to extract language-related feature information from the text, that is, to extract text features. In the embodiments of the present invention, the network structure of the text encoding network is not specifically limited.

[0051] It should be noted that, in the embodiment of the present invention, the speech coding network and the text coding network share some coding modules. The coding module here is a coding module shared by the speech coding network and the text coding network, that is, the coding module participates in the process of extracting speech features by the speech coding network and also participates in the process of extracting text features by the text coding network.

[0052] In the process of training the speech recognition model, the text encoding network can be regarded as a network that assists the training of the speech recognition model, and the text encoding network is used to extract text features from the label text of the label sample speech. In addition, after obtaining the text features of the label text, the alignment information of the label sample speech obtained in step 110 can be applied to align the text feature trap features, that is, align the text features extracted by the text encoding network with the speech features extracted by the speech encoding network in step 110. In an embodiment of the present invention, the result of feature alignment of the text features obtained in this way is recorded as the aligned text features. It can be understood that after feature alignment, the speech features and the aligned text features have the conditions for feature comparison in the same time period.

[0053] In this process, as an encoding module shared by the speech encoding network and the text encoding network, the encoding module participates in both the process of extracting speech features of the labeled sample speech based on the speech encoding network in step 110, and the process of extracting text features of the labeled text of the labeled sample speech based on the text encoding network in step 120. Therefore, the encoding module can better understand and process information from the speech modality and text modality, thereby more easily realizing cross-modal interaction and fusion, and further learning a more comprehensive and rich feature representation.

[0054] Step 130: performing parameter iteration on the speech encoding network, the text encoding network and the decoding network based on the speech features and the aligned text features.

[0055] Specifically, after obtaining the speech features of the labeled sample speech and the aligned text features of the labeled text of the labeled sample speech respectively, the speech features and the aligned text features from the same labeled sample speech can be made as consistent as possible, and parameters of the speech encoding network, the text encoding network and the decoding network can be iterated. In this way, the speech encoding network, the text encoding network and the decoding network can learn the joint feature representation of the two modalities of speech and text during the parameter iteration process, thereby improving the feature representation ability of the above networks.

[0056] Step 140, constructing the speech recognition model based on the speech encoding network and decoding network completed by parameter iteration.

[0057] Specifically, after the parameter iteration is completed, the speech encoding network and the decoding network can be cascaded to obtain a speech recognition model.

[0058] In the method provided in the embodiment of the present invention, labeled sample speech and labeled text of the labeled sample speech are applied as multimodal training data. Through the application of multimodal training data and the encoding module shared by the speech encoding network and the text encoding network, the joint feature representation of the two modalities of speech and text can be learned during the training process, thereby improving the feature representation ability of the speech recognition model, and then effectively improving the generalization ability of the speech recognition model.

[0059] Based on the above embodiment, the decoding network is an RNN-T decoder.

[0060] Specifically, RNN-T is an end-to-end speech recognition model, which is an improved version of the CTC (Connectionist Temporal Classification) model, and is designed to address some limitations of the CTC model in speech recognition tasks, such as the lack of language modeling capabilities and output dependency modeling capabilities. In the speech recognition model provided in the embodiment of the present invention, the text reconstruction unit of RNN-T is introduced as a decoding network, which strengthens the linguistic constraints on the output of the speech recognition model and effectively reduces the possibility of the appearance of "self-created" vocabulary in the speech recognition model.

[0061] Based on any of the above embodiments, in step 130, the performing parameter iteration on the speech encoding network, the text encoding network, and the decoding network based on the speech feature and the aligned text feature includes: Based on the difference between the speech feature and the aligned text feature, and the difference between the label text and the decoded text obtained by decoding the speech feature, parameter iteration is performed on the speech encoding network, the text encoding network and the decoding network.

[0062] Specifically, when performing parameter iteration on the speech encoding network, the text encoding network, and the decoding network, in addition to referring to the speech features and the aligned text features, the label text and the decoded text obtained by decoding the speech features may also be referred to.

[0063] It can be understood that in step 110, based on the decoding network decoding speech features, the decoded text of the labeled sample speech can be obtained, and the decoded text here can be understood as the intermediate amount of obtaining alignment information. Considering that the decoded text can be regarded as the speech recognition result of the labeled sample speech obtained based on the speech encoding network and the decoding network, and the label text itself is the real content text of the labeled sample speech, the two can be compared to evaluate the effect of the speech encoding network and the decoding network on the speech recognition of the labeled sample speech.

[0064] Therefore, after obtaining the speech features, aligned text features, labeled text and decoded text, the difference between the speech features and the aligned text features, as well as the difference between the labeled text and the decoded text can be calculated, thereby obtaining the loss during speech recognition model training, and then performing parameter iteration on the above-mentioned speech encoding network, text encoding network and decoding network based on the loss value.

[0065] Here, the loss during speech recognition model training It can be expressed as the following formula:

[0066] In the formula, is the voice feature, To align text features, is the mean squared error (MSE) loss between speech features and aligned text features; is the label text, It is the probability loss that the decoded text obtained by decoding based on speech features is consistent with the labeled text.

[0067] In the method provided in the embodiment of the present invention, parameter iteration is performed by combining the differences between speech features and aligned text features, as well as the differences between labeled text and decoded text obtained by decoding speech features. This can further optimize the cross-modal feature representation capability of the speech recognition model, thereby improving the generalization ability of the speech recognition model.

[0068] Based on any of the above embodiments, before step 130, the method further includes: Extracting masked speech features of unlabeled sample speech based on the speech coding network, and extracting unmasked speech features of the unlabeled sample speech based on a speech backbone network in the speech coding network, wherein the speech backbone network is the part of the speech coding network other than the coding module; Based on the masked speech features and the unmasked speech features, parameter iteration is performed on the speech coding network.

[0069] Specifically, before applying labeled sample speech and labeled text of labeled sample speech as multimodal training data, and performing multimodal training on the speech coding network, text coding network and decoding network, the speech coding network itself can also be pre-trained in speech modality, thereby optimizing the speech coding network's ability to represent speech features.

[0070] Here, the speech coding network can be divided into two parts, namely, the speech backbone network and the coding module, wherein the coding module is the shared part of the speech coding network and the text coding network, and the speech backbone network is the part of the speech coding network excluding the coding module. Among them, the speech backbone network can be a pre-trained network with speech feature extraction capabilities.

[0071] In this case, unlabeled sample speech can be obtained as pre-training data for the speech coding network. Here, unlabeled sample speech is different from labeled sample speech. Unlabeled sample speech is sample speech that has not been labeled and does not have corresponding label text, that is, unlabeled sample speech is unsupervised sample speech.

[0072] In a specific implementation, the unlabeled sample speech can be masked, and based on the speech coding network, the speech features of the masked unlabeled sample speech can be extracted, which are recorded as masked speech features. It can be understood that the masked speech features are features obtained after the masked unlabeled sample speech passes through the speech backbone network and the encoding module in the speech coding network.

[0073] In addition, the speech features of the unlabeled sample speech can also be extracted based on the speech backbone network. And, in order to distinguish them from the masked speech features extracted based on the speech coding network, the speech features of the unlabeled sample speech extracted based on the speech backbone network are recorded as unmasked speech features.

[0074] It can be understood that since the speech backbone network itself has been pre-trained and has the ability to extract speech features, the unmasked speech features extracted based on the speech backbone network can be used as a reference for the masked speech features extracted based on the speech coding network. Therefore, by comparing the masked speech features and the unmasked speech features, a comparative learning task can be constructed to achieve parameter iteration for the speech coding network.

[0075] As an example, the comparison between masked speech features and unmasked speech features can be achieved by calculating the Masked Language Model (MLM) loss. On this basis, the MLM loss can be applied to iterate the parameters of the speech coding network.

[0076] In the method provided in the embodiment of the present invention, by using unsupervised unlabeled sample speech, comparative learning is performed on the speech backbone network and the coding module in the speech coding network, so that before parameter iteration based on multimodal training data, the speech coding network already has excellent speech feature representation capabilities, thereby providing a basis for accelerating multimodal model training.

[0077] Based on any of the above embodiments, before step 130, the method further includes: Extracting unlabeled text features of unlabeled sample text based on the text encoding network, and decoding the unlabeled text features based on the decoding network to obtain restored text; Based on the unlabeled sample text and the restored text, parameters of the text encoding network and the decoding network are iterated.

[0078] Specifically, before applying labeled sample speech and labeled text of labeled sample speech as multimodal training data and performing multimodal training on the speech encoding network, text encoding network and decoding network, the text encoding network itself can also be pre-trained in text modality, thereby optimizing the text encoding network's representation capability for text features.

[0079] In the specific implementation, the unlabeled sample text can be used as pre-training data for the speech coding network. Here, the unlabeled sample text is a sample text that has not been labeled and has no corresponding labeled speech, that is, the unlabeled sample text is an unsupervised sample text.

[0080] In the specific implementation, the text features of the unmarked sample text can be extracted based on the text encoding network, which are recorded as unmarked text features here; then, the unmarked text features are used as the input of the decoding network, that is, the unmarked text features are decoded based on the decoding network, thereby obtaining the decoded text output by the decoding network, that is, the restored text.

[0081] After obtaining the restored text, the restored text can be compared with the unlabeled sample text to determine the loss value to achieve parameter iteration for the text encoding network and decoding network.

[0082] In the method provided in an embodiment of the present invention, by using unsupervised unlabeled sample text, parameter iteration is performed on the text encoding network and the decoding network, so that after parameter iteration based on multimodal training data, the text encoding network has excellent text feature representation capabilities, thereby providing a basis for accelerating multimodal model training.

[0083] In the method provided in the embodiment of the present invention, compared with the mainstream pre-training scheme, unlabeled sample text and labeled sample speech carrying labeled text are added as training data. Through this multimodal data combination and the structural design of the shared encoding module, the speech recognition model can learn the joint features of speech and text modalities, thereby improving the representation ability of the speech recognition model, thereby ensuring the generalization ability of the speech recognition model.

[0084] Based on any of the foregoing embodiments, the determination of the speech coding network includes: Get a pre-trained speech backbone network; The speech backbone network is cascaded with the coding module to obtain the speech coding network.

[0085] Specifically, the speech backbone network can be a pre-trained network with speech feature extraction capability. The speech backbone network can be a speech feature extraction model trained based on unsupervised speech data through a masking scheme. The pre-trained speech backbone network itself has learned the data of speech modality, and therefore has basic speech feature extraction capability.

[0086] However, since the pre-trained speech backbone network lacks the ability to constrain the text modality, directly applying the speech backbone network for speech recognition in low-resource scenarios will have model generalization problems. In this regard, in an embodiment of the present invention, after obtaining the pre-trained speech backbone network, the speech backbone network can be cascaded with the encoding module shared by the two modalities of speech and text, thereby obtaining a speech coding network, and then the speech coding network is trained to improve the generalization ability of the speech recognition model constructed thereby.

[0087] Here, the speech backbone network and the coding module are cascaded, and specifically the output of the speech backbone network is used as the input of the coding model. In the speech coding network thus obtained, both the speech backbone network and the coding module are used to extract speech features.

[0088] Furthermore, the encoding module referred to here can be a parameter-initialized, untrained encoding module, and the structure of the encoding module can refer to the structure of the voice backbone network. For example, in order to accelerate training convergence, the structure of the encoding module can be set to the same structure as the voice backbone network, and the number of layers of the encoding module can be halved compared to the voice backbone network.

[0089] For example, a speech backbone network can be composed of a feature extraction network and a context encoding module. Figure 2 It is a structural diagram of the feature extraction network provided by the present invention, such as Figure 2 As shown, the feature extraction network uses a 7-layer convolutional structure (corresponding to Figure 2The CNN1-CNN7 in the figure is used for speech preprocessing, and then the deep features of the speech are extracted through layer normalization (LayerNorm). Each layer of the convolution structure includes a 1D convolution (Conv1d), group normalization (GroupNorm) and a Gaussian Error Linear Unit (GELU). The convolution dimension is set to 512, the step size is [5, 2, 2, 2, 2, 2], and the convolution kernel is [10, 3, 3, 3, 3, 2, 2]. The context encoding module consists of a multi-layer Conformer structure. Each Conformer layer contains FFN (Feedforward Neural Networks), LayerDrop and Attention structures. The Conformer can be set to 12 layers, the FFN inner layer dimension is 3072, the number of Attention heads is 8, and the encoding layer output dimension is 768. Here, the application of the Conformer structure can take into account the extraction of local and global information, thereby optimizing the extraction of speech features.

[0090] The structure of the encoding module can be the same as that of the voice backbone network, and the network parameters of the voice backbone network can be halved. In order to accelerate training convergence, the parameters of the encoding module can be initialized using the last 6 layers of the pre-trained voice backbone network.

[0091] Based on any of the foregoing embodiments, in the speech coding network, determining the speech backbone network includes: Identify faculty and student networks; Extracting teacher speech features of unsupervised speech based on the teacher network, and extracting student speech features of masked speech based on the student network, wherein the masked speech is obtained by masking the unsupervised speech; Based on the teacher speech feature and the student speech feature, performing parameter iteration on the teacher network and the student network; The student network after parameter iteration is used as the speech backbone network.

[0092] Specifically, for the pre-training of the speech backbone network, a teacher-student framework can be used to construct a contrastive learning task.

[0093] Among them, the teacher network and the student network are both network structures used for speech feature extraction. The network structures of the teacher network and the student network can be the same or different. The number of student networks can be one or more, and the embodiment of the present invention does not make specific limitations on this.

[0094] In the embodiment of the present invention, the teacher network is an unmasked speech backbone network, and the student network is a masked speech backbone network. The difference between the two is whether the speech is masked during the speech feature extraction process.

[0095] After obtaining the teacher network and the student network, the unsupervised speech can be input into the teacher network, and the teacher network extracts speech features for the unsupervised speech. Here, the speech features extracted by the teacher network are recorded as teacher speech features. It should be noted that the unsupervised speech here refers to sample speech that has not been labeled and does not have corresponding label text. The unsupervised speech can be the same batch of sample speech as the unlabeled sample in the above embodiment, or it can be the sample speech of a different batch in the above embodiment. The embodiment of the present invention does not specifically limit this.

[0096] In addition, the unsupervised speech can be masked to obtain the unsupervised speech with mask, which is recorded as masked speech. The masked speech can be input into the student network, and the student network extracts speech features for the masked speech. The speech features extracted by the student network are recorded as student speech features.

[0097] After obtaining the teacher's speech features output by the teacher network and the student's speech features output by the student network, comparative learning can be performed on the teacher's speech features and the student's speech features. In this process, since the student network adopts a random masking method, the student's speech features carry the speech features of the masked positions predicted by the student network. By comparing the teacher's speech features extracted by the teacher network for the unmasked speech with the student's speech features predicted by the student network for the masked speech, the loss value can be calculated to achieve parameter iteration of the teacher network and the student network.

[0098] After the reference iteration is completed, the student network after parameter iteration can be used as the pre-trained speech backbone network.

[0099] Based on any of the above embodiments, in the step of determining the speech backbone network, the performing parameter iteration on the teacher network and the student network based on the teacher speech feature and the student speech feature includes: Based on the feature decoding network, the student speech feature is decoded to obtain the masked speech feature; Based on the difference between the masked speech feature and the teacher speech feature, parameter iteration is performed on the teacher network and the student network.

[0100] Specifically, in the process of combining the teacher network and the student network to construct the contrastive learning task, the feature decoding network is also applied. The feature decoding network here can be a decoder structure, for example, a decoder structure based on a CNN (Convolutional Neural Network) module, which can specifically include a 6-layer CNN, wherein each CNN layer can include a one-dimensional convolution, a linear regularization layer, and a GELU activation function layer, wherein the intermediate dimension of the one-dimensional convolution layer is 384, and the convolution kernel is [7,1].

[0101] The input of the feature decoding network can be connected to the output of the student network, that is, after obtaining the student speech features output by the student network, the student speech features can be input into the feature decoding network, and the student speech features are decoded by the feature decoding network to restore the speech features at the masked positions predicted in the student speech features, that is, to obtain the masked speech features.

[0102] It can be understood that the masked speech feature is the speech feature of the masked part of the masked speech predicted by the student network. After obtaining the masked speech feature, the loss value can be calculated by comparing the teacher speech feature and the masked speech feature, thereby realizing parameter iteration of the teacher network and the student network.

[0103] For example, Figure 3 It is a training diagram of the speech backbone network provided by the present invention, such as Figure 3 As shown, the unmasked speech backbone network is the teacher network, and the masked speech backbone network is the student network. There can be n student networks, corresponding to Figure 3 The masked speech backbone network 1-n in the. The speech features of the unsupervised speech based on the output of the unmasked speech backbone network, i.e., the teacher speech features The speech features of the masked speech output by the masked speech backbone network, namely the student speech features. Figure 3 In the figure, the generation of student speech features can be represented by the dotted box, that is, the features originally represented by blanks become student speech features after being predicted by the masked speech backbone network, that is, the masked part changes from blank representation to gray representation. After that, the student speech features can be input into the CNN decoder to obtain the masked speech features. Corresponding to the masked speech backbone networks 1-n, the masked speech features are represented as - .

[0104] After obtaining the teacher's voice features , and the masked speech features corresponding to each masked speech backbone network are expressed as - After that, the loss value between each masked speech backbone network and the unmasked speech backbone network can be obtained based on this. , which can be expressed as:

[0105] Wherein, i is an integer between 1 and n, referring to the i-th masked speech backbone network.

[0106] Then, the total loss of the voice backbone network can be obtained , which can be expressed as:

[0107] Based on any of the above embodiments, the method further includes: The speech recognition model is fine-tuned based on the scene speech in the target scene and the label text of the scene speech.

[0108] Specifically, the speech recognition model constructed based on step 140 can be regarded as a pre-trained speech recognition model suitable for general scenarios. In order to further meet the speech recognition needs in specific scenarios, the scene speech of the target scene and the label text of the scene speech can be applied on the basis of the existing speech recognition model to fine-tune the model, thereby obtaining a low-resource speech recognition model in the target scenario.

[0109] For example, when fine-tuning the model, parameter iteration can be guided based on the RNN-T loss function.

[0110] The target scenario referred to here is the scenario in which the speech recognition model is to be applied, such as medical scenarios, financial scenarios, astrophysics, etc. The scene speech in the target scenario is the supervised speech suitable for the target scenario, and the label text of the scene speech is the content text of the pre-labeled scene speech. By applying the scene speech in the target scenario and the label text of the scene speech, the speech recognition model is fine-tuned in a supervised manner to obtain a low-resource speech recognition model in the scenario. This process does not require the addition of backend structure, is simple and effective, and provides convenience for engineering implementation and solution promotion and implementation.

[0111] Based on any of the above embodiments, in the training method of the speech recognition model, the pre-training of the speech recognition model can be specifically divided into two stages, namely, a first pre-training stage and a second pre-training stage.

[0112] in, Figure 4 Schematic diagram of the first pre-training phase of the speech recognition model provided by the present invention. Figure 4 As shown, in the first pre-training stage of the speech recognition model, pre-training of the speech backbone network in the speech modality can be achieved.

[0113] Specifically, when pre-training the speech backbone network, a comparative learning task between the teacher network and multiple student networks can be constructed based on the teacher-student model framework, and the first pre-training can be performed through unsupervised speech. Among them, the teacher network is an unmasked speech backbone network, and the student network is a masked speech backbone network, and the masked speech backbone network can be connected to the decoder structure implemented based on the CNN module.

[0114] In the first pre-training process, the student network adopts a random mask method to minimize the loss between the target output of the teacher network and the decoded output of the student network. To train the speech backbone network, during this process, the parameters of the teacher network are updated following the student network.

[0115] Through the first stage of pre-training, the initial speech backbone network can be obtained.

[0116] It should be noted that, here, for any speech backbone network, its structure may include a cascaded feature extraction network and a context encoding module, wherein the feature extraction network is used to extract features at non-masked locations in the input speech, and the context encoding module predicts features at masked locations based on features at non-masked locations, thereby obtaining the complete speech features of the input speech.

[0117] Figure 5 is a flow chart of the second pre-training stage of the speech recognition model provided by the present invention, such as Figure 5 As shown, in the second pre-training stage of the speech recognition model, multimodal training for speech and text can be achieved.

[0118] Specifically, in the second pre-training stage, a cross-modal mapping enhancement structure can be introduced. This structure is essentially a shared encoding module, which is specifically reflected in the fact that the speech encoding network and the text encoding network share the same encoding module. The encoding module is implemented based on Transformer and is used to fit the two data modalities of speech and text. Figure 5 As shown, the speech coding network is a cascade structure of a speech backbone network and a shared coding module, and the text coding network is a cascade structure of a text backbone network and a shared coding module.

[0119] In the second pre-training stage, the speech backbone network obtained in the first pre-training stage can be used as the speech backbone network in the speech coding network of the second pre-training stage, and the shared coding module is initialized based on the speech backbone network. The parameters of the remaining modules can be obtained by random initialization.

[0120] The second pre-training stage requires the application of unsupervised speech, unsupervised text, and labeled sample speech with labeled text for training.

[0121] Among them, unsupervised speech can be used to train the speech backbone network and the shared encoding module. Specifically, the shared encoding module and the speech backbone network are used to construct a contrastive learning task, and the masked language model MLM loss is used. Instruction training; In addition, unsupervised text can be used to train the combined text encoding network and RNN-T structure decoding network. Specifically, the text encoding network is used to convert the unsupervised text into text features in the same feature space as the speech features, and then the decoding network is used to decode the text features into restored text, and the unsupervised text and restored text are used for RNN-T training. The loss here can be recorded as .

[0122] Furthermore, we can also use labeled sample speech with labeled text to learn text-to-speech identification. Specifically, we can input labeled sample speech, pass it through the speech encoding network and the RNN-T structure decoding network to obtain text-to-speech alignment information, and then input the labeled text into the text encoding network to obtain text features, and then use the alignment information to convert the text features into Conversion to speech features Align text features of the same length ,By comparing speech features and aligning text features for parameter iteration, cross-modal mapping from text to speech is achieved.

[0123] In the second pre-training stage, cross-modal fusion of text and speech is achieved through the application of unsupervised speech, unsupervised text, and labeled sample speech carrying labeled text. The speech encoding network and decoding network obtained in this way can be used as the speech recognition model after the first pre-training and the second pre-training.

[0124] In addition, after obtaining the speech recognition model, the pre-trained speech recognition model can be fine-tuned using supervised data in the target scenario, thereby obtaining a cross-modal enhanced low-resource speech recognition model.

[0125] In the embodiment of the present invention, the speech recognition model obtained by combining the first and second pre-training stages can have a more powerful speech-text joint representation capability, thereby effectively ensuring the generalization ability of speech recognition.

[0126] Based on any of the above embodiments, Figure 6 Schematic diagram of the flow of the speech recognition method provided by the present invention. Figure 6 As shown, the method includes: Step 610, obtaining the speech to be recognized; Step 620: Recognize the speech to be recognized based on a speech recognition model, wherein the speech recognition model is obtained based on the training method of the speech recognition model provided in the above embodiment.

[0127] Specifically, the speech to be recognized is the speech data that needs to be recognized. The speech to be recognized can be obtained through a sound pickup device, which can be a smart phone, a tablet computer, or a smart appliance such as a stereo, a television, and an air conditioner. After the sound pickup device obtains the speech to be recognized through a microphone array, it can also amplify and reduce noise of the speech to be recognized. In addition, the speech to be recognized can be a speech segment formed after the sound pickup is completed, or it can be a speech stream in the real-time sound pickup process, which is not specifically limited in the embodiment of the present invention.

[0128] After obtaining the speech to be recognized, the speech to be recognized can be used as the input of the speech recognition model, and the speech recognition model performs speech recognition on the input speech to be recognized, thereby obtaining the transcribed text of the speech to be recognized output by the speech recognition model.

[0129] It should be noted that the speech recognition model here is a model obtained based on the training method of the speech recognition model provided in any of the above embodiments. In the training method of the above speech recognition model, labeled sample speech and labeled text of the labeled sample speech are used as multimodal training data. Through the application of multimodal training data and the encoding module shared by the speech coding network and the text coding network, the joint feature representation of the two modes of speech and text can be learned during the training process, thereby improving the feature representation ability of the speech recognition model. Therefore, the speech recognition model is obtained based on the training method of the speech recognition model provided in any of the above embodiments, and speech recognition is performed on the speech to be recognized, which can ensure the reliability and accuracy of speech recognition.

[0130] Based on any of the above embodiments, Figure 7 Schematic diagram of the structure of the training device of the speech recognition model provided by the present invention. Figure 7 As shown, the device comprises: The speech encoding unit 710 is used to extract speech features of the labeled sample speech based on the speech encoding network, and decode the speech features based on the decoding network to obtain alignment information of the labeled sample speech; A text encoding unit 720, configured to extract text features of the label text of the label sample speech based on a text encoding network, and perform feature alignment on the text features based on the alignment information to obtain aligned text features, and a coding module that the speech encoding network and the text encoding network share a portion of; A parameter iteration unit 730, configured to perform parameter iteration on the speech encoding network, the text encoding network, and the decoding network based on the speech feature and the aligned text feature; The model building unit 740 is used to build the speech recognition model based on the speech encoding network and decoding network completed by parameter iteration.

[0131] In the device provided in the embodiment of the present invention, labeled sample speech and labeled text of the labeled sample speech are applied as multimodal training data. Through the application of multimodal training data and the encoding module shared by the speech encoding network and the text encoding network, the joint feature representation of the two modalities of speech and text can be learned during the training process, thereby improving the feature representation ability of the speech recognition model, and then effectively improving the generalization ability of the speech recognition model.

[0132] Based on any of the above embodiments, the parameter iteration unit is specifically used for: Based on the difference between the speech feature and the aligned text feature, and the difference between the label text and the decoded text obtained by decoding the speech feature, parameter iteration is performed on the speech encoding network, the text encoding network and the decoding network.

[0133] Based on any of the above embodiments, the device further includes a speech pre-training unit, which is used to: Extracting masked speech features of unlabeled sample speech based on the speech coding network, and extracting unmasked speech features of the unlabeled sample speech based on a speech backbone network in the speech coding network, wherein the speech backbone network is the part of the speech coding network other than the coding module; Based on the masked speech features and the unmasked speech features, parameter iteration is performed on the speech coding network.

[0134] Based on any of the above embodiments, the device further includes a text pre-training unit, which is used to: Extracting unlabeled text features of unlabeled sample text based on the text encoding network, and decoding the unlabeled text features based on the decoding network to obtain restored text; Based on the unlabeled sample text and the restored text, parameters of the text encoding network and the decoding network are iterated.

[0135] Based on any of the above embodiments, the device further includes a speech coding construction unit, which is used to: Get a pre-trained speech backbone network; The speech backbone network is cascaded with the coding module to obtain the speech coding network.

[0136] Based on any of the above embodiments, the speech coding construction unit is further used for: Identify faculty and student networks; Extracting teacher speech features of unsupervised speech based on the teacher network, and extracting student speech features of masked speech based on the student network, wherein the masked speech is obtained by masking the unsupervised speech; Based on the teacher speech feature and the student speech feature, performing parameter iteration on the teacher network and the student network; The student network after parameter iteration is used as the speech backbone network.

[0137] Based on any of the above embodiments, the speech coding construction unit is further used for: Based on the feature decoding network, the student speech feature is decoded to obtain the masked speech feature; Based on the difference between the masked speech feature and the teacher speech feature, parameter iteration is performed on the teacher network and the student network.

[0138] Based on any of the above embodiments, the device further includes a fine-tuning unit, configured to: The speech recognition model is fine-tuned based on the scene speech in the target scene and the label text of the scene speech.

[0139] Based on any of the above embodiments, Figure 8 : is a schematic diagram of the structure of the speech recognition device provided by the present invention, such as Figure 8 As shown, the device comprises: The acquisition unit 810 acquires the speech to be recognized; The recognition unit 820 recognizes the speech to be recognized based on a speech recognition model, and the speech recognition model is obtained based on the training method of the above-mentioned speech recognition model.

[0140] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the training method of the speech recognition model, and the method includes: Extracting speech features of the labeled sample speech based on a speech coding network, and decoding the speech features based on a decoding network to obtain alignment information of the labeled sample speech; Based on the text encoding network, extract the text features of the label text of the label sample speech, and perform feature alignment on the text features based on the alignment information to obtain aligned text features, and the speech encoding network and the text encoding network share a portion of the encoding module; Based on the speech features and the aligned text features, performing parameter iteration on the speech encoding network, the text encoding network and the decoding network; The speech recognition model is constructed based on the speech encoding network and the decoding network completed by parameter iteration.

[0141] Alternatively, the processor 910 may call the logic instructions in the memory 930 to execute the speech recognition method, which includes: Get the speech to be recognized; The speech to be recognized is recognized based on a speech recognition model, and the speech recognition model is obtained based on a training method of a speech recognition model.

[0142] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0143] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the training method of the speech recognition model provided by the above methods, the method comprising: Extracting speech features of the labeled sample speech based on a speech coding network, and decoding the speech features based on a decoding network to obtain alignment information of the labeled sample speech; Based on the text encoding network, extract the text features of the label text of the label sample speech, and perform feature alignment on the text features based on the alignment information to obtain aligned text features, and the speech encoding network and the text encoding network share a portion of the encoding module; Based on the speech features and the aligned text features, performing parameter iteration on the speech encoding network, the text encoding network and the decoding network; The speech recognition model is constructed based on the speech encoding network and the decoding network completed by parameter iteration.

[0144] Alternatively, when the computer program is executed by a processor, the computer can execute the speech recognition method provided by the above methods, the method comprising: Get the speech to be recognized; The speech to be recognized is recognized based on a speech recognition model, and the speech recognition model is obtained based on a training method of a speech recognition model.

[0145] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the training method of the speech recognition model provided by the above methods, the method comprising: Extracting speech features of the labeled sample speech based on a speech coding network, and decoding the speech features based on a decoding network to obtain alignment information of the labeled sample speech; Based on the text encoding network, extract the text features of the label text of the label sample speech, and perform feature alignment on the text features based on the alignment information to obtain aligned text features, and the speech encoding network and the text encoding network share a portion of the encoding module; Based on the speech features and the aligned text features, performing parameter iteration on the speech encoding network, the text encoding network and the decoding network; The speech recognition model is constructed based on the speech encoding network and the decoding network completed by parameter iteration.

[0146] Alternatively, when the computer program is executed by a processor, the computer program is implemented to perform the speech recognition method provided by the above methods, the method comprising: Get the speech to be recognized; The speech to be recognized is recognized based on a speech recognition model, and the speech recognition model is obtained based on a training method of a speech recognition model.

[0147] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0148] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a speech recognition model, characterized in that: include: Extracting speech features of the labeled sample speech based on a speech coding network, and decoding the speech features based on a decoding network to obtain alignment information of the labeled sample speech; Based on the text encoding network, extract the text features of the label text of the label sample speech, and perform feature alignment on the text features based on the alignment information to obtain aligned text features, and the speech encoding network and the text encoding network share a portion of the encoding module; Based on the speech features and the aligned text features, performing parameter iteration on the speech encoding network, the text encoding network and the decoding network; The speech recognition model is constructed based on the speech encoding network and the decoding network completed by parameter iteration.

2. The method for training a speech recognition model according to claim 1, characterized in that: The step of performing parameter iteration on the speech encoding network, the text encoding network, and the decoding network based on the speech feature and the aligned text feature comprises: Based on the difference between the speech feature and the aligned text feature, and the difference between the label text and the decoded text obtained by decoding the speech feature, parameter iteration is performed on the speech encoding network, the text encoding network and the decoding network.

3. The method for training a speech recognition model according to claim 1, characterized in that: The step of performing parameter iteration on the speech encoding network, the text encoding network and the decoding network based on the speech feature and the aligned text feature further comprises: Extracting masked speech features of unlabeled sample speech based on the speech coding network, and extracting unmasked speech features of the unlabeled sample speech based on a speech backbone network in the speech coding network, wherein the speech backbone network is the part of the speech coding network other than the coding module; Based on the masked speech features and the unmasked speech features, parameter iteration is performed on the speech coding network.

4. The method for training a speech recognition model according to claim 1, characterized in that: The step of performing parameter iteration on the speech encoding network, the text encoding network and the decoding network based on the speech feature and the aligned text feature further comprises: Extracting unlabeled text features of unlabeled sample text based on the text encoding network, and decoding the unlabeled text features based on the decoding network to obtain restored text; Based on the unlabeled sample text and the restored text, parameters of the text encoding network and the decoding network are iterated.

5. The method for training a speech recognition model according to claim 1, characterized in that: The determination of the speech coding network includes: Get a pre-trained speech backbone network; The speech backbone network is cascaded with the coding module to obtain the speech coding network.

6. The method for training a speech recognition model according to claim 5, characterized in that: The determination of the voice backbone network includes: Identify faculty and student networks; Extracting teacher speech features of unsupervised speech based on the teacher network, and extracting student speech features of masked speech based on the student network, wherein the masked speech is obtained by masking the unsupervised speech; Based on the teacher speech feature and the student speech feature, performing parameter iteration on the teacher network and the student network; The student network after parameter iteration is used as the speech backbone network.

7. The method for training a speech recognition model according to claim 6, characterized in that: The performing parameter iteration on the teacher network and the student network based on the teacher speech feature and the student speech feature comprises: Based on the feature decoding network, the student speech feature is decoded to obtain the masked speech feature; Based on the difference between the masked speech feature and the teacher speech feature, parameter iteration is performed on the teacher network and the student network.

8. The method for training a speech recognition model according to any one of claims 1 to 7, characterized in that: Also includes: The speech recognition model is fine-tuned based on the scene speech in the target scene and the label text of the scene speech.

9. A speech recognition method, characterized in that: include: Get the speech to be recognized; The speech to be recognized is recognized based on a speech recognition model, wherein the speech recognition model is obtained based on the training method of the speech recognition model according to any one of claims 1 to 8.

10. A training device for a speech recognition model, characterized in that: include: A speech encoding unit, used for extracting speech features of the labeled sample speech based on a speech encoding network, and decoding the speech features based on a decoding network to obtain alignment information of the labeled sample speech; A text encoding unit, used for extracting text features of the label text of the label sample speech based on a text encoding network, and aligning the text features based on the alignment information to obtain aligned text features, and an encoding module of a shared part of the speech encoding network and the text encoding network; A parameter iteration unit, configured to perform parameter iteration on the speech encoding network, the text encoding network, and the decoding network based on the speech feature and the aligned text feature; The model building unit is used to build the speech recognition model based on the speech encoding network and decoding network completed by parameter iteration.

11. A speech recognition device, characterized in that: include: An acquisition unit, used for acquiring speech to be recognized; A recognition unit is used to recognize the speech to be recognized based on a speech recognition model, wherein the speech recognition model is obtained based on the training method of the speech recognition model according to any one of claims 1 to 8.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the training method of the speech recognition model as described in any one of claims 1 to 8, or the speech recognition method as described in claim 9 is implemented.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the training method of the speech recognition model as described in any one of claims 1 to 8, or the speech recognition method as described in claim 9.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the training method of the speech recognition model as described in any one of claims 1 to 8, or the speech recognition method as described in claim 9.

Images