Speech recognition large model text aided training method, speech recognition method and device

Abstract

The invention discloses a voice recognition large model text auxiliary training method and device, and a voice recognition method and device, and relates to the technical field of voice recognition, and the method comprises the steps: obtaining a training set which is a text data set of a target field or an audio text pair data set of a source field, and training a trainable parameter entity through the training set, obtaining a trained parameter entity, obtaining a pseudo-audio feature corresponding to each piece of text data in the text data set based on the trained parameter entity, and performing parameter training on a text decoder and a text embedding layer in the initial speech recognition large model at least according to the text data set and the pseudo-audio features, and obtaining a target speech recognition large model. According to the method, the pseudo audio features are added in the training stage of the text decoder and the text embedding layer, and the real audio embedding features of the target field are simulated through the pseudo audio features, so that the optimization deviation of the text decoder is reduced, and the voice recognition performance of the target voice recognition large model on the target field audio is improved.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a text-assisted training method for a large speech recognition model, a speech recognition method, and an apparatus. Background Technology

[0002] In speech recognition, model training often relies on massive amounts of labeled speech-audio-text data pairs. However, in some specific domains, referred to as the target domain, text data is relatively easy to obtain, but the corresponding speech audio is difficult to obtain. In order to achieve efficient domain adaptation of large speech recognition models in the target domain, one current approach is to perform secondary pre-training on the text decoder of the large speech recognition model using massive amounts of domain-specific plain text, optimizing only the parameters of the text decoder.

[0003] However, using pure text pre-training, the text decoder can only learn the text feature distribution in the text space. In actual inference, the text decoder needs to process the audio embedding features output by the audio encoder. Since it has not pre-learned the speech acoustics knowledge of the target domain, the decoder optimization bias is caused, which reduces the speech recognition performance of the large speech recognition model for audio in the target domain. Summary of the Invention

[0004] In view of the above problems, this application provides a text-assisted training method, a speech recognition method, and an apparatus for a large-scale speech recognition model. This aims to generate pseudo-audio features of the target domain and use them in the training process of the text decoder, thereby reducing the optimization bias of the text decoder and improving the speech recognition performance of the large-scale speech recognition model for audio in the target domain. The specific solution is as follows:

[0005] The first aspect of this application provides a text-assisted training method for a large speech recognition model, including:

[0006] Obtain a training set, which is a text dataset of the target domain or an audio-text pair dataset of the source domain;

[0007] The trainable parameter entities are trained using the training set to obtain the trained parameter entities;

[0008] Based on the trained parameter entities, pseudo-audio features corresponding to each text data in the text dataset are obtained, and each pseudo-audio feature corresponding to the text data represents the audio embedding feature of the audio corresponding to the text data.

[0009] The text decoder and text embedding layer in the initial speech recognition large model are trained with parameters based at least on the text dataset and the pseudo-audio features to obtain the target speech recognition large model.

[0010] In one possible implementation, the trainable parameter entity is an initial pseudo-audio feature or a text encoder;

[0011] When the trainable parameter entity is the initial pseudo-audio feature, the training set is the text dataset;

[0012] When the trainable parameter entity is the text encoder, the training set is the audio-text pair dataset.

[0013] In one possible implementation, when the trainable parameter entity is the initial pseudo-audio feature, training the trainable parameter entity using the training set to obtain the trained parameter entity includes:

[0014] The audio encoder in the initial speech recognition model is removed, and the initial pseudo-audio features are then embedded into the audio encoder position to obtain the first model.

[0015] Freeze all parameters in the first model except for the initial pseudo-audio features, and feed the text dataset into the first model after freezing the parameters for training to obtain the trained pseudo-audio features.

[0016] The step of obtaining the pseudo-audio features corresponding to each text data in the text dataset based on the trained parameter entities includes:

[0017] The trained pseudo-audio features are used as the pseudo-audio features corresponding to each text data in the text dataset.

[0018] In one possible implementation, the process of determining the initial pseudo-audio features includes:

[0019] The first speech recognition big model performs text embedding encoding on each text data in the text dataset through the text embedding layer of the first speech recognition big model to obtain the text embedding feature corresponding to each text data. The first speech recognition big model is the initial speech recognition big model, or the first speech recognition big model is trained to obtain the initial speech recognition big model.

[0020] The target length is determined based on the text embedding features corresponding to all text data in the text dataset.

[0021] The first pseudo-audio feature is processed to obtain a second pseudo-audio feature with a sequence length equal to the target length, wherein the first pseudo-audio feature in the first iteration is a preset pseudo-audio feature;

[0022] A training round is initiated based on the second pseudo-audio feature to obtain the trained third pseudo-audio feature and the second large speech recognition model. The training round includes training the second pseudo-audio feature and training the text decoder and text embedding layer in the first large speech recognition model.

[0023] The second speech recognition model is used as the first speech recognition model, and the third pseudo-audio feature is used as the first pseudo-audio feature. The text embedding layer of the first speech recognition model is used to perform text embedding encoding on each text data in the text dataset until the preset iteration end condition is met and then the return stops.

[0024] The initial pseudo-audio feature is determined based on the third pseudo-audio feature or the first pseudo-audio feature after the iteration termination condition is met.

[0025] In one possible implementation, determining the target length based on the text embedding features corresponding to all text data in the text dataset includes:

[0026] Based on the text embedding features corresponding to each text data in the text dataset, determine the centroid vector corresponding to each text data;

[0027] Determine the target centroid vector based on the centroid vectors corresponding to all text data.

[0028] The text data whose centroid vector is most similar to the target centroid vector is determined from the text dataset and used as the target text data;

[0029] The number of lexical units corresponding to the target text data is determined as the target length.

[0030] In one possible implementation, determining the centroid vector corresponding to each text data point based on the text embedding features corresponding to each text data point in the text dataset includes:

[0031] The average value of each vector contained in the text embedding feature corresponding to each text data is calculated in each dimension to obtain the first mean vector, which is used as the centroid vector corresponding to each text data.

[0032] The step of determining the target centroid vector based on the centroid vectors corresponding to all text data includes:

[0033] The centroid vectors corresponding to all the text data are averaged in each dimension to obtain a second mean vector, which is used as the target centroid vector.

[0034] In one possible implementation, processing the first pseudo-audio feature to obtain a second pseudo-audio feature with a sequence length equal to the target length includes:

[0035] Determine the length difference between the target length of the current iteration and the target length of the previous iteration. When the current iteration is the first iteration, the target length of the previous iteration is the sequence length of the preset pseudo-audio feature.

[0036] If the length difference is positive, then a sub-feature with a sequence length equal to the length difference is extracted from the first pseudo-audio feature, and the sub-feature is concatenated with the first pseudo-audio feature to obtain the second pseudo-audio feature;

[0037] If the length difference is negative, then the absolute value of the length difference is calculated, and the sub-features with a sequence length equal to the absolute value are discarded from the first pseudo-audio features to obtain the second pseudo-audio features.

[0038] In one possible implementation, when the trainable parameter entity is the text encoder, the step of training the trainable parameter entity using the training set to obtain the trained parameter entity includes:

[0039] The text and audio from the audio-text pair dataset are fed into the text encoder and the audio encoder in the initial speech recognition large model, respectively, to obtain the output features of the text encoder and the audio encoder, respectively.

[0040] Calculate the feature loss between the output features of the text encoder and the audio encoder respectively;

[0041] The text encoder is trained using the feature loss to obtain the trained text encoder.

[0042] The step of obtaining the pseudo-audio features corresponding to each text data in the text dataset based on the trained parameter entities includes:

[0043] Each text data in the text dataset is fed into the trained text encoder to obtain the pseudo-audio features corresponding to each text data in the text dataset.

[0044] A second aspect of this application provides a speech recognition method, comprising:

[0045] Acquire the audio to be identified in the target area;

[0046] The audio to be recognized is fed into the configured target speech recognition large model to obtain the audio recognition text output by the model. The target speech recognition large model is obtained by training the text decoder and text embedding layer in the initial speech recognition large model with parameters based on the text dataset and the pseudo-audio features corresponding to each text data in the text dataset. The pseudo-audio features corresponding to each text data are obtained based on the parameter entities trained on the training set. The pseudo-audio features corresponding to each text data represent the audio embedding features of the audio corresponding to the text data. The training set is the text dataset of the target domain or the audio-text pair dataset of the source domain.

[0047] A third aspect of this application provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement the speech recognition large model text-assisted training method of the first aspect or any implementation thereof, or the speech recognition method of the second aspect.

[0048] A fourth aspect of this application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0049] The memory is used to store computer programs;

[0050] The processor is used to execute the computer program so that the electronic device can implement the speech recognition large model text-assisted training method of the first aspect or any implementation thereof, or the speech recognition method of the second aspect.

[0051] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the large-scale text-assisted training method for speech recognition of the first aspect or any implementation thereof, or the speech recognition method of the second aspect.

[0052] By employing the aforementioned technical solution, the text-assisted training method for a large-scale speech recognition model provided in this application acquires a training set, trains trainable parameter entities using the training set to obtain trained parameter entities, and obtains pseudo-audio features corresponding to each text data in the text dataset based on the trained parameter entities. At least based on the text dataset and pseudo-audio features, the parameters of the decoder and text embedding layer in the initial large-scale speech recognition model are trained to obtain the target large-scale speech recognition model. Thus, through learning, the parameter entities can reverse map or simulate text data in the target domain into corresponding pseudo-audio features. These pseudo-audio features can characterize the audio embedding features of the audio corresponding to the text data. Therefore, during the training phase, the input received by the text decoder and text embedding layer is audio embedding features and text data, consistent with the input during the inference phase. The addition of audio modalities during the training phase forces the text decoder and text embedding layer to learn how to better understand, align, and process information from audio modalities in the target domain, effectively bridging the modal gap, reducing the optimization bias of the text decoder, and consequently, when facing real target domain audio during the inference phase, the text decoder can decode more accurately, improving the speech recognition performance of the target large-scale speech recognition model for target domain audio. Attached Figure Description

[0053] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0054] Figure 1 This is a schematic diagram of a typical large-scale speech recognition model based on a large language model.

[0055] Figure 2 A flowchart illustrating a text-assisted training method for a large speech recognition model provided in this application;

[0056] Figure 3 A schematic diagram illustrating the training and application of initial pseudo-audio features;

[0057] Figure 4 This is a schematic diagram illustrating the training and application process of a text encoder.

[0058] Figure 5 A flowchart illustrating a speech recognition method provided in this application;

[0059] Figure 6 A schematic diagram of the structure of a text-assisted training device for a large speech recognition model provided in this application;

[0060] Figure 7A schematic diagram of the structure of a speech recognition device provided in this application;

[0061] Figure 8 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0062] Before introducing the proposed solution, let's first explain the relevant concepts used in this document:

[0063] prompt: Instructions. When interacting with AI (such as artificial intelligence models), you need to send instructions to the AI. These can be a text description, such as "Please recommend a popular song for me" when interacting with AI, or a parameter description in a certain format, such as describing the relevant drawing parameters to ask the AI ​​to draw a picture in a certain format.

[0064] Large language models (LLMs) typically refer to language models with a large number of parameters and capabilities. They learn the statistical regularities and semantic relationships of language through pre-training on large-scale text data. These models often use unsupervised learning methods to predict the next word or fill in missing words to capture the context and semantic information of the language. Large language models can generate coherent sentences, answer questions, and complete translation tasks. A key characteristic of LLMs is their massive scale, containing billions or even more parameters, which help them learn complex patterns in language data. Emergent capabilities of large language models include context learning, instruction following, and progressive reasoning.

[0065] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0066] It should also be noted that the terminology used in the implementation section of this application is only for explaining specific embodiments of this application and is not intended to limit this application. Those skilled in the art will understand that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0067] It should also be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0068] The large speech recognition model used in this application refers to a large speech recognition model based on a large language model, that is, a large speech recognition model using LLM as the decoder.

[0069] See Figure 1 This is a schematic diagram of a typical speech recognition model based on a large language model. This speech recognition model usually includes three core components: an audio encoder that converts audio features into audio embedding features (also known as audio feature embedding vectors), a text embedding layer that maps historically decoded text features (i.e., token sequences, hereinafter referred to as tokens) into text embedding features (also known as text embedding vectors), and an autoregressive decoder (LLM Decoder, hereinafter referred to as text decoder) designed with an LLM architecture.

[0070] It should be noted that there are no restrictions on the specific model structure selection of the core components and the large speech recognition model in this application (such as convolutional neural network CNN, recurrent neural network RNN, Transformer, number of layers, number of nodes, etc.), and the selection can be made according to the actual situation.

[0071] For ease of understanding, the following text will use... Figure 1The structure of the large-scale speech recognition model shown is used as an example for explanation. The speech recognition process of the large-scale speech recognition model is as follows: The speech encoder performs audio embedding encoding on the audio features corresponding to the input audio to obtain audio embedding features. The text embedding layer performs text embedding encoding on the token sequence that has been decoded at each time step to obtain text embedding features at each time step. The text decoder then decodes based on the audio embedding features and the text embedding features at each time step to obtain decoded tokens at each time step (in this embodiment, decoded tokens include at least one decoded token). The decoded tokens at each time step are concatenated with the decoded tokens at previous time steps to form the token sequence that has been decoded at the next time step, which is then sent to the text embedding layer for processing at the next time step. This process continues until the text decoder outputs an end symbol or the maximum number of decoding attempts is reached.

[0072] To fine-tune the aforementioned large-scale speech recognition model, the most common fine-tuning methods are full-parameter fine-tuning and partial-parameter fine-tuning. Full-parameter fine-tuning refers to fine-tuning all parameters of the entire large-scale speech recognition model using massive amounts of labeled speech audio-labeled text data pairs. Partial-parameter fine-tuning involves designing an adapter module within the large-scale speech recognition model, using a small amount of labeled speech audio-labeled text data pairs for domain adaptation, and updating only the adapter's parameters.

[0073] Specifically, for a dataset containing speech-audio-annotated text pairs within the source domain S... , Audio features representing speech audio (such as zero-crossing rate, Mel frequency cepstral coefficients, etc.). This represents the labeled text. Large-scale speech recognition models utilize audio embedding features encoded by an audio encoder within the text embedding space encoded by the text embedding layer. Fine-tuning training is performed using prompts, with decoding tokens obtained through an autoregressive approach using a text decoder. Then, the training is optimized to maximize the given audio embedding features of the prompt. conditional probability (i.e. Training is conducted with the training objective as the training objective.

[0074] However, the full-parameter fine-tuning method suffers from low training efficiency and poor cross-domain adaptability due to the large number of parameters, while the partial-parameter fine-tuning method suffers from moderate training efficiency and general cross-domain adaptability. Furthermore, the forced introduction of the adapter structure will destroy the integrity of the pre-trained representation of the coding layer, resulting in the accumulation of recognition errors due to the offset of speech feature distribution. In addition, the adapter has a bottleneck effect that limits the room for improvement in speech recognition performance.

[0075] Furthermore, both of the above methods require the use of audio. However, in some specific target fields, such as medicine and the military, text data is relatively easy to obtain, but the corresponding audio is difficult to obtain through public channels. The lack of audio makes it difficult for large speech recognition models to be applicable to the target fields.

[0076] To train a large-scale speech recognition model applicable to the target domain (hereinafter referred to as the target speech recognition model), an existing approach involves training the entire large-scale speech recognition model using source domain speech audio-annotated text data, and then retraining the text decoder and text embedding layer of the large-scale speech recognition model using plain text data from the target domain. However, using plain text pre-training, the text decoder can only learn the text feature distribution in the text space. In actual inference, the text decoder needs to process the audio embedding features output by the audio encoder. Since the encoding representation spaces of speech acoustic features and text features are inconsistent, and there is no pre-learning of speech acoustic knowledge in the target neighborhood, the decoder optimization is biased, reducing the speech recognition performance of the large-scale speech recognition model for target domain audio.

[0077] To address the aforementioned issues, a proposed approach was developed: Building upon the previous solution, the language model at the text decoder of the large speech recognition model filters out plain text data with high perplexity in the target domain, synthesizes speech audio based on speech synthesis technology, and forms speech audio-annotated text data pairs for model training.

[0078] However, introducing an additional speech synthesis process increases the overall computational cost and slows down training efficiency. Furthermore, due to the uncontrollable audio quality, monotonous timbre, and overly clean sound quality of synthesized speech, the overall training effect is not significantly improved, and cross-domain adaptability is generally limited.

[0079] To address these issues, this application further provides a speech recognition method and a speech recognition model training method to improve the problems existing in the aforementioned schemes and enhance the recognition performance of the speech recognition model in a specific field.

[0080] The following embodiments of this application provide a text-assisted training method for a large-scale speech recognition model and a speech recognition method, respectively. The provided text-assisted training method for a large-scale speech recognition model can be applied to electronic devices. Exemplarily, the electronic device may be a server, robot, smartphone, personal computer (PC), laptop computer, wireless electronic device in industrial control, etc.

[0081] The method provided in this application can be divided into a training phase and an inference phase. The training phase is the phase of training a large-scale speech recognition model, and the inference phase is the phase of using the trained target speech recognition model to perform speech recognition on the audio to be recognized. The training and inference phases can be deployed on the same device or on different devices. For example, the training phase can be deployed in the cloud or on a server, while the inference phase can be deployed on a smart terminal, such as a mobile phone, tablet, voice recorder, translator, robot, in-vehicle terminal, or wearable device.

[0082] This application provides a text-assisted training method for a large-scale speech recognition model. (Refer to...) Figure 2 This is a flowchart illustrating a text-assisted training method for a large-scale speech recognition model provided in an embodiment of this application. Figure 2 As shown, the text-assisted training method for this large-scale speech recognition model may include:

[0083] Step S201: Obtain the training set, which is a text dataset of the target domain or an audio-text pair dataset of the source domain.

[0084] To address the problem in existing text decoders where inconsistent inputs during training and inference phases lead to optimization biases during training and consequently reduced speech recognition performance, this application provides a method for constructing trainable parametric entities to obtain pseudo-audio features of the target domain. These pseudo-audio features are then used to match training conditions, thereby improving the model's cross-domain adaptive performance.

[0085] In order to train trainable parametric entities, this application can obtain a training set for training the parametric entities. Here, the training set can be a text dataset of the target domain or an audio-text pair dataset of the source domain.

[0086] The source domain refers to a general domain where speech audio can be obtained through public channels. The source domain audio-text pair dataset includes multiple speech audio-annotated text data pairs.

[0087] The target domain refers to the domain in which the large speech recognition model needs to be adapted. Audio in the target domain is difficult to obtain through public channels. For example, conference audio in the medical field is usually kept confidential and is difficult to obtain through public search channels such as the Internet.

[0088] Step S202: Train the trainable parameter entities using the training set to obtain trained parameter entities, and obtain the pseudo-audio features corresponding to each text data in the text dataset based on the trained parameter entities.

[0089] Among them, the pseudo-audio features corresponding to each text data represent the audio embedding features of the audio corresponding to the text data.

[0090] In the embodiments of this application, the parameter entity can be understood as a parameter structure. The parameter entity includes at least one parameter to be optimized. The parameter to be optimized in the parameter entity can be updated through training to obtain the trained parameter entity. The pseudo-audio features obtained based on the trained parameter entity can more accurately simulate the audio embedding features of the target domain output by the audio encoder.

[0091] Optionally, the training set can differ depending on the trainable parameter entity. For example, the trainable parameter entity can be an initial pseudo-audio feature or a text encoder. When the trainable parameter entity is an initial pseudo-audio feature, the training set can be a text dataset from the target domain. When the trainable parameter entity is a text encoder, the training set can be an audio-text pair dataset from the source domain.

[0092] Here, pseudo-audio features are used express, ,in, This represents the sequence length of the pseudo-audio features, and also the number of vectors contained in the pseudo-audio features; This represents the vector dimension of each vector contained in the pseudo-audio feature.

[0093] Optional, This is a preset value based on the module architecture of the audio encoder, text embedding layer, and text decoder, for example, 512.

[0094] Optional, These values ​​are dynamically set based on the speech recognition task in the target domain and the model structure of the large-scale speech recognition model. During the training of the initial pseudo-audio features in this step, To determine the invariable value.

[0095] The text encoder described above is configured to encode the input text into pseudo-audio features that are approximately consistent with the encoding results of the audio encoder for the corresponding audio.

[0096] Step S203: Train the parameters of the text decoder and text embedding layer in the initial speech recognition large model based at least on the text dataset and pseudo-audio features to obtain the target speech recognition large model.

[0097] Since only text datasets are available for the target domain, the core objective is to efficiently adapt these text datasets to the large-scale LLM-based speech recognition model. Considering that large-scale LLM-based speech recognition models differ from traditional speech recognition models, their decoder architecture inherently possesses the ability to process pure text data. Therefore, a text dataset combined with the aforementioned pseudo-audio features can be used to train the parameters of the text decoder LLM and text embedding layer in the model.

[0098] In this embodiment of the application, the large LLM-based speech recognition model that needs to be trained using a text dataset is defined as the initial large speech recognition model. Then, this application can train the parameters of the text decoder and text embedding layer in the initial large speech recognition model based at least on the text dataset and pseudo-audio features to obtain the target large speech recognition model.

[0099] It should be understood that training the text decoder and text embedding layer of the initial large-scale speech recognition model solely based on text datasets and pseudo-audio features may not be sufficient to enable the target large-scale speech recognition model to achieve accurate speech recognition. To ensure that the target large-scale speech recognition model not only possesses general speech recognition capabilities but also adapts to the target domain, optionally, the entire training process of the target large-scale speech recognition model includes both a training phase using text datasets and pseudo-audio features from the target domain and a training phase using audio text from the source domain (i.e., the general domain) to train the dataset.

[0100] Therefore, in one possible implementation, if the initial large-scale speech recognition model is a pre-built large-scale speech recognition model, the text decoder and text embedding layer in the initial large-scale speech recognition model can be trained first based on the text dataset of the target domain and the pseudo-audio features corresponding to all text data in the text dataset to obtain an intermediate large-scale speech recognition model. Then, the intermediate large-scale speech recognition model can be trained based on the audio text dataset of the source domain to obtain the target large-scale speech recognition model. If the initial large-scale speech recognition model is a model pre-trained on the audio text dataset of the source domain, the text decoder and text embedding layer in the initial large-scale speech recognition model can be trained based on the text dataset of the target domain and the pseudo-audio features corresponding to all text data in the text dataset to obtain the target large-scale speech recognition model.

[0101] Of course, there are other training methods, which will not be specifically limited here.

[0102] The text-assisted training method for a large-scale speech recognition model provided in this application involves acquiring a training set, training trainable parameter entities using the training set to obtain trained parameter entities, and obtaining pseudo-audio features corresponding to each text data in the text dataset based on the trained parameter entities. The parameters of the decoder and text embedding layer in the initial large-scale speech recognition model are then trained based on at least the text dataset and the pseudo-audio features to obtain the target large-scale speech recognition model. Thus, the parameter entities, through learning, can reverse map or simulate text data in the target domain into corresponding pseudo-audio features. These pseudo-audio features can characterize the audio embedding features of the audio corresponding to the text data. Therefore, the input received by the text decoder and text embedding layer during the training phase is audio embedding features and text data, consistent with the input during the inference phase. The addition of audio modalities during the training phase forces the text decoder and text embedding layer to learn how to better understand, align, and process information from audio modalities in the target domain, effectively bridging the modal gap, reducing the optimization bias of the text decoder, and consequently, when facing real target domain audio during the inference phase, the text decoder can decode more accurately, improving the speech recognition performance of the target large-scale speech recognition model for target domain audio.

[0103] Furthermore, compared to model training methods such as full-parameter training and partial-parameter training, this application only requires training parameter entities, text decoders, and text embedding layers with fewer parameters, resulting in lower resource consumption and higher training efficiency.

[0104] Furthermore, to be applicable to different domains, different parameter entities can be trained, thereby enabling hot-swappable switching between multiple domains through dynamic parameter entities, resulting in greater freedom.

[0105] In some embodiments of this application, the processes of step S202 "training the trainable parameter entities through the training set to obtain the trained parameter entities, and obtaining the pseudo-audio features corresponding to each text data in the text dataset based on the trained parameter entities" and step S203 "training the parameters of the text decoder and text embedding layer in the initial speech recognition large model according to the text dataset and pseudo-audio features" are described.

[0106] First, we will introduce steps S202 and S203 when the trainable parameter entities are the initial pseudo-audio features.

[0107] To train pseudo-audio features that can represent the average audio embedding features of audio within the target domain, embodiments of this application can remove the audio encoder from the initial large-scale speech recognition model, then embed the initial pseudo-audio features into the audio encoder position to obtain a first model. All parameters in the first model except for the initial pseudo-audio features are frozen. The text dataset is then fed into the first model with the frozen parameters for training, resulting in trained pseudo-audio features. The training objective is to maximize the performance of the pseudo-audio features given the initial pseudo-audio features (or the trained pseudo-audio features). The conditional probability, i.e. .

[0108] See Figure 3 The diagram shown illustrates the training and application process of the initial pseudo-audio features. Figure 3 Black shading indicates parameters that can be optimized for training, while gray shading indicates frozen parameters. For example... Figure 3 On the left, the first model includes initial pseudo-audio features ( The text embedding layer and text decoder are used to input the text dataset of the target domain into the text embedding layer. The text embedding layer performs text embedding encoding on the text data in the text dataset to obtain the text embedding features corresponding to the text data. The text embedding features are then input into the text decoder together with the initial pseudo-audio features to obtain decoded tokens. The loss is calculated by comparing the decoded tokens with the real tokens corresponding to the text data. The parameters of the text embedding layer and text decoder are frozen, so that the initial pseudo-audio features can be trained only on the initial pseudo-audio features based on the calculated loss to obtain trained pseudo-audio features. The trained pseudo-audio features can then be used as the pseudo-audio features corresponding to each text data in the text dataset.

[0109] By optimizing the initial pseudo-audio features as described above, the trained initial pseudo-audio features can learn the target domain-specific audio embedding feature representation, and thus can be used as a voice prompt to guide the plain text training in the subsequent step S203.

[0110] See Figure 3 On the right, in step S203, the trained pseudo-audio features are kept fixed, and the target domain text dataset is input into the text embedding layer. The text embedding layer performs text embedding encoding on the text data in the text dataset to obtain the text embedding features corresponding to the text data. The text embedding features are then input into the text decoder together with the trained pseudo-audio features to obtain decoded tokens. The loss is calculated by comparing the decoded tokens with the real tokens corresponding to the text data. The trained pseudo-audio features ( The parameters of the text decoder and text embedding layer are frozen, so that the parameters can be trained only on the text decoder and text embedding layer based on the calculated loss.

[0111] Since the pseudo-audio features corresponding to each text data in the text dataset are trained based on the text dataset of the target domain, these pseudo-audio features can characterize the average audio embedding features of audio in the target domain. They can be provided as speech prompts to the text decoder for fine-tuning training and optimization iteration, which can reduce the conditional mismatch when the model is trained and inferenced. This makes it easier for the target speech recognition model to obtain domain knowledge from the target domain, improves the cross-domain adaptability of the target speech recognition model from the source domain to the target domain, and improves the accuracy of the model in speech recognition of audio in the target domain.

[0112] It should be noted that after the text decoder and text embedding layer have been fine-tuned on the text dataset, they no longer need to use the trained pseudo-audio features during inference, because real audio embedding features are already available.

[0113] It should also be noted that, in one possible implementation, the above single-task training paradigm can be extended to a multi-task training paradigm. That is, the target domain can include multiple domains, and correspondingly, the trainable initial pseudo-audio features can include initial pseudo-audio features corresponding to multiple domains respectively. Thus, in the pseudo-audio feature training stage of step S202, multiple different initial pseudo-audio features can be trained to obtain trained pseudo-audio features corresponding to multiple domains respectively. Then, in the text decoder and text embedding layer training stage of step S203, the parameters of the text decoder and text embedding layer in the initial speech recognition large model can be trained together based on the trained pseudo-audio features corresponding to multiple domains and the text datasets of multiple domains. This multi-task training paradigm enables the target speech recognition large model to be applicable to multiple domains simultaneously, improving the practicality of the target speech recognition large model.

[0114] In one possible implementation, the aforementioned initial pseudo-audio features can be preset pseudo-audio features, i.e., set according to the speech recognition task in the target domain and the model structure of the large speech recognition model. and set Then, initial pseudo-audio features that meet the requirements are either manually preset or randomly generated.

[0115] Given that sentence lengths may vary across different domains, the pseudo-audio feature length can be adjusted as a model training hyperparameter to align the trained pseudo-audio features with the preferred lengths for each domain, thereby optimizing the adaptation effect to the target domain text. To this end, this application also provides another method for training the pseudo-audio feature length to determine a more preferred initial pseudo-audio feature.

[0116] In this embodiment, the text embedding layer of the first speech recognition large model can be used to perform text embedding encoding on each text data in the text dataset to obtain the text embedding feature corresponding to each text data. Then, the target length is determined according to the text embedding features corresponding to all text data in the text dataset. Here, the first speech recognition large model is the initial speech recognition large model, or the initial speech recognition large model is obtained by training the first speech recognition large model. The target length refers to the pseudo-audio feature length.

[0117] Optionally, the process of "determining the target length based on the text embedding features corresponding to all text data in the text dataset" is determined by calculating the centroid of the text dataset.

[0118] First, based on the text embedding features corresponding to each text data in the text dataset, determine the centroid vector corresponding to each text data. One possible implementation is to calculate the mean of each vector contained in the text embedding features corresponding to each text data in each dimension to obtain the first mean vector, which is used as the centroid vector corresponding to each text data.

[0119] Taking a text embedding feature corresponding to a text data as an example, which includes 10 vectors, each vector having 512 dimensional values, we can take the average of the first dimensional values ​​of each of the 10 vectors to obtain the first dimensional value in the first mean vector. Then, we can take the average of the second dimensional values ​​of each of the 10 vectors to obtain the second dimensional value in the first mean vector. This process continues until we take the average of each of the 512 dimensional values ​​of the 10 vectors to obtain the first mean vector. This first mean vector is the centroid vector corresponding to the text data.

[0120] Next, the target centroid vector is determined based on the centroid vectors corresponding to all text data. Optionally, similar to the implementation of the centroid vectors corresponding to the text data, this embodiment can calculate the average of the centroid vectors corresponding to all text data in the text dataset along each dimension to obtain a second mean vector, which serves as the target centroid vector. This target centroid vector represents the average centroid level of the text dataset.

[0121] Therefore, we can identify the text data whose centroid vector is most similar to the target centroid vector from the text dataset and use it as the target text data. The number of lexical units corresponding to this target text data is the text data that is closest to the average number of lexical units in the target domain. Therefore, the number of lexical units corresponding to the target text data can be determined as the target length.

[0122] The target length mentioned above can be considered as the optimal length of the pseudo-audio features in the target domain. Therefore, in this embodiment, the first pseudo-audio features need to be processed to obtain the second pseudo-audio features with a sequence length of the target length.

[0123] It should be noted that the first pseudo-audio feature in the first iteration (see the following description, there may be at least one return step, resulting in a second iteration, a third iteration, etc.) is a preset pseudo-audio feature. The preset method can refer to the preset method of the initial pseudo-audio feature mentioned above, and will not be repeated here.

[0124] Optionally, the process of "processing the first pseudo-audio feature to obtain a second pseudo-audio feature with a sequence length equal to the target length" may include: determining the length difference between the target length of the current iteration and the target length of the previous iteration. When the current iteration is the first iteration, the target length of the previous iteration is the sequence length of the preset pseudo-audio feature. If the length difference is positive, a sub-feature with a sequence length equal to the length difference is extracted from the first pseudo-audio feature, and the sub-feature is concatenated with the first pseudo-audio feature to obtain the second pseudo-audio feature. If the length difference is negative, the absolute value of the length difference is calculated, and the sub-feature with the absolute value of the sequence length is discarded from the first pseudo-audio feature to obtain the second pseudo-audio feature.

[0125] For example, if the target length in the first iteration is 5 and the target length in the second iteration is 3, then the length difference is -2. Then, the sub-features with a sequence length of 2 can be randomly discarded from the first pseudo-audio feature to obtain the second pseudo-audio feature. For example, if the first pseudo-audio feature is [0.85, 0.12, -0.45, 0.98, 0.01], and assuming that 0.85 and 0.12 are discarded, then the second pseudo-audio feature is [-0.45, 0.98, 0.01].

[0126] For example, if the target length in the first iteration is 5 and the target length in the second iteration is 8, then the length difference is 3. Therefore, a sub-feature of length 3 can be randomly copied from the first pseudo-audio feature to obtain the second pseudo-audio feature. For example, if the first pseudo-audio feature is [0.85, 0.12, -0.45, 0.98, 0.01], and assuming 0.12, -0.45, and 0.98 are copied, then the second pseudo-audio feature would be [0.85, 0.12, -0.45, 0.98, 0.01, 0.12, -0.45, 0.98].

[0127] It should be noted that the above method of concatenating sub-features after the first pseudo-audio feature is only an example. Other concatenation methods are also possible. In addition, the process of discarding or copying sub-features from the first pseudo-audio feature can also be done in other ways, such as non-contiguous discarding or copying, etc. This application does not impose specific limitations.

[0128] By discarding or copying the features mentioned above, the sequence length of the second pseudo-audio feature can be adjusted to the optimal value. However, the parameter values ​​in the second pseudo-audio feature may not be optimal. Therefore, it is necessary to start a training round based on the second pseudo-audio feature to obtain the trained third pseudo-audio feature and the second large-scale speech recognition model. Here, the training round includes training the second pseudo-audio feature and training the text decoder and text embedding layer in the first large-scale speech recognition model.

[0129] The training process for the second pseudo-audio feature can refer to the training process for the initial pseudo-audio feature in step S202 above. The training process for the text decoder and text embedding layer in the first speech recognition large model can refer to the training process for the text decoder and text embedding layer in the initial speech recognition large model in step S203 above. It will not be repeated here.

[0130] Next, the second large speech recognition model is used as the first large speech recognition model, and the third pseudo-audio feature is used as the first pseudo-audio feature. The text embedding layer of the first large speech recognition model is used to perform text embedding encoding on each text data in the text dataset until the preset iteration end condition is reached and then the return stops. Then, the initial pseudo-audio feature can be determined based on the third pseudo-audio feature or the first pseudo-audio feature after the iteration end condition is reached.

[0131] Optionally, the above iteration termination condition can be a model training termination condition or a pseudo-audio feature training termination condition, such as the loss between the decoded tokens output by the text decoder and the real tokens corresponding to the text data being less than a preset loss threshold.

[0132] Optionally, the above iteration termination condition can also be that the absolute value of the length difference between two adjacent iterations of the target length is less than a preset length difference threshold.

[0133] Of course, the above-mentioned iteration termination condition can also be other, and this application does not impose specific limitations.

[0134] Once the iteration termination condition is met, either the third pseudo-audio feature or the first pseudo-audio feature can be directly used as the initial pseudo-audio feature. Optionally, after the iteration termination condition is met, if the latest pseudo-audio feature is the third pseudo-audio feature, it can be used as the first pseudo-audio feature again, and the process can return to the step of text embedding encoding for each text data in the text dataset through the text embedding layer of the first speech recognition model for one iteration, stopping when the second pseudo-audio feature is obtained, and using the second pseudo-audio feature as the initial pseudo-audio feature.

[0135] By using the pseudo-audio feature length as a hyperparameter for hyperparameter training, it can be ensured that the sequence length of the initial pseudo-audio features is the optimal length for adapting to the target domain. Based on the initial pseudo-audio features of this optimal length, the parameters of the text decoder and text embedding layer are trained, enabling the trained text decoder to decode more accurately in the target domain, thereby improving the speech recognition performance of the target speech recognition model for audio in the target domain.

[0136] In addition to the above-mentioned methods for training pseudo-audio features, this application also provides a method for training a text encoder. The process of steps S202 and S203 when the trainable parameter entity is a trainable text encoder is described below.

[0137] To ensure that the pseudo-audio features represented by the text semantics of the target domain are approximately equivalent to the audio embedding features represented by the corresponding real audio, this application constructs a text encoder parallel to the audio encoder, such as... Figure 4 The diagram on the left. Next, the text and audio from the audio-text pair dataset are fed into the text encoder and the audio encoder in the initial large-scale speech recognition model, respectively, to obtain the output features of the text encoder and the audio encoder. Here, the output features of the audio encoder are the audio embedding features.

[0138] Calculate the feature loss between the output features of the text encoder and the audio encoder, such as KL divergence loss, then freeze the parameters of the audio encoder, and train the parameters of the text encoder based on the feature loss to bring the output distributions of the two encoders closer together. This achieves the goal that the pseudo-audio features represented by the text semantics of the target domain can be approximately equivalent to the audio embedding features represented by the corresponding real audio.

[0139] like Figure 4 The diagram on the right illustrates how, after obtaining the trained text encoder, each text data point in the text dataset can be fed into it to obtain the corresponding pseudo-audio features. Simultaneously, each text data point is also fed into a text embedding layer for text embedding encoding, yielding text embedding features for each text data point. Finally, the text embedding features and pseudo-audio features are fed into the text decoder to obtain decoded tokens. The loss between the decoded tokens and the ground truth tokens is calculated, and the parameters of the text encoder are frozen. This means the calculated loss can be used to train the parameters of the text decoder and the text embedding layer.

[0140] Because the text encoder encodes each piece of text data in the target domain to obtain pseudo-audio features, these pseudo-audio features are more closely aligned with the corresponding specific text, and thus closer to the audio embedding features of the audio corresponding to the text data. Furthermore, by directly encoding the text data in the target domain to obtain pseudo-audio features, the sequence length of these pseudo-audio features can be dynamically adjusted according to the length of the text data, better matching the training conditions of real-world paired datasets. Therefore, large-scale target speech recognition models trained using the text encoder method exhibit more stable training and stronger cross-domain adaptability.

[0141] This application also provides a speech recognition method, see [link to relevant documentation] Figure 5 This is a flowchart illustrating the speech recognition method provided in this application. Figure 5 As shown, the speech recognition method may include:

[0142] Step S501: Obtain the audio to be recognized in the target area.

[0143] Step S502: Input the audio to be recognized into the configured target speech recognition model to obtain the audio recognition text output by the model.

[0144] The target speech recognition large model is obtained by training the text decoder and text embedding layer in the initial speech recognition large model with parameters based on the text dataset and the pseudo-audio features corresponding to each text data in the text dataset. The pseudo-audio features corresponding to each text data are obtained based on the parameter entities trained on the training set. The pseudo-audio features corresponding to each text data represent the audio embedding features of the audio corresponding to the text data. The training set is a text dataset in the target domain or an audio-text pair dataset in the source domain.

[0145] It should be noted that the specific training process of the target speech recognition large model provided in this application can be referred to the text-assisted training method for speech recognition large models described above, and will not be repeated here.

[0146] The above describes a text-assisted training method for a large speech recognition model provided by the embodiments of this application. The following will describe the apparatus for performing the above-described text-assisted training method for a large speech recognition model.

[0147] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a text-assisted training device for a large-scale speech recognition model provided in an embodiment of this application. Figure 6 As shown, the text-assisted training device for the large speech recognition model may include:

[0148] The first data input unit 601 is used to acquire a training set, which is a text dataset in the target domain or an audio-text pair dataset in the source domain.

[0149] The first training unit 602 is used to train the trainable parameter entities through the training set to obtain the trained parameter entities. Based on the trained parameter entities, the pseudo-audio features corresponding to each text data in the text dataset are obtained. The pseudo-audio features corresponding to each text data represent the audio embedding features of the audio corresponding to the text data.

[0150] The second training unit 603 is used to train the parameters of the text decoder and text embedding layer in the initial speech recognition large model based at least on the text dataset and pseudo-audio features, so as to obtain the target speech recognition large model.

[0151] Each module in the aforementioned large-scale speech recognition text-assisted training device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0152] The above also introduced a speech recognition method provided by the embodiments of this application. The following will describe the apparatus for performing the above speech recognition method.

[0153] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a speech recognition device provided in an embodiment of this application. Figure 7 As shown, the voice recognition device may include:

[0154] The second data input unit 701 is used to acquire the audio to be recognized in the target area;

[0155] The data processing unit 702 is used to send the audio to be recognized into the configured target speech recognition large model to obtain the audio recognition text output by the model. The target speech recognition large model is obtained by training the parameters of the text decoder and text embedding layer in the initial speech recognition large model based on the text dataset and the pseudo-audio features corresponding to each text data in the text dataset. The pseudo-audio features corresponding to each text data are obtained based on the parameter entities trained on the training set. The pseudo-audio features corresponding to each text data represent the audio embedding features of the audio corresponding to the text data. The training set is a text dataset in the target domain or an audio-text pair dataset in the source domain.

[0156] Each module in the aforementioned speech recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0157] This application also provides an electronic device, which may include at least one processor and a memory connected to the processor, wherein:

[0158] Memory is used to store computer programs;

[0159] The processor is used to execute computer programs to enable the electronic device to implement any of the large-scale speech recognition model text-assisted training methods or speech recognition methods provided in the embodiments of this application.

[0160] refer to Figure 8 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 8 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0161] like Figure 8 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. When the electronic device is powered on, the RAM 803 also stores various programs and data required for the operation of the electronic device. The processing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0162] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, memory cards, hard drives, etc.; and communication devices 809. Communication device 809 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0163] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the large-scale speech recognition model text-assisted training methods or speech recognition methods provided in this application.

[0164] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the large-scale speech recognition model text-assisted training methods or speech recognition methods provided in this application.

[0165] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0166] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0167] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0168] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A text-assisted training method for a large-scale speech recognition model, characterized in that, include: Obtain a training set, which is a text dataset of the target domain or an audio-text pair dataset of the source domain; The trainable parameter entities are trained using the training set to obtain the trained parameter entities; Based on the trained parameter entities, pseudo-audio features corresponding to each text data in the text dataset are obtained, and each pseudo-audio feature corresponding to the text data represents the audio embedding feature of the audio corresponding to the text data. The text decoder and text embedding layer in the initial speech recognition large model are trained with parameters based at least on the text dataset and the pseudo-audio features to obtain the target speech recognition large model.

2. The text-assisted training method for a large-scale speech recognition model according to claim 1, characterized in that, The trainable parameter entity is the initial pseudo-audio feature or text encoder. When the trainable parameter entity is the initial pseudo-audio feature, the training set is the text dataset; When the trainable parameter entity is the text encoder, the training set is the audio-text pair dataset.

3. The text-assisted training method for a large-scale speech recognition model according to claim 2, characterized in that, When the trainable parameter entity is the initial pseudo-audio feature, the step of training the trainable parameter entity using the training set to obtain the trained parameter entity includes: The audio encoder in the initial speech recognition model is removed, and the initial pseudo-audio features are then embedded into the audio encoder position to obtain the first model. Freeze all parameters in the first model except for the initial pseudo-audio features, and feed the text dataset into the first model after freezing the parameters for training to obtain the trained pseudo-audio features. The step of obtaining the pseudo-audio features corresponding to each text data in the text dataset based on the trained parameter entities includes: The trained pseudo-audio features are used as the pseudo-audio features corresponding to each text data in the text dataset.

4. The text-assisted training method for a large-scale speech recognition model according to claim 2 or 3, characterized in that, The process of determining the initial pseudo-audio features includes: The first speech recognition big model performs text embedding encoding on each text data in the text dataset through the text embedding layer of the first speech recognition big model to obtain the text embedding feature corresponding to each text data. The first speech recognition big model is the initial speech recognition big model, or the first speech recognition big model is trained to obtain the initial speech recognition big model. The target length is determined based on the text embedding features corresponding to all text data in the text dataset. The first pseudo-audio feature is processed to obtain a second pseudo-audio feature with a sequence length equal to the target length, wherein the first pseudo-audio feature in the first iteration is a preset pseudo-audio feature; A training round is initiated based on the second pseudo-audio feature to obtain the trained third pseudo-audio feature and the second large speech recognition model. The training round includes training the second pseudo-audio feature and training the text decoder and text embedding layer in the first large speech recognition model. The second speech recognition model is used as the first speech recognition model, and the third pseudo-audio feature is used as the first pseudo-audio feature. The text embedding layer of the first speech recognition model is used to perform text embedding encoding on each text data in the text dataset until the preset iteration end condition is met and then the return stops. The initial pseudo-audio feature is determined based on the third pseudo-audio feature or the first pseudo-audio feature after the iteration termination condition is met.

5. The text-assisted training method for a large speech recognition model according to claim 4, characterized in that, The step of determining the target length based on the text embedding features corresponding to all text data in the text dataset includes: Based on the text embedding features corresponding to each text data in the text dataset, determine the centroid vector corresponding to each text data; Determine the target centroid vector based on the centroid vectors corresponding to all text data. The text data whose centroid vector is most similar to the target centroid vector is determined from the text dataset and used as the target text data; The number of lexical units corresponding to the target text data is determined as the target length.

6. The text-assisted training method for a large speech recognition model according to claim 5, characterized in that, The step of determining the centroid vector corresponding to each text data based on the text embedding features corresponding to each text data in the text dataset includes: The average value of each vector contained in the text embedding feature corresponding to each text data is calculated in each dimension to obtain the first mean vector, which is used as the centroid vector corresponding to each text data. The step of determining the target centroid vector based on the centroid vectors corresponding to all text data includes: The centroid vectors corresponding to all the text data are averaged in each dimension to obtain a second mean vector, which is used as the target centroid vector.

7. The text-assisted training method for a large speech recognition model according to claim 4, characterized in that, The process of processing the first pseudo-audio feature to obtain a second pseudo-audio feature with a sequence length equal to the target length includes: Determine the length difference between the target length of the current iteration and the target length of the previous iteration. When the current iteration is the first iteration, the target length of the previous iteration is the sequence length of the preset pseudo-audio feature. If the length difference is positive, then a sub-feature with a sequence length equal to the length difference is extracted from the first pseudo-audio feature, and the sub-feature is concatenated with the first pseudo-audio feature to obtain the second pseudo-audio feature; If the length difference is negative, then the absolute value of the length difference is calculated, and the sub-features with a sequence length equal to the absolute value are discarded from the first pseudo-audio features to obtain the second pseudo-audio features.

8. The text-assisted training method for a large speech recognition model according to claim 2, characterized in that, When the trainable parameter entity is the text encoder, the step of training the trainable parameter entity using the training set to obtain the trained parameter entity includes: The text and audio from the audio-text pair dataset are fed into the text encoder and the audio encoder in the initial speech recognition large model, respectively, to obtain the output features of the text encoder and the audio encoder, respectively. Calculate the feature loss between the output features of the text encoder and the audio encoder respectively; The text encoder is trained using the feature loss to obtain the trained text encoder. The step of obtaining the pseudo-audio features corresponding to each text data in the text dataset based on the trained parameter entities includes: Each text data in the text dataset is fed into the trained text encoder to obtain the pseudo-audio features corresponding to each text data in the text dataset.

9. A speech recognition method, characterized in that, include: Acquire the audio to be identified in the target area; The audio to be recognized is fed into the configured target speech recognition large model to obtain the audio recognition text output by the model. The target speech recognition large model is obtained by training the text decoder and text embedding layer in the initial speech recognition large model with parameters based on the text dataset and the pseudo-audio features corresponding to each text data in the text dataset. The pseudo-audio features corresponding to each text data are obtained based on the parameter entities trained on the training set. The pseudo-audio features corresponding to each text data represent the audio embedding features of the audio corresponding to the text data. The training set is the text dataset of the target domain or the audio-text pair dataset of the source domain.

10. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the large-scale speech recognition model text-assisted training method as described in any one of claims 1 to 8, or the speech recognition method as described in claim 9.

11. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the large-scale text-assisted training method for speech recognition as described in any one of claims 1 to 8, or the speech recognition method as described in claim 9.

12. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the large-scale speech recognition model text-assisted training method as described in any one of claims 1 to 8, or the speech recognition method as described in claim 9.

Images