Speech recognition and model training method, device, equipment and computer program product

Abstract

The present application discloses a speech recognition and model training method, apparatus, device and computer program product, which relates to the field of speech recognition technology. The present application jointly trains a speech recognition model and a text reconstruction model, and the two share the same decoder. The text reconstruction model also includes a text encoder built based on a large language model, which can extract text semantic representations for recognized text labels, calculate the feature alignment loss value between the audio semantic representation extracted by the audio encoder for the audio sample and the text semantic representation, and calculate the text recognition loss value between the text output by the decoder and the recognized text label. The large language model can extract rich semantic information, and the feature alignment loss value can be used to migrate the rich semantic information of the large language model to the audio encoder, thereby improving the recognition effect of the speech recognition model in specific fields. In the inference stage, the acoustic features of the speech signal to be recognized are input into the speech recognition model to obtain the speech recognition result output by the model.

Description

[0001] This application claims priority to a domestic application filed with the Patent Office of China on September 25, 2024, with application number 202411338067.6 and invention name “Speech recognition and model training method, device, equipment and computer program product”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of speech recognition technology, and more specifically, to a speech recognition and model training method, apparatus, device, and computer program product. Background Art

[0003] In recent years, with the development of intelligent voice technology, end-to-end speech recognition systems have become a research hotspot and have been widely used in scenarios such as voice assistants, conference transcription, and call transcription.

[0004] Compared to traditional speech recognition systems, end-to-end speech recognition systems implicitly integrate modules such as acoustic models, phonetic models, and pronunciation dictionaries, directly learning the mapping relationship from speech signals to text. This simplifies the training and inference process and avoids the error accumulation of cascaded systems. With the rise of deep learning technology, particularly the application of deep neural networks (DNNs), the accuracy of end-to-end speech recognition systems has significantly improved compared to traditional speech recognition systems. However, end-to-end speech recognition systems still face many challenges when applied to specific (professional) fields such as medicine, law, and finance.

[0005] Since end-to-end speech recognition models rely on a large amount of training data to achieve good results, for some specific fields, on the one hand, due to the long-tail effect of data, the proportion of specific field data in the training data is usually extremely low. On the other hand, it is particularly difficult to obtain sufficient amount of speech text data in specific fields. Therefore, when facing specific fields, the recognition effect of the end-to-end speech recognition model will be greatly reduced. Summary of the Invention

[0006] In view of the above problems, this application is proposed to provide a speech recognition and model training method, apparatus, device and computer program product to improve the recognition effect of speech recognition models in specific fields. The specific solution is as follows:

[0007] In a first aspect of the present application, a speech recognition method is provided, comprising:

[0008] Acquiring acoustic features of a speech signal to be recognized;

[0009] The acoustic features are input into the trained speech recognition model to obtain the speech recognition results output by the model; wherein:

[0010] The speech recognition model is jointly trained with the text reconstruction model during the training phase. The speech recognition model and the text reconstruction model share the same decoder. The text reconstruction model also includes a text encoder constructed based on a large language model. The joint training process uses audio samples as the input of the audio encoder in the speech recognition model, and the recognized text labels corresponding to the audio samples as the input of the text encoder. The total loss value of the joint training includes a text recognition loss value calculated based on the text output by the decoder and the recognized text labels, and a feature alignment loss value calculated based on the audio semantic representation extracted by the audio encoder and the text semantic representation extracted by the text encoder.

[0011] A second aspect of the present application provides a speech recognition model training method, comprising:

[0012] Acquire audio-text pair training data, wherein the audio-text pair training data includes audio samples and corresponding recognition text labels;

[0013] The audio sample is used as input to an audio encoder in a speech recognition model, and the recognized text label is used as input to a text encoder constructed based on a large language model in a text reconstruction model, wherein the text reconstruction model and the speech recognition model share the same decoder, the audio encoder is used to extract an audio semantic representation and feed it into the decoder with a first probability, the text encoder is used to extract a text semantic representation and feed it into the decoder with a second probability, and the decoder is used to output a decoded text, where the sum of the first probability and the second probability is equal to 1;

[0014] Calculating a text recognition loss value based on the text output by the decoder and the recognized text label, calculating a feature alignment loss value based on the audio semantic representation and the text semantic representation, and calculating a total loss value of the text recognition loss value and the feature alignment loss value;

[0015] The model parameters are updated according to the total loss value until the training end condition is reached, thereby obtaining a trained speech recognition model.

[0016] In one possible design, in another implementation of the second aspect of the embodiment of the present application, the speech recognition model also includes a shared encoder located between the audio encoder and the decoder, which encodes the input features and sends the encoded features to the decoder; the audio semantic representation extracted by the audio encoder is sent to the shared encoder with a first probability, and the text semantic representation extracted by the text encoder is sent to the shared encoder with a second probability.

[0017] In one possible design, in another implementation of the second aspect of the embodiments of the present application, the speech recognition model further includes an alignment module;

[0018] When the input of the shared encoder is the audio semantic representation, the alignment module is used to predict audio-text alignment information based on the audio semantic representation;

[0019] The total loss value also includes: a temporal classification CTC loss calculated based on the audio-text alignment information and the recognized text label.

[0020] In one possible design, in another implementation of the second aspect of the embodiments of the present application, the text encoder includes a large language model, a resampling module, and a time length prediction module;

[0021] The large language model is used to extract an initial text semantic representation from the input text;

[0022] When the input of the shared encoder is the audio semantic representation, the resampling module is used to resample the initial text semantic representation based on the audio-text alignment information output by the alignment module to obtain a resampled text semantic representation with the same number of frames as the audio semantic representation, and calculate the feature alignment loss value using the resampled text semantic representation and the audio semantic representation;

[0023] The duration prediction module is used to predict the duration of each frame in the initial text semantic representation to obtain duration prediction information;

[0024] The total loss value also includes: a duration prediction loss value calculated based on the duration prediction information and the audio-text alignment information.

[0025] In one possible design, in another implementation of the second aspect of the embodiments of the present application, after updating the model parameters according to the total loss value until the training end condition is reached, the method further includes:

[0026] Obtain text corpus in the target field;

[0027] Sending the text corpus into the large language model to obtain a target text semantic representation of the text corpus;

[0028] Predicting the duration of each frame in the semantic representation of the target text by the duration prediction module to obtain target duration prediction information;

[0029] Resampling the target text semantic representation according to the target duration prediction information through the resampling module to obtain a resampled text semantic representation;

[0030] Encoding the resampled text semantic representation through the shared encoder, sending the encoded features to the decoder, and predicting the decoded text through the decoder;

[0031] A text reconstruction loss value is calculated based on the decoded text predicted by the decoder and the text corpus, and the model parameters are updated according to the text reconstruction loss value until the training end condition is reached. The final speech recognition model is composed of the audio encoder, the shared encoder and the decoder.

[0032] In one possible design, in another implementation of the second aspect of the embodiments of the present application, after obtaining the target duration prediction information, the method further includes:

[0033] The target duration prediction information is adjusted according to the configured duration disturbance parameter to obtain disturbed target duration prediction information to guide the resampling module to perform resampling.

[0034] In one possible design, in another implementation of the second aspect of the embodiments of the present application, the process of obtaining text corpus in the target domain includes:

[0035] The first prompt instruction prompt is sent to the large language model to obtain the text corpus of the target domain generated by the large language model, wherein the first prompt instruction is used to instruct the model to generate the text corpus of the target domain.

[0036] A third aspect of the present application provides a speech recognition device, comprising:

[0037] An acoustic feature acquisition unit, configured to acquire acoustic features of a speech signal to be recognized;

[0038] A model recognition unit is used to input the acoustic features into the trained speech recognition model to obtain a speech recognition result output by the model; wherein:

[0039] The speech recognition model is jointly trained with the text reconstruction model during the training phase. The speech recognition model and the text reconstruction model share the same decoder. The text reconstruction model also includes a text encoder constructed based on a large language model. The joint training process uses audio samples as the input of the audio encoder in the speech recognition model, and the recognized text labels corresponding to the audio samples as the input of the text encoder. The total loss value of the joint training includes a text recognition loss value calculated based on the text output by the decoder and the recognized text labels, and a feature alignment loss value calculated based on the audio semantic representation extracted by the audio encoder and the text semantic representation extracted by the text encoder.

[0040] A fourth aspect of the present application provides a speech recognition model training device, comprising:

[0041] A training data acquisition unit, configured to acquire audio-text pair training data, wherein the audio-text pair training data includes audio samples and corresponding recognition text labels;

[0042] a model calculation unit, configured to use the audio sample as input to an audio encoder in a speech recognition model, and to use the recognized text label as input to a text encoder constructed based on a large language model in a text reconstruction model, wherein the text reconstruction model and the speech recognition model share the same decoder, the audio encoder is configured to extract an audio semantic representation and feed the representation into the decoder with a first probability, the text encoder is configured to extract a text semantic representation and feed the representation into the decoder with a second probability, the decoder is configured to output decoded text, and the sum of the first probability and the second probability is equal to 1;

[0043] a loss value calculation unit, configured to calculate a text recognition loss value based on the text output by the decoder and the recognized text label, calculate a feature alignment loss value based on the audio semantic representation and the text semantic representation, and calculate a total loss value of the text recognition loss value and the feature alignment loss value;

[0044] A parameter updating unit is used to update the model parameters according to the total loss value until the training end condition is reached to obtain a trained speech recognition model.

[0045] In a fifth aspect of the present application, an electronic device is provided, comprising: a memory and a processor;

[0046] The memory is used to store programs;

[0047] The processor is used to execute the program to implement the speech recognition method described in the first aspect of this application, or to implement the various steps of the speech recognition model training method described in any one of the second aspects of this application.

[0048] The sixth aspect of the present application provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements the speech recognition method described in the first aspect of the present application, or implements the various steps of the speech recognition model training method described in any one of the second aspects of the present application.

[0049] The seventh aspect of the present application provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the speech recognition method described in the first aspect of the present application, or implements the various steps of the speech recognition model training method described in any one of the second aspects of the present application.

[0050] With the above technical solution, the present application adopts a joint training strategy to jointly train the speech recognition model and the text reconstruction model. The two share the same decoder. The text reconstruction model also includes a text encoder constructed based on a large language model, which can extract text semantic representations for the recognized text labels corresponding to the audio samples. During the training process, the feature alignment loss value between the audio semantic representation extracted by the audio encoder for the audio sample and the text semantic representation is calculated, and the text recognition loss value between the text output by the decoder and the recognized text label is calculated. The semantic representations of audio and text are unified into the same semantic representation space through the feature alignment loss value. At the same time, since the large language model can extract rich semantic information, the rich semantic information of the large language model can be migrated to the audio encoder through the feature alignment loss value, so that the audio encoder can extract richer audio semantic representations, thereby improving the domain customization capability of the speech recognition model, that is, improving the recognition effect of the trained speech recognition model in a specific domain. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present application. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0052] Figure 1 A flowchart of a speech recognition model training method provided in an embodiment of the present application;

[0053] Figure 2 This example shows a diagram of a speech recognition model training architecture.

[0054] Figure 3 This example illustrates another diagram of the speech recognition model training architecture.

[0055] Figure 4 This example illustrates another diagram of the speech recognition model training architecture;

[0056] Figure 5 This example illustrates another diagram of the speech recognition model training architecture;

[0057] Figure 6 A schematic diagram of the structure of a speech recognition device provided in an embodiment of the present application;

[0058] Figure 7 A schematic diagram of the structure of a speech recognition model training device provided in an embodiment of the present application;

[0059] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] Before introducing this application solution, we first explain the relevant concepts involved in this article:

[0061] Prompt: Instructions. When interacting with an AI (such as an artificial intelligence model), you need to send instructions to the AI. This can be a text description, such as "Please recommend me a pop song" when interacting with the AI, or it can be a parameter description in a specific format, such as describing the relevant drawing parameters to ask the AI ​​to draw a drawing according to a certain format.

[0062] Large language models (LLMs) generally refer to language models with a large number of parameters and capabilities. They learn the statistical laws and semantic relationships of language by pre-training on large amounts of text data. These models typically use unsupervised learning methods to predict the next word or fill in missing words to capture the context and semantic information of the language. Large language models are capable of generating coherent sentences, answering questions, and completing translation tasks. LLMs are characterized by their large scale, containing billions or even more parameters, which help them learn complex patterns in language data. Emerging capabilities of large language models include contextual learning, instruction following, and step-by-step reasoning.

[0063] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0064] Traditional end-to-end speech recognition systems face many challenges when applied to specific fields such as medicine, law, finance, and other professional fields.

[0065] These challenges mainly stem from the following aspects: First, since the terms and concepts in professional fields are often very different from everyday language, they appear very rarely in the training data, resulting in the end-to-end speech recognition system usually unable to accurately identify these professional terms and concepts (such as diseases and drugs in the medical field, professional terms in the financial field, etc.); Second, although it is possible to construct domain-specific labeled data to perform domain adaptation training on the end-to-end speech recognition system, the labeling of domain-specific data requires huge costs, and some fields are highly open, and the labeled data is difficult to cover all domain scenarios, resulting in insufficient generalization ability of the model when facing domain scenarios.

[0066] To improve the recognition performance of end-to-end speech recognition systems in specific domains, one solution is to train an external language model using a text corpus from the target domain and integrate it into the end-to-end speech recognition model during the inference (decoding) process. During the decoding phase, the probabilities of the external language model are combined with those of the end-to-end speech recognition model to increase the probability of decoding results relevant to the target domain, thereby improving the recognition performance of the end-to-end speech recognition system in the target domain.

[0067] While this approach can improve recognition performance in the target domain, it involves using an external language model during inference, introducing a more complex model architecture, significantly increasing the computational cost of the decoding process, and slowing down inference speed, making it unsuitable for offline devices with limited computing resources. Furthermore, this approach requires tuning additional hyperparameters, increasing the complexity of training and deploying end-to-end speech recognition systems.

[0068] To avoid introducing an external language model (LM) during the decoding process and thus increasing computational costs, one solution is to use text-to-speech (TTS) technology to generate paired speech and text data based on the target domain text library, and then use this data to train or fine-tune the end-to-end speech recognition system to improve the recognition performance of the end-to-end speech recognition system in the target domain.

[0069] However, the above solutions require the introduction of additional TTS systems. The generated voice data is usually based on rules or deep learning algorithms, which cannot fully simulate the voice changes and complexity in the real world. The authenticity and diversity of the data are significantly lower than those of real audio data, resulting in a decrease in the effectiveness of the voice recognition system in real-world applications.

[0070] To this end, the present application further provides a speech recognition method and a speech recognition model training method to improve at least some of the problems existing in the aforementioned examples and enhance the recognition effect of the speech recognition model in specific fields.

[0071] The following embodiments of this application provide a speech recognition method and a speech recognition model training method. The provided speech recognition model training method can be applied to electronic devices. For example, the electronic devices may include servers, robots, smartphones, personal computers (PCs), laptops, and wireless electronic devices used in industrial control systems.

[0072] The method provided in this application can be divided into a training phase and an inference phase. The training phase is the phase of training the speech recognition model, and the inference phase is the phase of using the trained speech recognition model to perform speech recognition on the speech signal to be recognized. The training phase and the inference phase can be deployed in the same device or in different devices. For example, the training phase can be deployed in the cloud or on a server, and the inference phase can be deployed in an intelligent terminal, such as a mobile phone, tablet, voice recorder, translator, robot, vehicle-mounted terminal, or wearable device.

[0073] For ease of understanding, this application introduces the processes of the training phase and the inference phase respectively.

[0074] 1. Training Phase

[0075] It should be noted that the training process of the model can usually be divided into multiple iterations or multiple executions. This application exemplarily introduces one of the iterations or execution processes as an example. The steps mentioned below can be repeated and will not be repeated below.

[0076] See Figure 1 The present application provides a flow chart of a method for training a speech recognition model, as described below. The method includes:

[0077] Step S100: Acquire audio-text pair training data, where the audio-text pair training data includes audio samples and corresponding recognition text labels.

[0078] Specifically, audio samples and paired recognition text labels can be collected to form audio-text pair training data.

[0079] Step S110: Use the audio text to jointly train the speech recognition model and the text reconstruction model based on the training data. The two models share the same decoder. The text reconstruction model also includes a text encoder built based on the large language model. The joint training process uses audio samples as the input of the audio encoder in the speech recognition model and recognizes text labels as the input of the text encoder.

[0080] Specifically, combined Figure 2 As shown, the speech recognition model may include an audio encoder and a decoder, wherein the audio sample is used as the input of the audio encoder, and the audio semantic representation of the audio sample is extracted by the audio encoder.

[0081] The text reconstruction model includes a text encoder and a decoder, and the speech recognition model and the text reconstruction model share the same decoder.

[0082] The text encoder is built on a large language model, specifically the large language model's ability to extract rich text semantic representations. During the training phase, the recognized text labels corresponding to the audio samples can be input into the text encoder to obtain the text semantic representations extracted by the text encoder. In this embodiment, to facilitate the calculation of the feature alignment loss between the text semantic representation and the audio semantic representation in subsequent steps, the text semantic representation extracted by the text encoder can be a feature of the same dimensionality as the audio semantic representation.

[0083] For a piece of audio-text training data, only one of the audio semantic representation extracted by the audio encoder and the text semantic representation extracted by the text encoder can be fed into the decoder. In this embodiment, a first probability P1 and a second probability (1-P1) can be pre-set. The audio semantic representation is fed into the decoder with the first probability P1, and the text semantic representation is fed into the decoder with the second probability (1-P1). The value of P1 can be set by the user and ranges from (0, 1).

[0084] The decoder decodes the input feature representation to obtain the output text.

[0085] It can be understood that when the input of the decoder is audio semantic representation, the text output by the decoder can be regarded as the speech recognition result; when the input of the decoder is text semantic representation, the text output by the decoder can be regarded as the reconstructed text.

[0086] Step S120: Calculate the text recognition loss value based on the text output by the decoder and the recognized text label, calculate the feature alignment loss value based on the audio semantic representation and the text semantic representation, and calculate the total loss value of the text recognition loss value and the feature alignment loss value.

[0087] Combine Figure 2 As shown in the figure, when the decoder input is an audio semantic representation, the text recognition loss value calculated between the decoder output text and the recognized text label can specifically be a speech recognition loss value. When the decoder input is a text semantic representation, the text recognition loss value calculated between the decoder output text and the recognized text label can specifically be a text reconstruction loss value. To simplify the expression in this embodiment, the text recognition loss function is defined as ASR_Loss.

[0088] The text recognition loss value can be used to update the parameters of the encoder and decoder as a whole.

[0089] For the audio semantic representation output by the audio encoder and the text semantic representation output by the text encoder, calculate the feature alignment loss value between the two, and combine Figure 2 As shown, the feature alignment loss function is defined as MSE_Loss.

[0090] Under the constraint of the feature alignment loss function, the rich semantic information of the large language model (the text encoder is built based on the large language model and has the feature processing capabilities of the large language model) can be migrated to the audio encoder, allowing the audio encoder to extract richer audio semantic representations. Under the premise of limited audio-text pair training data, the text data can be fully utilized to improve the feature extraction capability of the audio encoder and improve the recognition effect of the speech recognition model in specific fields.

[0091] Step S130: Update the model parameters according to the total loss value until the training end condition is reached, thereby obtaining a trained speech recognition model.

[0092] Specifically, when updating the parameters of the speech recognition model and text reconstruction model based on the aforementioned total loss value, a pre-trained large language model can be used for the large language model in the text encoder, and it is not involved in the parameter update process. Of course, in some other optional implementations, the large language model can also be combined with other network modules to participate in the parameter update.

[0093] After the set training end conditions are reached, the trained speech recognition model can be obtained.

[0094] Obviously, the speech recognition model training method provided in this embodiment adopts a text reconstruction model to assist in the joint training of the speech recognition model, and unifies the semantic representations of audio and text into the same semantic representation space through the feature alignment loss value. At the same time, since the large language model can extract rich semantic information, the rich semantic information of the large language model can be migrated to the audio encoder through the feature alignment loss value, so that the audio encoder can extract richer audio semantic representations, thereby improving the domain customization capability of the speech recognition model, that is, improving the recognition effect of the trained speech recognition model in a specific domain.

[0095] The speech recognition model training method provided in this embodiment does not require the language model to be loaded inside or outside the trained speech recognition model, which simplifies the structure of the speech recognition model and reduces the time consumption of inference. In addition, the present application does not need to use audio synthesis technology to synthesize audio text training data to train the speech recognition model. By making full use of the text data in the existing audio text training data, the audio encoder is guided to extract richer audio semantic representations, thereby improving the domain customization capability of the speech recognition model and improving the recognition effect of the speech recognition model in specific fields.

[0096] Reference Figure 3 , provides another speech recognition model training architecture. Compared with the previous embodiment, the speech recognition model in this embodiment can further include a shared encoder. During the training phase, the shared encoder is shared by the speech recognition model and the text reconstruction model.

[0097] The shared encoder is located between the audio encoder and decoder, and also between the text encoder and decoder. The audio semantic representation extracted by the audio encoder is fed into the shared encoder with a first probability P1, and the text semantic representation extracted by the text encoder is fed into the shared encoder with a second probability (1-P1).

[0098] The shared encoder re-encodes the input feature representation and sends the encoded features to the decoder.

[0099] In this embodiment, a shared encoder is added between the audio encoder and decoder of the speech recognition model. During the joint training phase, when the text semantic representation is fed into the shared encoder, the loss function is the text reconstruction loss function. When parameters are updated according to this text reconstruction loss function, the parameters of the decoder and shared encoder can be updated simultaneously. In other words, the network module involved in the parameter update further includes a shared encoder. In an end-to-end speech recognition model, the encoder typically plays a greater role than the decoder. Therefore, by adding a shared encoder to the parameter update, the shared encoder can also be fine-tuned for the domain, further improving the recognition performance of the trained speech recognition model for specific domains.

[0100] Reference Figure 4 , provides another speech recognition model training architecture. Compared with the above-mentioned embodiment, the speech recognition model in this embodiment can further include an alignment module.

[0101] When the input of the shared encoder is an audio semantic representation, the alignment module is used to predict the audio-text alignment information based on the audio semantic representation, that is, the alignment relationship between each frame in the audio sample and each word segmentation unit in the text.

[0102] On this basis, the temporal classification CTC loss can be calculated based on the audio-text alignment information and the recognition text label, and the total loss value of the joint training process can further include the CTC loss value.

[0103] In this embodiment, by adding the CTC loss function, the speech recognition model can be guided to learn the alignment relationship between audio and text, thereby improving the recognition effect of the speech recognition model.

[0104] It should be noted that according to Figure 4 After the architecture training is completed, only the audio encoder, shared encoder and decoder can be taken to form the final speech recognition model for subsequent reasoning.

[0105] Reference Figure 4 As shown, for the text encoder introduced in the above embodiment, this embodiment provides an optional composition structure of the text encoder, which may include: a large language model, a resampling module and a time length prediction module.

[0106] The large language model is used to extract the initial text semantic representation from the input text.

[0107] When the input of the shared encoder is an audio semantic representation, the resampling module is used to resample the initial text semantic representation based on the audio-text alignment information output by the alignment module to obtain a resampled text semantic representation with the same number of frames as the audio semantic representation.

[0108] Specifically, due to the difference in the granularity of modeling units, text sequences are usually converted into unit sequences such as character sequences, pinyin sequences, and phoneme sequences and sent to the large language model. The semantic representation of the text sequence extracted by the large language model usually has a smaller frame number than the audio semantic representation extracted by the acoustic encoder. In this embodiment, a resampling module is used to expand the frames of the text semantic representation so that its frame number is consistent with the audio semantic representation, and then the resampled text semantic representation and audio semantic representation are used to calculate the feature alignment loss MSE_Loss.

[0109] The duration prediction module is used to predict the duration of each frame in the initial text semantic representation, generating duration prediction information. Based on this information and the audio-text alignment information output by the alignment module, a duration prediction loss function is calculated, defined as Duration_Loss.

[0110] The total loss value of the joint training process may also include the above-mentioned duration prediction loss value.

[0111] In summary, the speech recognition model training process provided in this embodiment is jointly trained with the text reconstruction model. The total loss function of the joint training process can be expressed as:

[0112] Loss total =ASR_Loss+αMSE_Loss+βCTC_Loss+γDuration_Loss;

[0113] Among them, α, β, and γ are the weights of MSE_Loss, CTC_Loss, and Duration_Loss respectively.

[0114] The speech recognition model training method provided in this embodiment, by designing the structure of the text encoder, ensures that it has the rich semantic feature extraction capabilities of the large language model. At the same time, the alignment information output by the alignment module guides the resampling module to perform feature resampling, so that the dimensions of the resampled text semantic representation and the audio semantic representation are the same, thereby facilitating the calculation of MSE_Loss. In addition, the text encoder also includes a duration prediction module, which can predict the duration of the initial text semantic representation extracted by the large language model. The duration prediction information is combined with the audio-text alignment information output by the alignment module to calculate the duration prediction loss value, guiding the training process of the duration prediction module.

[0115] When there is a lack of training data for audio text, only the text data of the target domain can be obtained. After extracting the initial text semantic representation through the large language model, the duration prediction module predicts the duration information of each frame feature in the initial text semantic representation, thereby guiding the resampling module to perform resampling. Since the aforementioned training phase maps the text semantic representation and audio semantic representation to the same semantic representation space, the resampled text semantic representation can simulate the audio semantic representation extracted by the corresponding audio encoder and be sent to the subsequent shared encoder and decoder for processing. The decoded text is output and the text recognition loss value between the decoded text and the input text is calculated. The text recognition loss value guides the model parameter update process, which can greatly improve the domain adaptation effect of the shared encoder in the absence of corresponding audio.

[0116] After the training is completed according to the training strategy of the above embodiment, the present embodiment may further include a process of performing domain adaptive training on the speech recognition model using target domain text.

[0117] Combine Figure 5 As shown, at this stage there is only domain text but no corresponding audio, so it can be Figure 4 In the corresponding training architecture, the network modules related to the audio branch are removed, and the training architecture of this stage is as follows Figure 5 As shown, it includes a large language model, a duration prediction module, a resampling module, a shared encoder and a decoder.

[0118] Through the training strategy introduced in the above embodiment, the duration prediction module has the ability to predict duration information.

[0119] The domain adaptation training phase can include the following training steps:

[0120] S1. Obtain text corpus in the target domain.

[0121] Specifically, the target domain may be the domain in which the speech recognition model is to be applied. For example, if the speech recognition model is to be applied in the medical field, the medical field may be used as the target domain in this step.

[0122] For scenarios where audio-text pairing data in the target domain is scarce, in this embodiment, only text corpus in the target domain may be obtained. Compared with audio data, text corpus is easier to obtain.

[0123] In this step, the text corpus of the target domain can be obtained from a public dataset or automatically generated. This embodiment provides a method for automatically generating text corpus of the target domain, which can make use of the text generation capability of the large language model. Specifically:

[0124] The first prompt instruction prompt is sent to the large language model to obtain the text corpus of the target domain generated by the large language model, wherein the first prompt instruction is used to instruct the model to generate the text corpus of the target domain.

[0125] The first prompt instruction prompt may include the target domain theme, reference keywords, reference examples, word count requirements for the generated text, text volume requirements, etc. An example of the first prompt instruction prompt is provided below:

[0126] "The target field is medical conferences. Related topics include Topic 1, Topic 2, Topic 3, etc. Possible keywords include Keyword 1, Keyword 2, Keyword 3, etc. Typical sample texts in this field are Sample Text 1, Sample Text 2, and Sample Text 3. Based on the above information, please generate 1,000 lines of text related to this field, with each line containing 10 to 50 words."

[0127] S2. Send the text corpus into the large language model to obtain the target text semantic representation of the text corpus.

[0128] Combine Figure 5 As shown, the text corpus generated in the previous step is used as an input text sequence and input into the large language model, and the large language model is used to extract the target text semantic representation of the input text corpus.

[0129] S3. Predict the duration of each frame in the semantic representation of the target text through the duration prediction module to obtain target duration prediction information.

[0130] After the training process of the aforementioned embodiment, the duration prediction module has acquired the duration prediction capability. In this step, the duration prediction module can predict the duration of each frame in the semantic representation of the target text to obtain target duration prediction information.

[0131] S4. Resample the target text semantic representation according to the target duration prediction information through the resampling module to obtain the resampled text semantic representation.

[0132] Specifically, the resampling module, guided by the target duration prediction information, resamples the target text semantic representation to produce a resampled text semantic representation. Because the training phase mapped the text semantic representation and the audio semantic representation into the same semantic representation space, the resampled text semantic representation can now mimic the audio semantic representation extracted by the corresponding audio encoder.

[0133] S5. Encode the resampled text semantic representation through the shared encoder, send the encoded features to the decoder, and predict the decoded text through the decoder.

[0134] S6. Calculate the text reconstruction loss value based on the decoded text and text corpus predicted by the decoder, and update the model parameters according to the text reconstruction loss value until the training end condition is reached. The final speech recognition model is composed of the audio encoder, shared encoder and decoder.

[0135] As in the previous article, the text reconstruction loss function is defined as ASR_Loss. The text corpus obtained in the aforementioned step S1 can be used as the text label to calculate the text reconstruction loss value between the decoded text and the text label predicted by the decoder. The text reconstruction loss function can adopt a variety of loss function types, such as the cross entropy CE loss function.

[0136] After the above-mentioned domain adaptation training stage, the shared encoder and decoder can be domain-adaptively fine-tuned using the text corpus of the target domain, further improving the recognition effect of the speech recognition model in the target domain.

[0137] During the domain adaptation training phase, the parameters of the large language model can be updated synchronously. The loss function of the domain adaptation training phase can be expressed as:

[0138] Loss total =ASR_Loss;

[0139] Furthermore, before the above steps S3 and S4, the following steps may be further added:

[0140] The target duration prediction information is adjusted according to the configured duration disturbance parameters to obtain the disturbed target duration prediction information to guide the resampling module in step S4 to perform resampling.

[0141] That is, during the domain adaptation training phase, appropriate duration perturbation parameters can be added to the duration prediction module (such as multiplying the predicted duration of each frame by a parameter greater than 1 or less than 1 with a set probability P2), so that the generated resampled text semantic representation is richer, thereby better training the shared encoder and decoder.

[0142] To sum up, the speech recognition model training method provided in this application, on the one hand, does not require an external language model for decoding fusion, does not increase the amount of inference calculations, and does not increase the complexity of training and deployment; on the other hand, with the help of the rich semantic representation capabilities of the large language model, the audio semantic representation and text semantic representation are mapped to the same semantic representation space, and the rich semantic feature extraction capabilities of the large language model are migrated to the audio encoder, thereby improving the semantic perception ability of the audio encoder.

[0143] In addition, this application further adds a domain adaptation training stage, using text corpus from the target domain to adaptively train the model. This process can simultaneously fine-tune the shared encoder and decoder for the domain, improving the customization capability of the speech recognition model for the target domain. It can simulate the diversity of audio semantic representations in the absence of audio text training data, and achieve better domain adaptation effects.

[0144] On the basis of the speech recognition model training method introduced in any of the above embodiments, the speech to be recognized can be recognized based on the trained speech recognition model.

[0145] In some embodiments of the present application, a speech recognition method is further provided, which may include the following steps:

[0146] S1. Acquire acoustic features of the speech signal to be recognized.

[0147] S2. Input the acoustic features into the trained speech recognition model to obtain a speech recognition result output by the model; wherein:

[0148] The speech recognition model is jointly trained with the text reconstruction model during the training phase. The speech recognition model and the text reconstruction model share the same decoder. The text reconstruction model also includes a text encoder constructed based on a large language model. The joint training process uses audio samples as the input of the audio encoder in the speech recognition model, and the recognized text labels corresponding to the audio samples as the input of the text encoder. The total loss value of the joint training includes a text recognition loss value calculated based on the text output by the decoder and the recognized text labels, as well as a feature alignment loss value calculated based on the audio semantic representation extracted by the audio encoder and the text semantic representation extracted by the text encoder.

[0149] For the training process of the speech recognition model, please refer to the introduction of the speech recognition model training process in the aforementioned embodiments for details, which will not be repeated in this embodiment.

[0150] The speech recognition model trained with the above training strategy has the ability to be customized to the target domain. Therefore, for the speech signals to be recognized in the target domain, the speech recognition model can achieve better recognition results.

[0151] The following describes a speech recognition device provided in an embodiment of the present application. The speech recognition device described below and the speech recognition method described above can refer to each other.

[0152] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of a speech recognition device disclosed in an embodiment of the present application.

[0153] like Figure 6 As shown, the device may include:

[0154] The acoustic feature acquisition unit 11 is used to acquire the acoustic features of the speech signal to be recognized;

[0155] The model recognition unit 12 is used to input the acoustic features into the trained speech recognition model to obtain the speech recognition results output by the model; wherein:

[0156] The speech recognition model is jointly trained with the text reconstruction model during the training phase. The speech recognition model and the text reconstruction model share the same decoder. The text reconstruction model also includes a text encoder constructed based on a large language model. The joint training process uses audio samples as the input of the audio encoder in the speech recognition model, and the recognized text labels corresponding to the audio samples as the input of the text encoder. The total loss value of the joint training includes a text recognition loss value calculated based on the text output by the decoder and the recognized text labels, and a feature alignment loss value calculated based on the audio semantic representation extracted by the audio encoder and the text semantic representation extracted by the text encoder.

[0157] Furthermore, the speech recognition model training device provided in the embodiment of the present application is described. The speech recognition model training device described below and the speech recognition model training method described above can be referred to in correspondence with each other.

[0158] See also Figure 7 , Figure 7 This is a structural diagram of a speech recognition model training device disclosed in an embodiment of the present application.

[0159] like Figure 7 As shown, the device may include:

[0160] A training data acquisition unit 21 is used to acquire audio-text pair training data, wherein the audio-text pair training data includes audio samples and corresponding recognition text labels;

[0161] A model calculation unit 22 is configured to use the audio sample as input to an audio encoder in a speech recognition model, and to use the recognized text label as input to a text encoder constructed based on a large language model in a text reconstruction model, wherein the text reconstruction model and the speech recognition model share the same decoder, wherein the audio encoder is configured to extract an audio semantic representation and feed the representation into the decoder with a first probability, wherein the text encoder is configured to extract a text semantic representation and feed the representation into the decoder with a second probability, wherein the decoder is configured to output decoded text, wherein the sum of the first probability and the second probability is equal to 1;

[0162] a loss value calculation unit 23, configured to calculate a text recognition loss value based on the text output by the decoder and the recognized text label, calculate a feature alignment loss value based on the audio semantic representation and the text semantic representation, and calculate a total loss value of the text recognition loss value and the feature alignment loss value;

[0163] The parameter updating unit 24 is used to update the model parameters according to the total loss value until the training end condition is met, thereby obtaining a trained speech recognition model.

[0164] In one possible implementation, the speech recognition model also includes a shared encoder located between the audio encoder and the decoder, which encodes the input features and sends the encoded features to the decoder; the audio semantic representation extracted by the audio encoder is sent to the shared encoder with a first probability, and the text semantic representation extracted by the text encoder is sent to the shared encoder with a second probability.

[0165] In one possible implementation, the speech recognition model further includes an alignment module;

[0166] When the input of the shared encoder is the audio semantic representation, the alignment module is used to predict audio-text alignment information based on the audio semantic representation;

[0167] The total loss value also includes: a temporal classification CTC loss calculated based on the audio-text alignment information and the recognized text label.

[0168] In one possible implementation, the text encoder includes a large language model, a resampling module, and a time-length prediction module;

[0169] The large language model is used to extract an initial text semantic representation from the input text;

[0170] When the input of the shared encoder is the audio semantic representation, the resampling module is used to resample the initial text semantic representation based on the audio-text alignment information output by the alignment module to obtain a resampled text semantic representation with the same number of frames as the audio semantic representation, and calculate the feature alignment loss value using the resampled text semantic representation and the audio semantic representation;

[0171] The duration prediction module is used to predict the duration of each frame in the initial text semantic representation to obtain duration prediction information;

[0172] The total loss value also includes: a duration prediction loss value calculated based on the duration prediction information and the audio-text alignment information.

[0173] In one possible implementation, the apparatus of the present application may further include: a domain adaptive fine-tuning training unit, configured to further employ a domain adaptive fine-tuning training strategy to perform model training after the parameter updating unit updates the model parameters according to the total loss value until the training end condition is met. This process may include:

[0174] Obtain text corpus in the target field;

[0175] Sending the text corpus into the large language model to obtain a target text semantic representation of the text corpus;

[0176] Predicting the duration of each frame in the semantic representation of the target text by the duration prediction module to obtain target duration prediction information;

[0177] Resampling the target text semantic representation according to the target duration prediction information through the resampling module to obtain a resampled text semantic representation;

[0178] Encoding the resampled text semantic representation through the shared encoder, sending the encoded features to the decoder, and predicting the decoded text through the decoder;

[0179] A text reconstruction loss value is calculated based on the decoded text predicted by the decoder and the text corpus, and the model parameters are updated according to the text reconstruction loss value until the training end condition is reached. The final speech recognition model is composed of the audio encoder, the shared encoder and the decoder.

[0180] In one possible implementation, after obtaining the target duration prediction information, the domain adaptive fine-tuning training unit can also be used to adjust the target duration prediction information according to the configured duration perturbation parameters to obtain the perturbed target duration prediction information to guide the resampling module to perform resampling.

[0181] In one possible implementation, the process of the domain adaptive fine-tuning training unit acquiring text corpus in the target domain includes:

[0182] The first prompt instruction prompt is sent to the large language model to obtain the text corpus of the target domain generated by the large language model, wherein the first prompt instruction is used to instruct the model to generate the text corpus of the target domain.

[0183] An electronic device is also provided in an embodiment of the present application. Figure 8 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as servers, personal computers, mobile phones, etc. Figure 8 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0184] like Figure 8 As shown, the electronic device may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 602 or the program loaded from the storage device 608 to the random access memory (RAM) 603 to implement the speech recognition method or speech recognition model training method of the aforementioned embodiment of the present application. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing device 601, ROM 602 and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0185] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Figure 8 The electronic device is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed instead.

[0186] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any one of the speech recognition methods or speech recognition model training methods provided in the embodiments of the present application.

[0187] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When one or more computer programs are executed by an electronic device, the electronic device can implement any speech recognition method or speech recognition model training method provided in an embodiment of the present application.

[0188] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0189] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0190] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0191] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0192] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referenced to each other.

Claims

1. A speech recognition method, characterized in that: include: Acquiring acoustic features of a speech signal to be recognized; The acoustic features are input into the trained speech recognition model to obtain the speech recognition results output by the model; wherein: The speech recognition model is jointly trained with the text reconstruction model during the training phase. The speech recognition model and the text reconstruction model share the same decoder. The text reconstruction model also includes a text encoder constructed based on a large language model. The joint training process uses audio samples as the input of the audio encoder in the speech recognition model, and the recognized text labels corresponding to the audio samples as the input of the text encoder. The total loss value of the joint training includes a text recognition loss value calculated based on the text output by the decoder and the recognized text labels, and a feature alignment loss value calculated based on the audio semantic representation extracted by the audio encoder and the text semantic representation extracted by the text encoder. The feature alignment loss value can unify the semantic representations of audio and text into the same semantic representation space, and migrate the rich semantic information of the large language model to the audio encoder.

2. A speech recognition model training method, characterized in that: include: Acquire audio-text pair training data, wherein the audio-text pair training data includes audio samples and corresponding recognition text labels; The audio sample is used as input to an audio encoder in a speech recognition model, and the recognized text label is used as input to a text encoder constructed based on a large language model in a text reconstruction model, wherein the text reconstruction model and the speech recognition model share the same decoder, the audio encoder is used to extract an audio semantic representation and feed it into the decoder with a first probability, the text encoder is used to extract a text semantic representation and feed it into the decoder with a second probability, and the decoder is used to output a decoded text, where the sum of the first probability and the second probability is equal to 1; Calculating a text recognition loss value based on the text output by the decoder and the recognized text label, calculating a feature alignment loss value based on the audio semantic representation and the text semantic representation, and calculating a total loss value of the text recognition loss value and the feature alignment loss value; The feature alignment loss value can unify the semantic representation of audio and text into the same semantic representation space, and transfer the rich semantic information of the large language model to the audio encoder; The model parameters are updated according to the total loss value until the training end condition is reached, thereby obtaining a trained speech recognition model.

3. The method according to claim 2, characterized in that The speech recognition model also includes a shared encoder located between the audio encoder and the decoder, which encodes the input features and sends the encoded features to the decoder; the audio semantic representation extracted by the audio encoder is sent to the shared encoder with a first probability, and the text semantic representation extracted by the text encoder is sent to the shared encoder with a second probability.

4. The method according to claim 3, characterized in that The speech recognition model also includes an alignment module; When the input of the shared encoder is the audio semantic representation, the alignment module is used to predict audio-text alignment information based on the audio semantic representation; The total loss value also includes: a temporal classification CTC loss calculated based on the audio-text alignment information and the recognized text label.

5. The method according to claim 4, characterized in that The text encoder includes a large language model, a resampling module and a time length prediction module; The large language model is used to extract an initial text semantic representation from the input text; When the input of the shared encoder is the audio semantic representation, the resampling module is used to resample the initial text semantic representation based on the audio-text alignment information output by the alignment module to obtain a resampled text semantic representation with the same number of frames as the audio semantic representation, and calculate the feature alignment loss value using the resampled text semantic representation and the audio semantic representation; The duration prediction module is used to predict the duration of each frame in the initial text semantic representation to obtain duration prediction information; The total loss value also includes: a duration prediction loss value calculated based on the duration prediction information and the audio-text alignment information.

6. The method according to claim 5, characterized in that After updating the model parameters according to the total loss value until the training end condition is reached, the method further includes: Obtain text corpus in the target field; Sending the text corpus into the large language model to obtain a target text semantic representation of the text corpus; Predicting the duration of each frame in the semantic representation of the target text by the duration prediction module to obtain target duration prediction information; Resampling the target text semantic representation according to the target duration prediction information through the resampling module to obtain a resampled text semantic representation; Encoding the resampled text semantic representation through the shared encoder, sending the encoded features to the decoder, and predicting the decoded text through the decoder; A text reconstruction loss value is calculated based on the decoded text predicted by the decoder and the text corpus, and the model parameters are updated according to the text reconstruction loss value until the training end condition is reached. The final speech recognition model is composed of the audio encoder, the shared encoder and the decoder.

7. The method according to claim 6, characterized in that After obtaining the target duration prediction information, it also includes: The target duration prediction information is adjusted according to the configured duration disturbance parameter to obtain disturbed target duration prediction information to guide the resampling module to perform resampling.

8. The method according to claim 6 or 7, characterized in that The process of obtaining text corpus in the target domain includes: The first prompt instruction prompt is sent to the large language model to obtain the text corpus of the target domain generated by the large language model, wherein the first prompt instruction is used to instruct the model to generate the text corpus of the target domain.

9. A speech recognition device, characterized in that: include: An acoustic feature acquisition unit, configured to acquire acoustic features of a speech signal to be recognized; A model recognition unit is used to input the acoustic features into the trained speech recognition model to obtain a speech recognition result output by the model; wherein: The speech recognition model is jointly trained with the text reconstruction model during the training phase. The speech recognition model and the text reconstruction model share the same decoder. The text reconstruction model also includes a text encoder constructed based on a large language model. The joint training process uses audio samples as the input of the audio encoder in the speech recognition model, and the recognized text labels corresponding to the audio samples as the input of the text encoder. The total loss value of the joint training includes a text recognition loss value calculated based on the text output by the decoder and the recognized text labels, and a feature alignment loss value calculated based on the audio semantic representation extracted by the audio encoder and the text semantic representation extracted by the text encoder. The feature alignment loss value can unify the semantic representations of audio and text into the same semantic representation space, and migrate the rich semantic information of the large language model to the audio encoder.

10. A speech recognition model training device, characterized in that: include: A training data acquisition unit, configured to acquire audio-text pair training data, wherein the audio-text pair training data includes audio samples and corresponding recognition text labels; a model calculation unit, configured to use the audio sample as input to an audio encoder in a speech recognition model, and to use the recognized text label as input to a text encoder constructed based on a large language model in a text reconstruction model, wherein the text reconstruction model and the speech recognition model share the same decoder, the audio encoder is configured to extract an audio semantic representation and feed the representation into the decoder with a first probability, the text encoder is configured to extract a text semantic representation and feed the representation into the decoder with a second probability, the decoder is configured to output decoded text, and the sum of the first probability and the second probability is equal to 1; a loss value calculation unit, configured to calculate a text recognition loss value based on the text output by the decoder and the recognized text label, calculate a feature alignment loss value based on the audio semantic representation and the text semantic representation, and calculate a total loss value of the text recognition loss value and the feature alignment loss value; The feature alignment loss value can unify the semantic representation of audio and text into the same semantic representation space, and transfer the rich semantic information of the large language model to the audio encoder; A parameter updating unit is used to update the model parameters according to the total loss value until the training end condition is reached to obtain a trained speech recognition model.

11. An electronic device, characterized in that: include: memory and processor; The memory is used to store programs; The processor is used to execute the program to implement the speech recognition method as claimed in claim 1, or to implement each step of the speech recognition model training method as claimed in any one of claims 2 to 8.

12. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the speech recognition method according to claim 1, or implements the various steps of the speech recognition model training method according to any one of claims 2 to 8.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the speech recognition method as claimed in claim 1, or implements the various steps of the speech recognition model training method as claimed in any one of claims 2 to 8.

Images