Speech recognition method and apparatus, electronic device, and storage medium

Abstract

The application provides a speech recognition method and device, electronic equipment and storage medium, the method comprises: determining the to be recognized speech, and the context information corresponding to the to be recognized speech;The to be recognized speech and the context information are input to the speech recognition model, and the recognition text output by the speech recognition model is obtained;The speech recognition model is constructed on the basis of a large language speech recognition model;The speech recognition model is trained based on the semantic relationship between the first text and the second synthesized speech;The second synthesized speech is generated based on the second text corresponding to the first text, and the first text and the second text are semantically related or semantically unrelated, which can make the model fully utilize the context information in the speech recognition process, overcome the defects that the existing large language speech recognition system lacks context utilization capability special training and is difficult to fully utilize the context information, and improve the accuracy and robustness of speech recognition in complex scenes such as homophones, near homophones, proper nouns and entity disambiguation.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a speech recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of artificial intelligence technology, automatic speech recognition has been widely applied in various scenarios. Traditional end-to-end speech recognition systems rely on acoustic models and general language models to complete the transcription. The recognition results are highly dependent on the audio signal itself and are difficult to adapt to the needs of complex scenarios.

[0003] The training of existing automatic speech recognition models often relies on audio-text pairs, focusing only on the relationship between input speech and output text. This leads to insufficient understanding of scenarios and needs by the model, resulting in insufficient entity recall accuracy in complex scenarios such as long contexts, homophones, near-homophones, proper nouns, and entity disambiguation, and significant shortcomings in recognition performance. Summary of the Invention

[0004] This invention provides a speech recognition method, apparatus, electronic device, and storage medium to solve the problem in the prior art that it is difficult to fully utilize contextual information to assist speech recognition, resulting in poor recognition accuracy in complex reasoning scenarios.

[0005] This invention provides a speech recognition method, comprising: Determine the speech to be recognized and the context information corresponding to the speech to be recognized; The speech to be recognized and the context information are input into the speech recognition model to obtain the recognized text output by the speech recognition model; The speech recognition model is built on the basis of the large language speech recognition model; The speech recognition model is trained based on the semantic relationship between the first text and the second synthesized speech; the second synthesized speech is generated based on the second text corresponding to the first text, and the first text and the second text are semantically related or semantically unrelated.

[0006] According to a speech recognition method provided by the present invention, the speech recognition model is trained based on the following steps: Based on the second text, speech synthesis is performed to obtain the second synthesized speech; The first text and the second synthesized speech are input into the large language speech recognition model to obtain the predicted semantic relationship output by the large language speech recognition model; Based on the predicted semantic relations and semantic relation labels, the large language speech recognition model is trained to obtain the speech recognition model; The semantic relationship label represents the semantic relationship between the first text and the second text; The semantic relationship tags include semantic category tags and / or semantic consistency tags; The semantic category labels include labels representing implication relationships, labels representing contradictory relationships, and labels representing neutral relationships; the semantic consistency labels include labels representing semantic consistency and labels representing semantic inconsistency.

[0007] According to a speech recognition method provided by the present invention, the second text is determined based on the following steps: Determine the first text; based on the first text, generate a second text that is semantically related to the first text, and / or, perform at least one of causal reversal, conceptual substitution, or inference subversion on the first text to generate a second text that is semantically unrelated to the first text; And / or, Obtain text pairs with semantic relationship annotations, identify one text in the text pair as the first text, identify the other text in the text pair as the second text, and determine the semantic relationship label based on the semantic relationship annotations.

[0008] According to a speech recognition method provided by the present invention, the step of training a large language speech recognition model based on the predicted semantic relations and semantic relation labels to obtain the speech recognition model includes: Obtain sample text, and generate sample context based on the target entities contained in the sample text; Speech synthesis is performed based on the sample text to obtain sample speech, and sample triples are constructed based on the sample context, the sample text, and the sample speech. Based on the predicted semantic relations and semantic relation labels, the large language speech recognition model is trained to obtain an initial speech recognition model; Based on the sample triples, the first text, and the second synthesized speech, the initial speech recognition model is jointly trained to obtain the speech recognition model. The tasks in the joint training process include a speech recognition task constructed based on the sample triples, and a semantic relationship prediction task constructed based on the first text and the second synthesized speech.

[0009] According to a speech recognition method provided by the present invention, the sample context includes entity-related context and logically related context; the step of generating the sample context based on the target entities contained in the sample text includes: Entity recognition is performed on the sample text to obtain an entity set; The target entity is obtained by filtering the entity set by key entities. The target entity is subjected to correlation divergence to obtain anchor point information that can be used to deduce the target entity; Generate an entity-related context that includes the target entity, and generate a logically related context that does not include the target entity based on the anchor information.

[0010] According to a speech recognition method provided by the present invention, the step of synthesizing speech based on the sample text to obtain sample speech includes: Based on the text length, information content, and semantic relevance between the sample context and the corresponding sample text, the quality of the sample context is verified to obtain the verification result. Based on the verification result, a qualified sample context is determined from the sample context; Based on the sample text corresponding to the qualified sample context and the reference speech, speech synthesis is performed to obtain sample speech with different timbres and different speech rates.

[0011] According to a speech recognition method provided by the present invention, the large language speech recognition model includes a speech encoder, a modality adaptation module, and a large language model; The step of inputting the first text and the second synthesized speech into the large language speech recognition model to obtain the predicted semantic relationship output by the large language speech recognition model includes: The second synthesized speech is input into the large language speech recognition model, and processed by the speech encoder and the modality adaptation module to obtain a speech representation aligned with the word embedding space of the large language model. The first text is input into the large language speech recognition model, and processed by the word embedding layer of the large language model to obtain a text representation; The speech representation and the text representation are input into the large language model to obtain the predicted semantic relationship output by the large language model.

[0012] The present invention also provides a voice recognition device, comprising: An information determination unit is used to determine the speech to be recognized and the context information corresponding to the speech to be recognized; A speech recognition unit is used to input the speech to be recognized and the context information into a speech recognition model to obtain the recognized text output by the speech recognition model; The speech recognition model is built on the basis of the large language speech recognition model; The speech recognition model is trained based on the semantic relationship between the first text and the second synthesized speech; the second synthesized speech is generated based on the second text corresponding to the first text, and the first text and the second text are semantically related or semantically unrelated.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the speech recognition method as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech recognition method as described above.

[0015] The speech recognition method, apparatus, electronic device, and storage medium provided by this invention determine the speech to be recognized and its corresponding context information, and input it into a speech recognition model built on a large language speech recognition model and trained based on the semantic relationship between a first text and a second synthesized speech to obtain the recognized text. This allows the model to make full use of context information during the speech recognition process, overcoming the shortcomings of existing large language speech recognition systems that lack dedicated training for context utilization and are difficult to fully utilize context information. This improves the accuracy and robustness of speech recognition in complex scenarios such as homophones, near-homophones, and entity disambiguation. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the speech recognition method provided by the present invention; Figure 2 This is a flowchart of the sample triplet construction process provided by the present invention; Figure 3 This is a flowchart of the sample context determination process provided by the present invention; Figure 4 This is a structural diagram of the large language speech recognition model provided by the present invention; Figure 5 This is a schematic diagram of the structure of the speech recognition device provided by the present invention; Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0019] With the development of artificial intelligence, automatic speech recognition technology has been widely applied in various scenarios. Traditional end-to-end speech recognition systems mainly rely on acoustic models and language models to complete the recognition task. The recognition result is determined by the speech signal, and the ability to utilize external contextual information is limited.

[0020] Compared to traditional speech recognition systems, LLM-ASR (Large Language Model-Automatic Speech Recognition) leverages the language understanding capabilities of large language models to improve recognition performance. In real-world applications, speech content is often strongly correlated with contextual information. For example, in scenarios such as meeting minutes, dialogue interactions, customer service Q&A, and professional domain voice input, users typically provide contextual information such as historical dialogues, background descriptions, domain knowledge, and task instructions. Correctly utilizing this contextual information helps the model understand the current semantics, disambiguate, and improve recognition accuracy. Especially in scenarios involving homophones, near-homophones, proper nouns, and entity disambiguation, relying solely on audio information often fails to yield correct recognition results; the correct meaning needs to be determined in conjunction with the contextual semantics.

[0021] Current automatic speech recognition technologies primarily employ end-to-end neural networks or architectures combined with large language models to complete speech-to-text tasks. For context-assisted recognition, existing methods mainly assist recognition by incorporating contextual information such as historical text, domain knowledge, hot words, and task prompts, through prompt learning or contextual concatenation.

[0022] However, existing solutions mainly rely on audio-text pairs for training, focusing on speech-to-text mapping learning. Although they can assist in recognition through historical text and domain knowledge, they mainly focus on the input method of contextual information and lack a special training mechanism for the model's ability to utilize context. As a result, although the model has a certain language ability, it is difficult to make full use of contextual information to assist speech recognition. In complex scenarios such as long context, entity disambiguation, and homonym recognition, there are still problems with the inability to correctly recall entity words, and the recognition performance is still insufficient.

[0023] To address this, the present invention provides a speech recognition method that aims to specifically train a large language speech recognition model by leveraging the semantic relationship between the first text and the second synthesized speech. This fills the gap in traditional solutions by lacking training on the model's ability to utilize contextual information, thereby enhancing the model's semantic understanding of contextual information. As a result, it can more fully utilize external contextual information to assist speech recognition during the inference stage, effectively improving the recognition accuracy and robustness in complex scenarios such as homophones, near-homophones, proper nouns, and entity disambiguation.

[0024] It should be noted that all actions involving the acquisition of information and data in this invention are carried out in accordance with the relevant data protection laws and policies of the country where the invention is located, and with the authorization of the owner of the relevant device.

[0025] Figure 1 This is a flowchart illustrating the speech recognition method provided by the present invention, as shown below. Figure 1 As shown, the method includes: Step 110: Determine the speech to be recognized and the context information corresponding to the speech to be recognized; Step 120: Input the speech to be recognized and contextual information into the speech recognition model to obtain the recognized text output by the speech recognition model; The speech recognition model is built on the basis of the large language speech recognition model; The speech recognition model is trained based on the semantic relationship between the first text and the second synthesized speech; the second synthesized speech is generated based on the second text corresponding to the first text, and the first text and the second text are either semantically related or semantically unrelated.

[0026] Specifically, when performing speech recognition, it is first necessary to determine the speech to be recognized, that is, the speech signal that needs to be recognized and transcribed into text. This can be a speech signal acquired in real time, such as audio acquired in real time through acquisition devices such as microphones in scenarios such as meeting recordings, dialogue interactions, and customer service Q&A; or it can be a pre-recorded and stored speech signal, such as a pre-recorded speech file or audio data acquired from other devices or external systems; the embodiments of the present invention do not specifically limit this.

[0027] Similarly, in this embodiment of the invention, there are no specific limitations on the language, duration, number of speakers, etc. of the speech to be recognized.

[0028] After identifying the speech to be recognized, it is also necessary to obtain its corresponding context information. For example, historical dialogues can be extracted from the current dialogue history as context information, or relevant information can be obtained from pre-configured domain knowledge bases, hot word lists, task prompts, background descriptions, etc., or the user can directly input or specify the context information.

[0029] Furthermore, the acquired contextual information should have a certain correlation with the speech content to be recognized, so as to play an auxiliary role in the subsequent recognition process.

[0030] Here, contextual information can be directly related to the speech content to be recognized, meaning the contextual information directly contains entity words from the speech content; or it can be indirectly related, meaning the contextual information does not directly contain entity words from the speech content, but contains relevant information that has a logical inference relationship with them. Correctly utilizing this contextual information helps the model understand the current semantics, eliminate ambiguity, and improve recognition accuracy.

[0031] Once the speech to be recognized and its corresponding context information are determined, both can be input into the speech recognition model. With the help of the context information, the speech recognition model performs speech recognition on the speech to be recognized and finally outputs the recognized text corresponding to the speech content of the speech to be recognized.

[0032] Here, the speech recognition model is built on the Large Language Speech Recognition Model LLM-ASR, and can be pre-trained based on the semantic relationship between the first text and the second synthesized speech.

[0033] In other words, before applying a speech recognition model for speech recognition, it can be trained. Here, LLM-ASR is a model that uses the language understanding capabilities of a large language model to complete speech-to-text transcription. In this embodiment of the invention, the structure of LLM-ASR is followed, and specialized training is performed on this basis to enable the model to understand and utilize contextual information, and finally a trained model, i.e., a speech recognition model, can be obtained.

[0034] In specific training, it is necessary to first determine the first text and the second text. The first text is the text used as context input for model training, and the second text is the text corresponding to the first text used to generate synthesized speech. Then, speech synthesis can be performed based on the second text to obtain the second synthesized speech. After that, the first text can be used as context and the second synthesized speech can be used as audio input, so that the model can learn the semantic relationship between the first text and the second synthesized speech, thereby enabling the model to understand the semantic relationship between the context and the audio content.

[0035] Additionally, it should be noted that the first text and the second text can be semantically related or semantically unrelated; semantically related means that they are related, consistent, or mutually inferable in terms of semantics, while semantically unrelated means that they are inconsistent, contradictory, or independent of each other in terms of semantics.

[0036] By creating different semantically related or semantically unrelated scenarios between the first and second texts, rich semantic relationship supervision signals can be provided to the model. This can guide the model to learn various relationship types between the context and the audio content, such as semantic consistency and semantic independence, and thus enable it to make fuller use of contextual information for speech recognition during the inference stage.

[0037] For example, in a specific application scenario, the speech content to be recognized may involve a proper noun, which is difficult to distinguish from another common word with the same or similar pronunciation based solely on audio information. In this case, if the context information contains domain knowledge or historical dialogue content related to the proper noun, the speech recognition model trained as described above can accurately determine that the speech to be recognized contains the proper noun by combining the context information, thereby outputting the correct recognized text, achieving entity disambiguation, and avoiding recognition errors.

[0038] It should be noted that through semantic relation-specific training, the model can learn to understand contextual information at the parameter level, rather than simply introducing context into the inference input side, thereby enabling the model to make fuller and more stable use of contextual information.

[0039] The speech recognition method provided by this invention determines the speech to be recognized and its corresponding context information, and inputs it into a speech recognition model built on the basis of a large language speech recognition model and trained based on the semantic relationship between the first text and the second synthesized speech to obtain the recognized text. This method enables the model to make full use of context information during the speech recognition process, overcoming the shortcomings of existing large language speech recognition systems that lack dedicated training for context utilization and are difficult to fully utilize context information. It improves the accuracy and robustness of speech recognition in complex scenarios such as homophones, near-homophones, proper nouns, and entity disambiguation.

[0040] Based on the above embodiments, the speech recognition model is trained using the following steps: Speech synthesis is performed based on the second text to obtain the second synthesized speech; The first text and the second synthesized speech are input into the large language speech recognition model to obtain the predicted semantic relationship output by the large language speech recognition model. Based on predicted semantic relations and semantic relation labels, a large language speech recognition model is trained to obtain a speech recognition model; Semantic relation tags represent the semantic relationship between the first text and the second text; Semantic relationship tags include semantic category tags and / or semantic consistency tags; Semantic category labels include labels representing implication relationships, labels representing contradictory relationships, and labels representing neutral relationships; semantic consistency labels include labels representing semantic consistency and labels representing semantic inconsistency.

[0041] Specifically, the training process of a speech recognition model includes the following steps: First, after obtaining the paired first and second texts, the second text can be input into the speech synthesis model to synthesize the corresponding speech signal. For example, a zero-shot TTS (Text-To-Speech) model can be used to synthesize speech. This model can generate speech with various timbres and speaking rates based on a small amount of reference audio without fine-tuning for specific speakers, thus providing rich acoustic input for model training.

[0042] That is, since audio inputs are often diverse in actual speech recognition scenarios, speech synthesis methods with different timbres and speech rates can be used when generating the second synthesized speech, such as the zero-shot TTS model, to ensure the diversity of acoustic features in the training data.

[0043] Subsequently, the first text and the second synthesized speech can be input into LLM-SAR for semantic understanding and relation judgment, thereby obtaining a predicted result representing the semantic association state between the two, i.e., predicting the semantic relationship. Specifically, this is organized in the form of instruction fine-tuning. Without changing the main task and main structure of LLM-SAR, the model is guided by task instructions, such as Prompt. The model uses the input second synthesized speech as audio features and the first text combined with the task instructions as text features for joint processing, analyzing and inferring the semantic relationship between the two, such as consistency or inconsistency, or implication, contradiction, or neutrality.

[0044] Then, the loss of the model in the above task can be calculated by comparing the predicted semantic relationships with the true semantic relationship labels, and the model can be trained accordingly. Here, the semantic relationship labels are pre-defined supervision signals that represent the true semantic relationship between the first text and the second text, i.e., true labels. Since the second synthesized speech is synthesized from the second text, the semantic relationship between the first text and the second text is equivalent to the true semantic relationship between the context of the input model and the audio content.

[0045] Specifically, in this embodiment of the invention, the difference between the predicted semantic relationship and the semantic relationship label can be calculated, such as by calculating the cross-entropy loss, and the parameters of LLM-SAR can be backpropagated and updated and optimized based on the difference, so that LLM-SAR can continuously improve the accuracy of prediction during iterative training.

[0046] Through this training, LLM-SAR learned how to analyze the semantic relationship between the context and the input speech, thus evolving into a speech recognition model with powerful contextual understanding capabilities.

[0047] It should be noted that, in this embodiment of the invention, semantic relationship labels include two main categories: semantic category labels and semantic consistency labels, which correspond to two different contextual semantic understanding training tasks: semantic relationship judgment task and semantic consistency judgment task.

[0048] The semantic category labels are specifically used to indicate the category classification in the semantic relationship judgment task. They include labels that represent an entailment relationship, that is, the audio content can be inferred from the context, or there is a clear entailment relationship between the two semantically; labels that represent a contradiction relationship, that is, the context content and the audio content are semantically opposed or conflicting; and labels that represent a neutral relationship, that is, there is neither a clear entailment relationship nor a direct contradiction between the context content and the audio content, and the two are semantically relatively independent.

[0049] Semantic consistency labels are used to indicate consistency in semantic consistency judgment tasks. They include labels that represent semantic consistency (i.e., the context content and the audio content are consistent in the core semantics they express) and labels that represent semantic inconsistency (i.e., the context content and the audio content deviate from or are completely inconsistent with the core semantics they express).

[0050] In practice, the above training process can be represented by two auxiliary tasks: a semantic relationship judgment task and a semantic consistency judgment task. When performing the semantic relationship judgment task, the input to the model is the second synthesized speech and the first text serving as context. The model is required to determine the type of semantic relationship between the two. The model ultimately outputs a classification result, i.e., a predicted semantic relationship, such as neutral, contradictory, or implied. Through this task, the model can learn in detail the logical hierarchical relationship between contextual cues and audio content.

[0051] When performing the semantic consistency judgment task, the input consists of a second synthesized speech and a first text. The model is required to determine whether the semantics of the context content and the audio content are consistent. The model ultimately outputs a binary classification prediction result, such as consistent or inconsistent. This task guides the model to learn the degree of matching between the core semantics of the context and the audio content.

[0052] In this embodiment of the invention, by constructing data pairs containing contextual text and synthesized speech, and designing auxiliary training tasks for semantic relationship judgment and / or semantic consistency judgment, the large language speech recognition model is explicitly guided to learn the complex semantic associations between text context and audio content. Compared with traditional solutions that only concatenate contextual text during inference, this embodiment of the invention enhances the model's deep understanding and reasoning ability of the context from the model parameter level, so that the trained speech recognition model can make fuller, more stable and accurate use of contextual information when facing complex contexts and logical inference scenarios.

[0053] Based on the above embodiments, the model is based on the speech representation of the second synthesized speech. and the text representation of the first text. Conditional probability distribution for predicting relation categories ,Pick As output: in, To predict semantic relationships, Three types of labels for semantic relation determination tasks. Two types of labels for the semantic consistency judgment task. These are the trainable parameters of the model.

[0054] Based on the above embodiments, the second text is determined based on the following steps: Determine a first text; based on the first text, generate a second text that is semantically related to the first text, and / or, perform at least one of the following on the first text: causal reversal, conceptual substitution, or inference subversion, to generate a second text that is semantically unrelated to the first text; And / or, Obtain text pairs with semantic relationship annotations, identify one text in the text pair as the first text, and the other text in the text pair as the second text, and determine the semantic relationship label based on the semantic relationship annotation.

[0055] Specifically, the embodiments of the present invention provide a variety of methods for determining the second text, including automatic generation by a large language model and direct acquisition from open source datasets. In practical applications, one of these methods can be used alone or in combination.

[0056] In the first approach, which is the automatic generation approach based on a large language model, a first text is first determined, which can be obtained from a collected text corpus. Then, the first text can be input into the large language model, which rewrites or continues the first text based on the input text, and automatically generates a second text that is semantically related to it, corresponding to a scenario with semantic consistency or implication.

[0057] Optionally, in order to generate negative sample data, in this embodiment of the invention, instead of simply negating the first text, at least one logical manipulation operation is performed, such as causal reversal, conceptual substitution, or inference subversion, to generate a second text that is semantically contradictory to the first text or has no direct logical connection with it.

[0058] Here, causal inversion involves swapping the cause and effect in the first text; concept substitution involves replacing an entity or concept in the first text with another easily confused entity or concept; and inference subversion involves deriving a completely opposite or invalid conclusion based on known facts in the first text. This avoids generating overly simplistic negative samples, such as simply adding or deleting negative words in the first text, which would increase the challenge and complexity of the training data.

[0059] In this embodiment of the invention, when constructing training data for the semantic consistency judgment task, logical rewriting methods such as causal inversion, concept substitution, and inference subversion can effectively prevent the model from using simple word matching shortcuts, such as identifying negative words to complete the judgment task, and force the model to truly understand the relationship between text and audio in terms of deep logic and core concepts.

[0060] In the second approach, which is based on directly obtaining data from open-source datasets, in order to further enrich the training data, existing text pairs with semantic relationship annotations can be obtained, such as the cmnli task data in the open-source CLUE dataset, which is itself to determine the semantic relationship between two sentences.

[0061] Then, one text in the text pair is directly designated as the first text, serving as the input context, while the other text in the text pair is designated as the second text, used for conversion into the second synthesized speech via TTS. Since the text pair itself already contains accurate annotation information, semantic relationship labels, such as implication, contradiction, and neutrality, can be directly determined based on these existing annotations.

[0062] This approach enables the transfer of large amounts of readily available textual natural language processing data to contextual understanding tasks in speech recognition at extremely low cost.

[0063] In this embodiment of the invention, highly complex semantically irrelevant text is generated by logical rewriting, which avoids the problem of overly simple negative samples and improves the model's ability to distinguish complex logic and easily confused concepts. At the same time, open-source text with semantic relationship annotations is introduced to perform cross-modal transformation on the data to construct training data, which can generate high-quality auxiliary task training data at low cost and on a large scale, thereby greatly improving the efficiency of the model's training for contextual semantic understanding and the quality of the data foundation.

[0064] Based on the above embodiments, a large language speech recognition model is trained based on predicted semantic relations and semantic relation labels to obtain a speech recognition model, including: Obtain sample text, and generate sample context based on the target entities contained in the sample text; Speech synthesis is performed based on sample text to obtain sample speech. Sample triples are then constructed based on sample context, sample text, and sample speech. Based on the predicted semantic relations and semantic relation labels, the large language speech recognition model is trained to obtain the initial speech recognition model; Based on sample triples, as well as the first text and the second synthesized speech, the initial speech recognition model is jointly trained to obtain the speech recognition model. The tasks in the joint training process include a speech recognition task based on sample triples and a semantic relation prediction task based on the first text and the second synthesized speech.

[0065] Specifically, the process of training a large language speech recognition model based on predicted semantic relations and semantic relation labels to obtain a speech recognition model may include: First, the training data for the main task is constructed. That is, a large number of sample texts need to be determined; these sample texts are unsupervised text data used to construct the training data for the main speech recognition task. The sample texts usually contain rich entity information and have a clear and complete semantic structure.

[0066] Here, we can first acquire large-scale unsupervised text data as the raw corpus, i.e., a text corpus. This text data can include general domain corpora and domain-specific corpora.

[0067] Text data can come from sources such as internet corpora, encyclopedia data, dialogue data, and domain-specific text data. Additionally, a large number of open-source datasets in the field of natural language processing can be used as raw corpora, such as CMeEE, MSRA, CLUE, and CCFBDCI.

[0068] After obtaining the original corpus, in this embodiment of the invention, it is also necessary to clean it to remove invalid characters, low-quality text and duplicate data. Then, based on word frequency statistics and entity judgment, text data containing rich entity information and clear semantic structure can be selected from the cleaned corpus as sample text, and short text data with incomplete semantics can be filtered out to ensure the data quality of the training data constructed later.

[0069] Then, the corresponding sample context can be generated based on the target entities contained in the sample text. Figure 2 This is a flowchart of the sample triplet construction process provided by the present invention, such as... Figure 2 As shown, sample text can be input into a large language model, which will then perform entity identification and information extraction to obtain key entity words, i.e., target entities. The large language model will then continue to think divergently based on the extracted target entities to construct contextual information that is related to the input sample text, i.e., sample context.

[0070] Here, the sample context has an explicit or implicit relationship with the sample text in terms of content, and is used to simulate contextual information such as background descriptions and historical dialogues provided by users in real-world scenarios during training.

[0071] Subsequently, speech synthesis can be performed based on the sample text to obtain the corresponding sample speech; that is, the sample text is converted into a speech signal through speech synthesis technology.

[0072] Then, these three elements can be combined to construct a sample triplet based on the sample context, sample text, and sample speech. This triplet data fully simulates a speech recognition scenario with contextual information.

[0073] Then, a two-stage context-enhanced training method was used to train the LLM-ASR to obtain a speech recognition model.

[0074] Specifically, in the first stage, namely the training stage of context understanding and audio-text alignment capabilities, the loss is calculated based on the predicted semantic relationships and the actual semantic relationship labels output by LLM-ASR, and then LLM-ASR is fine-tuned by instructions.

[0075] This stage of specialized training focuses on enhancing the model's ability to understand context and its modal alignment between audio and text. Since the model after this training stage still needs further optimization and updates, it can be called the initial speech recognition model.

[0076] In the second stage, namely the multi-task joint training stage, the sample triplet data is combined with the auxiliary task data (first text and second synthesized speech) and used together for training the initial speech recognition model.

[0077] Specifically, the tasks in the joint training process include a speech recognition task based on sample triples and a semantic relationship prediction task based on auxiliary task data.

[0078] The speech recognition task is the main task of LLM-SAR, which is to accurately identify the corresponding sample text based on the input sample speech and sample context.

[0079] Semantic relation prediction tasks refer to auxiliary tasks such as semantic relation judgment tasks and semantic consistency judgment tasks, which aim to enhance the model's logical reasoning and semantic matching capabilities.

[0080] The overall loss of joint training includes the loss from the main task and the loss from the auxiliary task. By guiding the same model to complete different tasks with different instructions, the model can master contextual understanding and be seamlessly applied to speech recognition tasks.

[0081] In addition, negative samples with empty context are randomly added during joint training to avoid the model becoming overly dependent on the context or experiencing context illusion.

[0082] In this embodiment of the invention, context understanding is first trained using an auxiliary task, and then multi-task joint training is performed using sample triples. This enables the model to learn the ability to understand and reason about context from the parameter level and to utilize it in speech recognition tasks. At the same time, the joint training mechanism, combined with randomly added empty context negative samples, effectively avoids the model from becoming overly dependent on the context or experiencing context illusions during the reasoning stage, significantly improving the accuracy and robustness of the model in complex semantic scenarios.

[0083] Based on the above embodiments, the overall loss of joint training can be expressed by the following formula: In the formula, The overall loss; For the loss in speech recognition tasks; Loss for the semantic relation prediction task; for The weighting coefficients.

[0084] The loss of the semantic relation prediction task can be specifically expressed by the following formula: In the formula, This represents the total number of samples in the semantic relation prediction task, i.e., the number of the first text or the second synthesized speech. The number of classification categories is 3 if the auxiliary task is a semantic relation judgment task, corresponding to the three categories of implication, contradiction, and neutrality; and 2 if the auxiliary task is a semantic consistency judgment task, corresponding to the two categories of consistency and inconsistency. Indicates the first The semantic relation labels corresponding to each sample are of category 1. The true value (usually in one-hot encoded form, i.e., the true class is 1 and the rest are 0); The model predicts the first Each sample belongs to category The conditional probability.

[0085] The loss of the speech recognition task can be specifically represented by the following formula: In the formula, This represents the total sequence length or the total number of text tokens within the current computation batch in the speech recognition task. The label smoothing coefficient is introduced in speech recognition. Introducing a very small smoothing coefficient can prevent the model from being overconfident in a certain text output, thereby alleviating overfitting and improving the model's generalization ability when faced with easily confused homophones / near-homophones. Indicates the first At each time step or location, the index value of the actual target word in the vocabulary corresponding to the sample text. Integer indices are used to distinguish them from the one-hot vector components in auxiliary tasks. ; This represents the vocabulary size of a large language speech recognition model, i.e., the total number of possible text token types that the model can output; This indicates that the vocabulary includes the actual target words. An index of other words besides; The model represents the first time. At each position, the true target word was correctly predicted. The probability of; The model represents the first time. At this position, the incorrect prediction was for another word in the vocabulary. The probability of.

[0086] Based on the above embodiments, the sample context includes entity-related context and logically related context; Based on the target entities contained in the sample text, a sample context is generated, including: Entity recognition is performed on the sample text to obtain an entity set; The entity set is filtered by key entities to obtain the target entity; By performing correlation divergence on the target entity, anchor point information that can be used to deduce the target entity can be obtained; Generate entity-related contexts that include the target entity, and generate logically related contexts that do not include the target entity based on anchor information.

[0087] Specifically, Figure 3 This is a flowchart of the sample context determination process provided by the present invention, such as... Figure 3 As shown, the process of generating sample context based on the target entities contained in the sample text includes the following steps: First, entity recognition can be performed on the sample text to extract entity words, thereby obtaining an entity set. In practice, named entity recognition technology can be used to extract entity words from each sample text, or an entity matching method based on a preset dictionary / rules can be used to accurately identify various entities contained in the sample text, thereby summarizing them into an entity set.

[0088] Here, the entity set covers entity types including personal names, place names, organization names, proper nouns, product names, terms, time, numbers, etc.

[0089] Next, the entity set needs to be filtered for key entities to obtain the target entities. That is, since the sample text may contain a large number of ordinary entities, in order to make the construction of the sample context more targeted, in this embodiment of the invention, a large language model can be used to judge the importance of entities in the entity set, and key entities can be filtered out and used as target entities.

[0090] It should be noted that when the sample text is short or the semantics are compact, there may be a situation where all entities in the entity set are key entities, and in this case, all of these entities can be used as target entities.

[0091] Subsequently, the target entity can be subjected to relevance divergence to obtain anchor information that can be used to deduce the target entity. That is, the large language model takes the selected target entity as the center and, based on knowledge relevance divergence thinking, mines background information or other entities closely related to the target entity. For example, through knowledge graphs or the inherent knowledge of the large language model, it can associate and expand outward to other entities or background information that are closely related to the target entity in terms of history, geography, attributes, logic, etc., and extract features with unique directional characteristics as anchor information to ensure that the subsequent model can still uniquely deduce the target entity through logical reasoning even with only this anchor information.

[0092] Here, the anchor information is obtained after correlation divergence and can serve as a unique clue or strong logical support to reverse deduce and uniquely determine the target entity or related entities.

[0093] Finally, the large language model can directly generate entity-related contexts containing the target entity, as well as logically related contexts that do not contain the target entity, based on anchor information.

[0094] Among them, entity-related context refers to the target entity contained in the sample text that appears directly in the generated sample context; this type of sample context is mainly used to enhance the model's ability to retrieve and utilize explicit entity clues.

[0095] Logically relevant contexts refer to situations where, during the generation of sample contexts, direct mention of the target entity is deliberately avoided. Instead, the anchor information obtained in the previous step is integrated into the sample context, resulting in related entities or information that have a logical reasoning relationship with the target entity. This type of sample context is mainly used to guide the model to perform logical reasoning in conjunction with the context, enabling it to infer the target entity from related entities.

[0096] In this embodiment of the invention, through entity extraction, relevance divergence, and anchor point mining, two types of structured contexts, entity-related and logically related, are explicitly distinguished and constructed. This not only enhances the model's ability to match explicit entity clues but also effectively covers implicit reasoning scenarios through logically related contexts, greatly expanding the range of contexts available to the model. This effectively breaks through the limitation of traditional solutions that can only work in explicit matching scenarios, thereby improving the recognition accuracy in complex scenarios such as homophones, near-homophones, proper nouns, and pronoun resolution.

[0097] Based on the above embodiments, speech synthesis is performed on the sample text to obtain sample speech, including: Based on the text length, information content, and semantic relevance of the sample context to the corresponding sample text, the quality of the sample context is verified, and the verification result is obtained. Based on the verification results, determine the qualified sample context from the sample context; Speech synthesis is performed based on the sample text corresponding to the qualified sample context and the reference speech to obtain sample speech with different timbres and different speech rates.

[0098] Specifically, the process of synthesizing speech from sample text to obtain sample speech can include: See Figure 3 As can be seen, after generating the sample context, in order to prevent the introduction of noise, it is necessary to perform quality verification on each sample context to verify whether its quality meets the requirements, thereby obtaining the verification result.

[0099] Specifically, this could involve performing a quality check on each sample context based on its text length, information content, and semantic relevance to the corresponding sample text. This check would verify whether the length is appropriate, whether it contains sufficient valid information, and whether it deviates from the semantics of the sample text. Ultimately, this would yield a check result indicating whether each sample context is qualified.

[0100] Text length refers to the number of characters or vocabulary in the sample context. It is used to assess whether the sample context is too short, resulting in semantic incompleteness, or too long, introducing excessive redundant noise.

[0101] Information content refers to the richness of core concepts, entity words, logical clues, etc. contained in the sample context. It is used to measure whether the sample context truly has substantive content to assist speech recognition and avoids vague and meaningless expressions.

[0102] Semantic relevance refers to the degree of connection between the generated sample context and the corresponding sample text in terms of core semantics, scene background, logical relationships, etc.

[0103] Next, based on the verification results, qualified sample contexts can be determined from the sample contexts. That is, based on the verification results of the previous step, sample contexts that contain noise, have vague content, or have no obvious semantic relationship with the sample text are removed, and the remaining contexts are qualified, i.e., qualified sample contexts, thus ensuring the data quality of the training data constructed subsequently.

[0104] Finally, speech synthesis can be performed on the sample text corresponding to the context of the qualified sample to obtain sample speech with different timbres and speaking rates. See also Figure 2 It is evident that, after ensuring the quality of the sample context, the sample text corresponding to the qualified sample context can be input into a zero-shot TTS model, such as CosyVoice2. This type of model supports generating sample speech with acoustic features consistent with the reference speech based solely on a small number of reference speech samples in the reference speech library, without fine-tuning for a specific speaker.

[0105] Here, the reference speech is a prompt speech provided to the zero-shot speech synthesis model, which can mimic the acoustic features of the prompt speech. By randomly inputting different reference speeches, this embodiment of the invention can automatically generate speech signals covering a variety of timbres and different speech rates, thereby ensuring the richness and diversity of the constructed triplet training data in the speaker dimension.

[0106] In this embodiment of the invention, the data quality of the generated sample context is automatically controlled through multi-dimensional verification, effectively filtering out noise and invalid samples and ensuring high-quality training data. At the same time, sample speech with different timbres and speaking speeds is generated using reference speech, eliminating the need for manual annotation of a large amount of audio-text pairing data, reducing data acquisition costs, greatly enriching the diversity of acoustic features in the training data, and improving the generalization ability and robustness of the speech recognition model when facing different speakers.

[0107] Based on the above embodiments, the large language speech recognition model includes a speech encoder, a modality adaptation module, and a large language model. The first text and the second synthesized speech are input into the large language speech recognition model to obtain the predicted semantic relationships output by the large language speech recognition model, including: The second synthesized speech is input into the large language speech recognition model, and processed by the speech encoder and modality adaptation module to obtain a speech representation aligned with the word embedding space of the large language model. The first text is input into the large language speech recognition model, and processed by the word embedding layer of the large language model to obtain the text representation; By inputting the speech representation and text representation into the large language model, the predicted semantic relationships output by the large language model are obtained.

[0108] Specifically, Figure 4 This is a structural diagram of the large language speech recognition model provided by the present invention, as shown below. Figure 4 As shown, LLM-ASR consists of a speech encoder, a modality adapter module, and a large language model (LLM).

[0109] In actual training, the second synthesized speech is input into the large language speech recognition model, where it is processed by the speech encoder and modality adaptation module. Specifically, the input second synthesized speech first passes through the ASR Encoder to extract acoustic features, and then is passed to the Adapter for feature transformation and downsampling. After adaptation, the features originally in the acoustic space are converted into a vector sequence perfectly aligned with the word embedding space (the semantic feature space where the features / vectors reside) within the large language model, thus obtaining the speech representation. .

[0110] Simultaneously, the first text is input into the large language speech recognition model, where it is processed by the word embedding layer of the large language model. That is, the first text, serving as the context, is input into the large language model after being concatenated with the specific task instruction (Instruction). The large language model's built-in word embedding layer (LLM Embedding) performs table lookup and vectorization mapping to map discrete text terms into continuous dense vectors, i.e., continuous text representations. .

[0111] Finally, speech representation and text representation Sequence-level concatenation is performed, such as combining data according to a predefined Prompt template, and then inputting it into the deep Transformer structure of the Large Language Model (LLM) for self-attention interaction and logical reasoning. The LLM integrates the speech input with the context and outputs a response that represents the relationship between the two, i.e., a predicted semantic relationship.

[0112] In this embodiment of the invention, by clarifying the collaborative division of labor between the speech encoder, modality adaptation module and the large language model in the large language speech recognition model, and by aligning and deeply interacting speech representation and text representation in a unified word embedding space, the modal barrier between speech and text is effectively broken down, enabling the large language model to directly and accurately perform joint reasoning on cross-modal inputs, thereby ensuring the accuracy of the model's understanding of the complex semantic relationships between context and audio content.

[0113] The speech recognition device provided by the present invention is described below. The speech recognition device described below can be referred to in correspondence with the speech recognition method described above.

[0114] Figure 5 This is a schematic diagram of the structure of the voice recognition device provided by the present invention, as shown below. Figure 5 As shown, the device includes: The information determination unit 510 is used to determine the speech to be recognized and the context information corresponding to the speech to be recognized; The speech recognition unit 520 is used to input the speech to be recognized and the context information into the speech recognition model to obtain the recognized text output by the speech recognition model; The speech recognition model is built on the basis of the large language speech recognition model; The speech recognition model is trained based on the semantic relationship between the first text and the second synthesized speech; the second synthesized speech is generated based on the second text corresponding to the first text, and the first text and the second text are semantically related or semantically unrelated.

[0115] The speech recognition device provided by this invention determines the speech to be recognized and its corresponding context information, and inputs it into a speech recognition model built on the basis of a large language speech recognition model and trained based on the semantic relationship between the first text and the second synthesized speech to obtain the recognized text. This allows the model to make full use of context information during the speech recognition process, overcoming the shortcomings of existing large language speech recognition systems that lack dedicated training for context utilization and are difficult to fully utilize context information. This improves the accuracy and robustness of speech recognition in complex scenarios such as homophones, near-homophones, proper nouns, and entity disambiguation.

[0116] Based on the above embodiments, the device further includes a model training unit, used for: Based on the second text, speech synthesis is performed to obtain the second synthesized speech; The first text and the second synthesized speech are input into the large language speech recognition model to obtain the predicted semantic relationship output by the large language speech recognition model; Based on the predicted semantic relations and semantic relation labels, the large language speech recognition model is trained to obtain the speech recognition model; The semantic relationship label represents the semantic relationship between the first text and the second text; The semantic relationship tags include semantic category tags and / or semantic consistency tags; The semantic category labels include labels representing implication relationships, labels representing contradictory relationships, and labels representing neutral relationships; the semantic consistency labels include labels representing semantic consistency and labels representing semantic inconsistency.

[0117] Based on the above embodiments, the model training unit is used for: Determine the first text; based on the first text, generate a second text that is semantically related to the first text, and / or, perform at least one of causal reversal, conceptual substitution, or inference subversion on the first text to generate a second text that is semantically unrelated to the first text; And / or, Obtain text pairs with semantic relationship annotations, identify one text in the text pair as the first text, identify the other text in the text pair as the second text, and determine the semantic relationship label based on the semantic relationship annotations.

[0118] Based on the above embodiments, the model training unit is used for: Obtain sample text, and generate sample context based on the target entities contained in the sample text; Speech synthesis is performed based on the sample text to obtain sample speech, and sample triples are constructed based on the sample context, the sample text, and the sample speech. Based on the predicted semantic relations and semantic relation labels, the large language speech recognition model is trained to obtain an initial speech recognition model; Based on the sample triples, the first text, and the second synthesized speech, the initial speech recognition model is jointly trained to obtain the speech recognition model. The tasks in the joint training process include a speech recognition task constructed based on the sample triples, and a semantic relationship prediction task constructed based on the first text and the second synthesized speech.

[0119] Based on the above embodiments, the sample context includes entity-related context and logically related context; The model training unit is used for: Entity recognition is performed on the sample text to obtain an entity set; The target entity is obtained by filtering the entity set by key entities. The target entity is subjected to correlation divergence to obtain anchor point information that can be used to deduce the target entity; Generate an entity-related context that includes the target entity, and generate a logically related context that does not include the target entity based on the anchor information.

[0120] Based on the above embodiments, the model training unit is used for: Based on the text length, information content, and semantic relevance between the sample context and the corresponding sample text, the quality of the sample context is verified to obtain the verification result. Based on the verification result, a qualified sample context is determined from the sample context; Based on the sample text corresponding to the qualified sample context and the reference speech, speech synthesis is performed to obtain sample speech with different timbres and different speech rates.

[0121] Based on the above embodiments, the large language speech recognition model includes a speech encoder, a modality adaptation module, and a large language model; The model training unit is used for: The second synthesized speech is input into the large language speech recognition model, and processed by the speech encoder and the modality adaptation module to obtain a speech representation aligned with the word embedding space of the large language model. The first text is input into the large language speech recognition model, and processed by the word embedding layer of the large language model to obtain a text representation; The speech representation and the text representation are input into the large language model to obtain the predicted semantic relationship output by the large language model.

[0122] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a speech recognition method. This method includes: determining the speech to be recognized and the context information corresponding to the speech; inputting the speech to be recognized and the context information into a speech recognition model to obtain the recognized text output by the speech recognition model; the speech recognition model is built on the basis of a large language speech recognition model; the speech recognition model is trained based on the semantic relationship between a first text and a second synthesized speech; the second synthesized speech is generated based on a second text corresponding to the first text, wherein the first text and the second text are semantically related or semantically unrelated.

[0123] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0124] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the speech recognition method provided by the above methods, the method comprising: determining a speech to be recognized and context information corresponding to the speech to be recognized; inputting the speech to be recognized and the context information into a speech recognition model to obtain recognized text output by the speech recognition model; the speech recognition model is constructed on the basis of a large language speech recognition model; the speech recognition model is trained based on the semantic relationship between a first text and a second synthesized speech; the second synthesized speech is generated based on a second text corresponding to the first text, wherein the first text and the second text are semantically related or semantically unrelated.

[0125] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the speech recognition method provided by the methods described above. The method includes: determining a speech to be recognized and context information corresponding to the speech to be recognized; inputting the speech to be recognized and the context information into a speech recognition model to obtain recognized text output by the speech recognition model; the speech recognition model is constructed based on a large language speech recognition model; the speech recognition model is trained based on the semantic relationship between a first text and a second synthesized speech; the second synthesized speech is generated based on a second text corresponding to the first text, wherein the first text and the second text are semantically related or semantically unrelated.

[0126] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech recognition method, characterized in that, include: Determine the speech to be recognized and the context information corresponding to the speech to be recognized; The speech to be recognized and the context information are input into the speech recognition model to obtain the recognized text output by the speech recognition model; The speech recognition model is built on the basis of the large language speech recognition model; The speech recognition model is trained based on the semantic relationship between the first text and the second synthesized speech; the second synthesized speech is generated based on the second text corresponding to the first text, and the first text and the second text are semantically related or semantically unrelated.

2. The speech recognition method according to claim 1, characterized in that, The speech recognition model is trained based on the following steps: Based on the second text, speech synthesis is performed to obtain the second synthesized speech; The first text and the second synthesized speech are input into the large language speech recognition model to obtain the predicted semantic relationship output by the large language speech recognition model; Based on the predicted semantic relations and semantic relation labels, the large language speech recognition model is trained to obtain the speech recognition model; The semantic relationship label represents the semantic relationship between the first text and the second text; The semantic relationship tags include semantic category tags and / or semantic consistency tags; The semantic category labels include labels representing implication relationships, labels representing contradictory relationships, and labels representing neutral relationships; The semantic consistency tags include tags that represent semantic consistency and tags that represent semantic inconsistency.

3. The speech recognition method according to claim 2, characterized in that, The second text is determined based on the following steps: Determine the first text; based on the first text, generate a second text that is semantically related to the first text, and / or, perform at least one of causal reversal, conceptual substitution, or inference subversion on the first text to generate a second text that is semantically unrelated to the first text; And / or, Obtain text pairs with semantic relationship annotations, identify one text in the text pair as the first text, identify the other text in the text pair as the second text, and determine the semantic relationship label based on the semantic relationship annotations.

4. The speech recognition method according to claim 2, characterized in that, The step of training the large language speech recognition model based on the predicted semantic relations and semantic relation labels to obtain the speech recognition model includes: Obtain sample text, and generate sample context based on the target entities contained in the sample text; Speech synthesis is performed based on the sample text to obtain sample speech, and sample triples are constructed based on the sample context, the sample text, and the sample speech. Based on the predicted semantic relations and semantic relation labels, the large language speech recognition model is trained to obtain an initial speech recognition model; Based on the sample triples, the first text, and the second synthesized speech, the initial speech recognition model is jointly trained to obtain the speech recognition model. The tasks in the joint training process include a speech recognition task constructed based on the sample triples, and a semantic relationship prediction task constructed based on the first text and the second synthesized speech.

5. The speech recognition method according to claim 4, characterized in that, The sample context includes entity-related context and logically related context; The step of generating a sample context based on the target entities contained in the sample text includes: Entity recognition is performed on the sample text to obtain an entity set; The target entity is obtained by filtering the entity set by key entities. The target entity is subjected to correlation divergence to obtain anchor point information that can be used to deduce the target entity; Generate an entity-related context that includes the target entity, and generate a logically related context that does not include the target entity based on the anchor information.

6. The speech recognition method according to claim 4, characterized in that, The process of synthesizing speech based on the sample text to obtain sample speech includes: Based on the text length, information content, and semantic relevance between the sample context and the corresponding sample text, the quality of the sample context is verified to obtain the verification result. Based on the verification result, a qualified sample context is determined from the sample context; Based on the sample text corresponding to the qualified sample context and the reference speech, speech synthesis is performed to obtain sample speech with different timbres and different speech rates.

7. The speech recognition method according to any one of claims 2 to 6, characterized in that, The large language speech recognition model includes a speech encoder, a modality adaptation module, and a large language model. The step of inputting the first text and the second synthesized speech into the large language speech recognition model to obtain the predicted semantic relationship output by the large language speech recognition model includes: The second synthesized speech is input into the large language speech recognition model, and processed by the speech encoder and the modality adaptation module to obtain a speech representation aligned with the word embedding space of the large language model. The first text is input into the large language speech recognition model, and processed by the word embedding layer of the large language model to obtain a text representation; The speech representation and the text representation are input into the large language model to obtain the predicted semantic relationship output by the large language model.

8. A voice recognition device, characterized in that, include: An information determination unit is used to determine the speech to be recognized and the context information corresponding to the speech to be recognized; A speech recognition unit is used to input the speech to be recognized and the context information into a speech recognition model to obtain the recognized text output by the speech recognition model; The speech recognition model is built on the basis of the large language speech recognition model; The speech recognition model is trained based on the semantic relationship between the first text and the second synthesized speech; the second synthesized speech is generated based on the second text corresponding to the first text, and the first text and the second text are semantically related or semantically unrelated.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the speech recognition method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the speech recognition method as described in any one of claims 1 to 7.

Images