Speech recognition methods, devices, electronic devices and storage media

Abstract

This invention provides a speech recognition method, apparatus, electronic device, and storage medium. The method includes: acquiring speech to be recognized; performing speech recognition on the speech to be recognized based on a speech recognition module in an integrated framework to obtain acoustic features and recognition result sampling features of the speech to be recognized; and performing text correction using the acoustic features and recognition result sampling features based on an error correction module in the integrated framework to obtain a speech recognition result of the speech to be recognized. The integrated framework is trained based on sample speech and labeled text of the sample speech. The method, apparatus, electronic device, and storage medium provided by this invention provide speech recognition results with stronger semantic rationality obtained through speech recognition and text correction. Furthermore, joint training of the speech recognition task and text correction task under the integrated framework eliminates the need to acquire additional parallel corpora carrying acoustic features, reducing training costs.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a speech recognition method, apparatus, electronic device, and storage medium. Background Technology

[0002] As one of the important interfaces for human-computer interaction, speech recognition technology has brought users a more convenient experience and lowered the threshold for human-machine interaction.

[0003] In speech recognition solutions, the error rate of recognized characters is typically used as a constraint criterion for training and optimization, or a subsequent error correction module is used to further optimize the recognition results. While the error correction module can utilize the constraints of the language model to optimize the reasonableness of the speech recognition results, it suffers from excessive freedom due to the multiple candidate results of speech recognition as input. This makes it susceptible to the influence of the statistical probability of the language model, causing the error correction results to deviate from the user's original expression. Summary of the Invention

[0004] This invention provides a speech recognition method to address the shortcomings of existing speech recognition technologies, such as high degree of freedom in error correction and easy deviation from the original expression.

[0005] This invention provides a speech recognition method, comprising:

[0006] Acquire the speech to be recognized;

[0007] Based on the speech recognition module in the integrated framework, speech recognition is performed on the speech to be recognized to obtain the acoustic features of the speech to be recognized and the sampling features of the recognition result.

[0008] Based on the error correction module in the integrated framework, the acoustic features and the recognition result sampling features are applied to perform text error correction, thereby obtaining the speech recognition result of the speech to be recognized;

[0009] The integrated framework is trained based on sample speech and the labeled text of the sample speech.

[0010] According to a speech recognition method provided by the present invention, the training steps of the integrated framework include:

[0011] Based on the initial recognition module in the initialization framework, speech recognition is performed on the sample speech to obtain sample acoustic features, sample recognition result sampling features, and preliminary recognition results;

[0012] Based on the initial error correction module in the initialization framework, text error correction is performed by applying the acoustic features of the sample and the sampling features of the sample recognition result to obtain the sample recognition result;

[0013] Based on the preliminary recognition results, the sample recognition results, and the labeled text of the sample speech, a first loss value is determined;

[0014] Based on the first loss value, the parameters of the initialization framework are iterated to obtain the integrated framework.

[0015] According to a speech recognition method provided by the present invention, determining a first loss value based on the preliminary recognition result, the sample recognition result, and the labeled text of the sample speech includes:

[0016] Based on the initial error correction module, text error correction is performed on the masked text to obtain the corrected text, wherein the masked text is obtained by masking the sample text.

[0017] Based on the corrected text and the sample text, determine the alignment loss value;

[0018] Based on the preliminary recognition results, the sample recognition results, and the labeled text of the sample speech, the recognition loss value is determined;

[0019] The first loss value is determined based on the recognition loss value and the alignment loss value.

[0020] According to a speech recognition method provided by the present invention, the step of iterating the parameters of the initialization framework based on the first loss value to obtain the integrated framework includes:

[0021] The sample recognition results are analyzed for downstream tasks to obtain the analysis results of the corresponding downstream tasks; the downstream tasks include at least one of grammar evaluation task, intent classification task and translation task;

[0022] Based on the analysis results and the downstream task labels of the labeled text, a second loss value is determined;

[0023] Based on the first loss value and the second loss value, the parameters of the initialization framework are iterated to obtain the integrated framework.

[0024] According to a speech recognition method provided by the present invention, when the downstream task includes the grammar evaluation task, the step of performing downstream task analysis on the sample recognition result to obtain the analysis result of the downstream task corresponding to the sample recognition result includes:

[0025] Part-of-speech tagging is performed on each word segment in the sample recognition text, and / or syntactic structure recognition is performed on the sample recognition text to obtain the analysis results of the grammar evaluation task corresponding to the sample recognition results.

[0026] According to a speech recognition method provided by the present invention, when the sample speech is phrase-type data, the downstream tasks include a grammar evaluation task and an intent classification task;

[0027] In the case that the sample speech is long-form data, the downstream tasks include grammar evaluation and translation tasks.

[0028] According to a speech recognition method provided by the present invention, the speech recognition module in an integrated framework performs speech recognition on the speech to be recognized to obtain the acoustic features and recognition result sampling features of the speech to be recognized, including:

[0029] Based on the multi-scale coding module in the speech recognition module, the speech to be recognized is acoustically coded to obtain multi-scale acoustic features including shallow hidden layer output and deep hidden layer output.

[0030] Based on the decoding module in the speech recognition module, the acoustic features are semantically decoded, and the hidden layer features generated in the semantic decoding are sampled to obtain the recognition result sampling features.

[0031] According to a speech recognition method provided by the present invention, the step of applying the acoustic features and the recognition result sampling features to perform text error correction based on the error correction module in the integrated framework to obtain the speech recognition result of the speech to be recognized includes:

[0032] Based on the fusion module in the integrated framework, the correlation between the acoustic features and the recognition result sampling features is applied to perform feature fusion on the acoustic features and the recognition result sampling features to obtain fused features;

[0033] Based on the error correction module, the fusion features are applied to perform text error correction, and the speech recognition result of the speech to be recognized is obtained.

[0034] The present invention also provides a voice recognition device, comprising:

[0035] The acquisition unit is used to acquire the speech to be recognized;

[0036] The recognition unit is used to perform speech recognition on the speech to be recognized based on the speech recognition module in the integrated framework, and obtain the acoustic features of the speech to be recognized and the recognition result sampling features.

[0037] The error correction unit is used to perform text error correction based on the error correction module in the integrated framework, applying the acoustic features and the recognition result sampling features, to obtain the speech recognition result of the speech to be recognized;

[0038] The integrated framework is trained based on sample speech and the labeled text of the sample speech.

[0039] The present invention also provides an electronic device, including a microphone, a memory, a processor, and a computer program stored in the memory and executable on the processor. The microphone is used to acquire speech to be recognized. The processor executes an integrated framework in the computer program to realize speech recognition of the speech to be recognized. The integrated framework includes a speech recognition module and an error correction module. The speech recognition module is used to acquire acoustic features and recognition result sampling features of the speech to be recognized. The error correction module is used to apply the acoustic features and the recognition result sampling features to perform text error correction to obtain the speech recognition result of the speech to be recognized. The integrated framework is trained based on sample speech and the labeled text of the sample speech.

[0040] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech recognition method as described above.

[0041] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the speech recognition method as described above.

[0042] The speech recognition method, apparatus, electronic device, and storage medium provided by this invention, within an integrated framework, use the acoustic features of the speech to be recognized and the sampling features of the recognition result as the basis for text correction, thereby making the speech recognition result obtained through speech recognition and text correction more semantically reasonable. Furthermore, by jointly training the speech recognition task and the text correction task within the integrated framework, there is no need to acquire additional parallel corpora carrying acoustic features, thus reducing training costs. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating the speech recognition method provided by the present invention;

[0045] Figure 2 This is one of the flowcharts illustrating the training method of the integrated framework provided by the present invention;

[0046] Figure 3 This is a flowchart illustrating step 240 in the training method of the integrated framework provided by the present invention;

[0047] Figure 4 This is the second flowchart illustrating the training method of the integrated framework provided by the present invention;

[0048] Figure 5 This is a schematic diagram of the structure of the speech recognition device provided by the present invention;

[0049] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0051] In speech recognition solutions, an error correction module is typically added after the speech recognition module to perform secondary optimization of the recognition results. However, in practical applications, error correction modules that use multiple candidate results from speech recognition as input lack acoustic constraints, resulting in excessive freedom and susceptibility to the statistical probability of the language model, causing the correction results to deviate from the user's original expression.

[0052] However, if the acoustic information extracted during speech recognition is simply added to the input of the error correction module, more parallel data is needed to train the error correction module. The use of additional data is difficult and can easily lead to space limitations for the error correction module.

[0053] To address the above problems, embodiments of the present invention provide a speech recognition method. Figure 1 This is a flowchart illustrating the speech recognition method provided by the present invention, as shown below. Figure 1 As shown, the method includes:

[0054] Step 110: Obtain the speech to be recognized.

[0055] Specifically, the speech to be recognized may include the speech that needs to be recognized. The speech to be recognized can be obtained by a sound pickup device, which may be a smartphone, tablet computer, or smart appliance such as a speaker, television, and air conditioner. After the sound pickup device obtains the speech to be recognized through a microphone array, it may also amplify and / or reduce noise, etc. The embodiments of the present invention do not specifically limit this.

[0056] Step 120: Based on the speech recognition module in the integrated framework, perform speech recognition on the speech to be recognized to obtain the acoustic features of the speech to be recognized and the sampling features of the recognition result;

[0057] Step 130: Based on the error correction module in the integrated framework, apply the acoustic features and the recognition result sampling features to perform text error correction, and obtain the speech recognition result of the speech to be recognized;

[0058] The integrated framework is trained based on sample speech and the labeled text of the sample speech.

[0059] Understandably, in related technologies, speech recognition and text correction tasks are independent. Therefore, the multiple candidate results generated by the speech recognition task need to be used as input for the text correction task. However, since the text correction task cannot obtain the acoustic features of the speech to be recognized itself, excessive degrees of freedom can cause the error correction results output by the text correction task to deviate from the user's original expression. If the acoustic information of the speech to be recognized is directly added to the input of the text correction task, then more parallel data needs to be added during the independent training of the text correction task.

[0060] To address this issue, this invention establishes an integrated framework encompassing speech recognition and error correction. In this integrated framework, the acoustic features of the speech to be recognized output by the speech recognition module and the sampled features of the recognition result are used as inputs to the error correction module. This allows the error correction module to reference the acoustic features of the speech to be recognized during the error correction process, thus preventing deviations in the correction results. Furthermore, in this integrated architecture, the speech recognition module and the error correction module are jointly optimized. Therefore, there is no need to acquire additional parallel corpora carrying acoustic features simply because the error correction module now includes acoustic features as input, reducing training costs.

[0061] Before executing step 120, an integrated framework can be obtained through training. Specifically, an initialization framework can be constructed first. This initialization framework can include an initial recognition module and an initial error correction module. The initial recognition module can be a model prepared for speech recognition after parameter initialization, or it can be a pre-trained model with speech recognition capabilities. The initial error correction module can be a model prepared for text error correction after parameter initialization, or it can be a pre-trained language model with text error correction capabilities. This embodiment of the invention does not specifically limit the specifics of this.

[0062] In addition, sample speech and its labeled text can be collected. For example, sample speech can be collected and the labeled text corresponding to the sample speech can be obtained through manual annotation. Alternatively, the labeled text can be collected first, and then the sample speech corresponding to the labeled text can be obtained through manual recording or speech synthesis.

[0063] Subsequently, the sample speech can be used as input to the initialization framework. The initial recognition module within the initialization framework performs speech recognition on the sample speech, obtaining its acoustic features and recognition result sampling features. Then, the initial error correction module within the initialization framework applies the acoustic features of the sample speech and the recognition result sampling features to perform text error correction, obtaining the recognition result for the sample speech output by the initialization framework, i.e., the sample recognition result. After obtaining the sample recognition result, a loss function can be calculated based on the difference between the sample recognition result and the labeled text of the sample speech. The parameters of the entire initialization framework are then iterated based on the loss function to obtain the integrated framework. That is, in the training process of the integrated framework, only the sample speech and labeled text typically required for training a speech recognition model are needed to achieve good results.

[0064] The resulting integrated framework, in practical applications, can be used to perform speech recognition on the speech to be recognized through the speech recognition module within the integrated framework. In this process, the speech recognition module first extracts acoustic features from the speech to be recognized, and then decodes the extracted acoustic features to obtain the recognition result and the sampling features of the recognition result.

[0065] The acoustic features here can be the acoustic features of each frame of the speech to be recognized extracted by Fast Fourier Transform (FFT) after the speech to be recognized is segmented and windowed, such as Mel Frequency Cepstrum Coefficient (MFCC) features or Perceptual Linear Predictive (PLP) features, etc. It can also be the hidden layer features that reflect local and / or global acoustic information obtained by further feature encoding operations on the basis of MFCC, PLP and other features. The acoustic features referred to here can be single-scale features or multi-scale fused features. The embodiments of the present invention do not make specific limitations on this.

[0066] The recognition result sampling features are used to reflect the features of the recognition result of the speech to be recognized obtained based on the speech recognition module. The recognition result sampling features can be obtained by sampling and encoding the recognition result, or by sampling the hidden layer features that can characterize the recognition result. The embodiments of the present invention do not make specific limitations on this.

[0067] After obtaining the acoustic features of the speech to be recognized and the sampling features of the recognition result through the speech recognition module, the error correction module in the integrated framework can be used to apply the acoustic features and the sampling features of the recognition result for text error correction, thereby obtaining a speech recognition result with stronger semantic rationality after error correction. It can be understood that, compared with traditional text error correction tasks, the error correction module in this embodiment of the invention also refers to the acoustic features of the speech to be recognized. Under the constraints of acoustic features, semantic-based text error correction can better fit the original meaning of the speech to be recognized, thereby enhancing the semantic rationality of speech recognition.

[0068] The method provided in this invention uses the acoustic features of the speech to be recognized and the sampling features of the recognition result generated by speech recognition as the basis for text correction within an integrated framework, thereby making the speech recognition result obtained through speech recognition and text correction more semantically reasonable. Furthermore, by jointly training the speech recognition task and the text correction task within the integrated framework, there is no need to acquire additional parallel corpora carrying acoustic features, thus reducing training costs.

[0069] Based on the above embodiments, Figure 2 This is one of the flowcharts illustrating the training method of the integrated framework provided by this invention, such as... Figure 2 As shown, the training steps of the integrated framework include:

[0070] Step 210: Based on the initial recognition module in the initialization framework, perform speech recognition on the sample speech to obtain sample acoustic features, sample recognition result sampling features, and preliminary recognition results.

[0071] Step 220: Based on the initial error correction module in the initialization framework, apply the sample acoustic features and the sample recognition result sampling features to perform text error correction and obtain the sample recognition result.

[0072] Specifically, the initialization framework may include an initial recognition module and an initial error correction module. The initial recognition module may be a model prepared for speech recognition after parameter initialization, or a pre-trained model with speech recognition function. The initial error correction module may be a model prepared for text error correction after parameter initialization, or a pre-trained language model with text error correction function. This embodiment of the invention does not specifically limit the specifics of these modules.

[0073] Preferably, the initial recognition module can be obtained by initial training using sample text and its label text; the initial error correction module can be obtained by initial training using full text data. The initial training of the initial error correction model can be aimed at the masked language model task and the error correction task. For example, the masked speech model task can be trained first, and then the error correction task can be fine-tuned based on the model obtained from the masked speech model task training.

[0074] After determining the initialization framework, sample speech, and their labeled text, training can begin on the initialization framework. Specifically, the sample speech can be used as input to the initialization framework. The initial recognition module within the initialization framework performs speech recognition on the sample speech, obtaining the acoustic features of the sample speech, the sampling features of the recognition result, and the preliminary recognition result. Here, for ease of description, the acoustic features of the sample speech and the sampling features of the recognition result are denoted as sample acoustic features and sample recognition result sampling features, respectively. It can be understood that the preliminary recognition result is the recognition result output by the initial recognition model; that is, the initial recognition result is in text form. The initial recognition result sampling features can be obtained by sampling features from the initial recognition result itself, or by sampling features from the hidden layer features used to obtain the initial recognition result. For example, Gumbel softmax can be used to sample the hidden layer features used to obtain the initial recognition result to obtain the initial recognition result sampling features.

[0075] Subsequently, the initial error correction module in the initialization framework applies the acoustic features of the sample speech and the sampling features of the recognition result to perform text error correction, thereby obtaining the recognition result of the sample speech output by the initialization framework, i.e., the sample recognition result.

[0076] Step 230: Based on the preliminary recognition result, the sample recognition result, and the labeled text of the sample speech, determine the first loss value.

[0077] Specifically, the preliminary recognition result obtained in step 210 and the sample recognition result obtained in step 220 can both be understood as the results obtained by the initialization framework in performing speech recognition on the sample speech. The preliminary recognition result and the sample recognition result can be compared with the label text of the sample speech, and the loss value generated by the initialization framework in performing speech recognition and text correction, i.e., the first loss value, can be determined based on the difference between the preliminary recognition result and the label text, as well as the difference between the sample recognition result and the label text.

[0078] Step 240: Based on the first loss value, perform parameter iteration on the initialization framework to obtain the integrated framework.

[0079] Specifically, after calculating the first loss value, the first loss value can be used as the loss value required for parameter iteration of the initialization framework. Parameter iteration is then performed to obtain the initialization framework after parameter iteration, i.e., the integrated framework.

[0080] The method provided in this embodiment of the invention refers to the intermediate results of the initialization framework, namely the preliminary recognition results, and the final output of the initialization framework, namely the sample recognition results, during the joint training process. The integrated framework obtained by parameter iteration in this way pays more attention to the integrity of speech recognition and text error correction. Speech recognition based on the integrated framework can also have stronger semantic rationality.

[0081] In related technologies, error correction models often use text as input. However, in this embodiment of the invention, the error correction module uses acoustic features and sampling features of the recognition result as input. Considering that the initial error correction module in the initialization framework can be a pre-trained language model with text as input, in order to pull up the feature representation with text as input and the feature representation with acoustic features and sampling features of the recognition result as input, thereby achieving alignment in the feature space, based on any of the above embodiments, step 230 specifically includes:

[0082] Based on the initial error correction module, text error correction is performed on the masked text to obtain the corrected text, wherein the masked text is obtained by masking the sample text.

[0083] Based on the corrected text and the sample text, determine the alignment loss value;

[0084] Based on the preliminary recognition results, the sample recognition results, and the labeled text of the sample speech, the recognition loss value is determined;

[0085] The first loss value is determined based on the recognition loss value and the alignment loss value.

[0086] Specifically, during the parameter iteration of the initialization framework using sample speech and its labeled text, parallel text pairs, namely sample text and its mask text, can be added. The mask text is used as the input of the initial error correction module, and the sample text is used as the target of the error correction text obtained by the initial error correction module in correcting the mask text. The alignment loss value is obtained by calculating the difference between the sample text and the error correction text.

[0087] It is understandable that the recognition loss value determined based on the preliminary recognition results, sample recognition results, and label text, as well as the alignment loss value determined based on the sample text and the error correction text, can all be used to initialize the parameter iteration of the initial error correction module in the framework. Furthermore, the parameter iteration of the initial error correction module by applying the recognition loss value and the parameter iteration of the initial error correction module by applying the alignment loss value are performed alternately.

[0088] That is, after obtaining the recognition loss value and the alignment loss value, the first loss value can be determined, and the parameters can be iterated based on the first loss value. In this process, the initial error correction module is trained alternately based on the recognition loss value and the alignment loss value in the first loss value. The acoustic features, recognition result sampling features, and text feature spaces, which are all inputs to the initial error correction module, can be aligned, thereby constructing a unified spatial representation of speech and text. This enables the initial error correction module built based on the language model to adapt to error correction tasks with acoustic features and recognition result sampling features as inputs.

[0089] Based on any of the above embodiments, the determination of the first loss value can be expressed as the following formula:

[0090] L p =L asr +βL cor +γL MLM

[0091] In the formula, L p L is the first loss value. asr L is the recognition loss value determined based on the preliminary recognition results and the label text. Cor L is the recognition loss value determined based on the sample recognition results and the label text. MLM To align the loss values; β and γ represent the balance coefficients used to balance the various types of losses mentioned above.

[0092] Among them, L asr =max θ ∑ i log P(y i |X;θ);

[0093] In the formula, X represents the sample speech as input, θ represents the initial training parameters of the recognition module, and y i This represents the i-th character of the tag text.

[0094] in,

[0095] In the formula, H l H represents the shallow hidden layer features output by the initial recognition module. h H represents the deep hidden layer features output by the initial recognition module. l and H h The acoustic features that make up the sample This represents the sampling output of the initial recognition module after Gumbel softmax, i.e., the sampled features of the sample recognition result. θ represents the training parameters of the initial recognition module. y represents the training parameters of the initial error correction module. i This represents the i-th character of the tag text.

[0096] in,

[0097] In the formula, This indicates text input that has been masked, i.e., masked text. y represents the training parameters of the initial error correction module. j This indicates the decoding result output corresponding to the masked part, that is, the part of the sample text that is covered up.

[0098] In speech recognition solutions, the error rate of recognized characters is typically used as a constraint criterion for training and optimization, or a correction module trained with the error rate of labeled characters as a constraint criterion is used to further optimize the recognition results. However, both approaches focus solely on the character error rate constraint, lacking understanding of the scenarios in which downstream applications of speech recognition are used, resulting in unsatisfactory performance in these applications. To address this issue, this invention introduces a multi-task evaluation system for integrated framework training.

[0099] Based on any of the above embodiments Figure 3 This is a flowchart illustrating step 240 in the training method of the integrated framework provided by the present invention, as shown below. Figure 3 As shown, step 240 includes:

[0100] Step 241: Perform downstream task analysis on the sample recognition results to obtain the analysis results of the downstream tasks corresponding to the sample recognition results; the downstream tasks include at least one of the following: grammar evaluation task, intent classification task, and translation task;

[0101] Step 242: Based on the analysis results and the downstream task labels of the labeled text, determine the second loss value;

[0102] Step 243: Based on the first loss value and the second loss value, perform parameter iteration on the initialization framework to obtain the integrated framework.

[0103] Specifically, when iterating the parameters of the initialization framework, not only can the first loss value, which represents the loss generated by the initialization framework in speech recognition and text correction, be used, but a second loss value, which reflects the rationality of the sample recognition results output by the initialization framework in downstream tasks, can also be combined.

[0104] Here, the determination of the second loss value can be based on steps 241 and 242. Specifically, in step 241, downstream task analysis can be performed on the sample recognition results output by the initialization framework. It is understood that different downstream tasks correspond to different downstream task analysis methods. For example, the grammar evaluation task is used to evaluate the grammatical rationality of the text. The analysis for the grammar evaluation task can be part-of-speech recognition of each word in the sample recognition results, or syntactic structure recognition of the sample recognition results; the intent classification task is used to analyze the intent contained in the text. The analysis for the intent classification task can be intent classification of the sample recognition results; the translation task is used to perform text translation. The analysis for the translation task can be text translation of the sample recognition results.

[0105] Understandably, if the speech recognition performance of the initialization framework can adapt to downstream tasks, this will be reflected in the sample recognition results, specifically in the analysis results of the downstream tasks corresponding to the sample recognition results being similar to or even identical to the downstream task labels of the labeled text. Here, the downstream task labels of the labeled text refer to the results obtained from downstream task analysis of the labeled text. For example, in an intent classification task, the closer the intent classification result of the sample recognition result is to the intent classification result of the labeled text, the better the initialization framework can adapt to the intent classification task, and the stronger the scene comprehensibility under the intent classification task.

[0106] Based on this, in step 240, the second loss value can be determined by comparing the differences between the analysis results and the downstream task labels.

[0107] Subsequently, the parameters of the initialization framework can be iterated based on the first and second loss values. For example, the first and second loss values ​​can be weighted and summed, and the parameters can be iterated based on the loss value obtained from the weighted summation.

[0108] The method provided in this invention determines a second loss value for parameter iteration by comparing the analysis results of downstream tasks with the downstream task labels of the labeled text. This allows the initialization framework to better adapt to downstream tasks during parameter iteration. The resulting integrated framework outputs speech recognition results with stronger scenario comprehensibility for downstream tasks and achieves better results in downstream applications.

[0109] Based on any of the above embodiments, when the downstream task includes the syntax evaluation task, step 241 includes:

[0110] Part-of-speech tagging is performed on each word segment in the sample recognition text, and / or syntactic structure recognition is performed on the sample recognition text to obtain the analysis results of the grammar evaluation task corresponding to the sample recognition results.

[0111] Specifically, for downstream tasks that include grammar evaluation, the analysis of downstream tasks related to grammar evaluation can include part-of-speech tagging and / or syntactic structure recognition. That is, for grammar evaluation of the sample recognition results, two methods can be used: first, comparing the part-of-speech tags of each word in the sample recognition results with those in the labeled text; second, comparing the syntactic structure of the sample recognition results with that of the labeled text. Here, syntactic structure can specifically include the dependency relationships between words in the text and the positional encoding of words with dependency relationships within the text. Both part-of-speech tagging and syntactic structure recognition can be obtained using general natural language processing tools, and will not be elaborated upon here.

[0112] Based on any of the above embodiments, when the downstream task includes the syntax evaluation task, the second loss value in step 242 includes the syntax evaluation task loss value. The syntax evaluation task loss value can be determined based on the part-of-speech loss value and / or the syntactic structure loss value. For example, it can be the part-of-speech loss value or the syntactic structure loss value, or it can be the weighted sum of the part-of-speech loss value and the syntactic structure loss value. The embodiments of the present invention do not specifically limit this.

[0113] Here, the part-of-speech loss value L pos and syntactic structure loss value L dp This can be expressed as the following formula:

[0114]

[0115]

[0116] Among them, Y' emb It represents the features obtained by fusing the acoustic features of the sample and the sampling features of the sample recognition results, or it represents the encoded features of the masked text; This represents the training parameters of the initial error correction module; y pi This represents the label output by the corresponding part-of-speech tagging result, i.e., the part-of-speech tag in the downstream task label of the label text. arc y head The label represents the output of the corresponding syntactic structure recognition result, that is, the syntactic structure label in the downstream task label of the label text, including the decoding results of the relation and position information of dependency syntax relations.

[0117] Based on any of the above embodiments, when the downstream task includes the intent classification task, step 241 includes:

[0118] The sample text is classified by intent to obtain the analysis result of the intent classification task corresponding to the sample recognition result, i.e., the intent classification result.

[0119] Accordingly, the second loss value in step 242 includes the intent classification task loss value, L. int This can be expressed as the following formula:

[0120]

[0121] Among them, Y' emb This refers to the features obtained by fusing the acoustic features of the sample and the sampling features of the sample recognition results, or it refers to the encoded features of the masked text. y represents the training parameters of the initial error correction module. int This represents the label output for the corresponding intent classification result, i.e., the intent label in the downstream task label of the label text.

[0122] Based on any of the above embodiments, when the downstream task includes the translation task, step 241 includes:

[0123] The sample recognition text is translated to obtain the analysis result of the translation task corresponding to the sample recognition result, i.e., the translated text.

[0124] Accordingly, the second loss value in step 242 includes the translation task loss value, L. trs This can be expressed as the following formula:

[0125]

[0126] Among them, Y' emb This refers to the features obtained by fusing the acoustic features of the sample and the sampling features of the sample recognition results, or it refers to the encoded features of the masked text. This represents the training parameters of the initial error correction module. The training parameters for the translation task are derived from existing text translation task parameters and are not updated during training. trs The label represents the translation target corresponding to the text, that is, the translation label in the downstream task label of the labeled text.

[0127] Based on any of the above embodiments, the downstream tasks applied during training are also different for different types of sample speech.

[0128] Among them, the grammar evaluation task is for the full sample speech, that is, regardless of the type of sample speech, the grammar evaluation task can be used as a downstream task for training the integrated framework.

[0129] Intent classification tasks are mostly applied in human-computer interaction scenarios, such as voice control of electronic devices. Considering that control commands are mostly short phrases, intent classification tasks are mostly for phrase-type data. That is, for sample speech that belongs to phrase-type data, intent classification tasks can be used as downstream tasks for training an integrated framework.

[0130] Translation tasks are frequently used in scenarios such as meetings and audio / video presentations. The texts being translated are mostly long sentences or paragraphs, making them primarily suited for long-form data. In other words, for sample speech data that falls under the category of long-form data, translation tasks can be used as a downstream task in training an integrated framework.

[0131] It is understandable that long speech data and short speech data are relative terms. Short speech data has a shorter speech duration and covers less information compared to long speech data.

[0132] Based on any of the above embodiments, step 120 includes:

[0133] Based on the multi-scale coding module in the speech recognition module, the speech to be recognized is acoustically coded to obtain multi-scale acoustic features including shallow hidden layer output and deep hidden layer output.

[0134] Based on the decoding module in the speech recognition module, the acoustic features are semantically decoded, and the hidden layer features generated in the semantic decoding are sampled to obtain the recognition result sampling features.

[0135] Specifically, in the integrated framework, the speech recognition module is an encoder + decoder structure. The speech recognition module can include a sequentially connected multi-scale encoding module and a decoding module. The multi-scale encoding module is used to encode the acoustic features of the input speech to be recognized. To enrich the diverse expression of acoustic features, this acoustic feature encoding is multi-scale. Specifically, it can obtain shallow hidden layer outputs that focus more on local acoustic information, and then perform further feature extraction on the shallow hidden layer outputs to obtain deep hidden layer outputs that focus more on global acoustic information. In this embodiment of the invention, the shallow hidden layer outputs and deep hidden layer outputs can be used as acoustic features, thereby providing richer and more diverse acoustic information for the error correction module to perform text correction.

[0136] After the multi-scale coding module completes the acoustic feature encoding, the decoding module can perform semantic decoding on the encoded acoustic features. The hidden features generated during the semantic decoding process can reflect the features of the final speech recognition result. Therefore, the recognition result sampling features can be obtained by sampling the hidden features.

[0137] The method provided in this embodiment of the invention provides richer and more diverse multi-scale acoustic features for the operation of the error correction module within an integrated framework, thus providing conditions for the integrated framework to output more reasonable speech recognition results.

[0138] Based on any of the above embodiments, step 130 includes:

[0139] Based on the fusion module in the integrated framework, the correlation between the acoustic features and the recognition result sampling features is applied to perform feature fusion on the acoustic features and the recognition result sampling features to obtain fused features;

[0140] Based on the error correction module, the fusion features are applied to perform text error correction, and the speech recognition result of the speech to be recognized is obtained.

[0141] Specifically, within the integrated framework, a fusion module can be added between the speech recognition module and the error correction module. This fusion module performs feature fusion on the acoustic features output by the speech recognition module and the sampled features of the recognition result, resulting in fused features that belong to the same feature space as conventional text features. This allows the error correction module to directly apply the fused features for text error correction.

[0142] Furthermore, when fusing acoustic features and recognition result sampling features, the fusion module can refer to the correlation between the acoustic features and recognition result sampling features to determine their weights during feature fusion, thereby performing weighted fusion based on these weights. For example, a cross-attention mechanism can be used to fuse acoustic features and recognition result sampling features.

[0143] Based on any of the above embodiments Figure 4 This is the second flowchart illustrating the training method of the integrated framework provided by this invention, as shown below. Figure 4 As shown, the integrated framework can be trained based on an initialization framework that includes an initial recognition module and an initial error correction module.

[0144] The initialization error correction module can be trained using the full amount of text data. This text data can include text corpora matching the audio data or additional plain text corpora. To match the error correction task, the text data can be constructed to obtain the text that needs error correction. For example, common error types in speech recognition can be simulated through substitution errors, deletion errors, and insertion errors. Here, substitution errors involve randomly selecting some characters or words in the sentence and replacing them with homophones or near-homophones; deletion errors involve randomly selecting some characters or words in the sentence and deleting them; and insertion errors involve randomly selecting some characters or words in the sentence and inserting new characters or words.

[0145] Following this, the initial training of the error correction module can be performed. In this process, the language model can first be trained using the full amount of original text data, employing the existing masked language model training method. Then, based on the constructed error-correcting text data, the language model is fine-tuned. The objective function for this fine-tuning can be trained using the known Connectionist Temporal Classification (CTC) algorithm, employing non-autoregressive training. This will result in an initial error correction module that conforms to the text information features required for the downstream error correction task.

[0146] In addition, the initialization of the recognition module can also be obtained through speech recognition initialization. This initialization can be achieved using matching speech and text, for example, it can be obtained by training with CTC.

[0147] After obtaining the initialization framework, the initialization recognition module and the initialization error correction module can be jointly trained. During this process, the acoustic features of the sample speech obtained by the initialization recognition module and the sample recognition result sampling features can be used as input to the initialization error correction module after feature fusion.

[0148] The initial recognition results output by the initial recognition module in the initialization framework can be compared with the labeled text of the sample speech to generate a speech recognition loss value L for the speech recognition task. asr The sample recognition results output by the initial error correction module in the initialization framework can be compared with the labeled text of the sample speech to generate a text error correction loss value L for the speech error correction task. Cor .

[0149] In addition, to align the fused features obtained from feature fusion with the features of the plain text input, parallel text pairs can be added, namely, sample text and its mask text. The mask text is used as the input to the initial error correction module, and the sample text is used as the target of the error-corrected text obtained by the initial error correction module correcting the mask text. By calculating the difference between the sample text and the error-corrected text, the alignment loss value L with the mask language model as the target is obtained. MLM Furthermore, for the initial error correction module, the application of text correction loss values ​​and alignment loss values ​​for parameter iteration is performed alternately.

[0150] Here, the speech recognition loss value L asr Text correction loss value L Cor Alignment loss value L MLM By applying weights, we can obtain the first loss value.

[0151] Furthermore, for the sample recognition results and error-corrected text output by the initialization framework, at least one of the following objectives—intent classification, grammatical evaluation, and translation—can be introduced to constrain speech recognition for downstream tasks. Specifically, the analysis results obtained from downstream task analysis of the sample recognition results and error-corrected text can be compared with the downstream task labels of the labeled text and sample text to obtain results including, but not limited to, part-of-speech loss values ​​L. pos Syntactic structure loss value L dp Intent classification task loss value L int Translation task loss value L trs The second loss value includes at least one of them.

[0152] After obtaining the first and second loss values, the parameters of the initialization framework can be iterated based on these two values ​​to obtain an integrated framework.

[0153] The method provided in this invention performs joint training on speech recognition and text correction tasks within an integrated framework, eliminating the need for additional parallel corpora carrying acoustic features and reducing training costs. During joint training, the intermediate results of the initialization framework (i.e., preliminary recognition results) and the final output of the initialization framework (i.e., sample recognition results) are referenced. The resulting integrated framework, obtained through parameter iteration, focuses more on the holistic nature of speech recognition and text correction. Speech recognition implemented based on this integrated framework also possesses stronger semantic rationality.

[0154] Furthermore, the initial error correction module is trained alternately based on the recognition loss value and alignment loss value in the first loss value. The acoustic features, recognition result sampling features, and text feature spaces, which are all inputs to the initial error correction module, can be aligned, thereby constructing a unified spatial representation of speech and text. This enables the initial error correction module built based on the language model to adapt to error correction tasks with acoustic features and recognition result sampling features as inputs.

[0155] Furthermore, by comparing the analysis results of downstream tasks with the downstream task labels of the labeled text, a second loss value is determined for parameter iteration, enabling the initialization framework to better adapt to downstream tasks during parameter iteration. The resulting integrated framework outputs speech recognition results with stronger scenario comprehensibility for downstream tasks, achieving better results in downstream applications.

[0156] Based on any of the above embodiments Figure 5 This is a schematic diagram of the structure of the voice recognition device provided by the present invention, as shown below. Figure 5 As shown, the device includes:

[0157] Acquisition unit 510 is used to acquire the speech to be recognized;

[0158] The recognition unit 520 is used to perform speech recognition on the speech to be recognized based on the speech recognition module in the integrated framework, and obtain the acoustic features of the speech to be recognized and the recognition result sampling features.

[0159] The error correction unit 530 is used to perform text error correction based on the error correction module in the integrated framework, applying the acoustic features and the recognition result sampling features, to obtain the speech recognition result of the speech to be recognized.

[0160] The integrated framework is trained based on sample speech and the labeled text of the sample speech.

[0161] The apparatus provided in this invention uses the acoustic features of the speech to be recognized and the sampling features of the recognition result generated by speech recognition as the basis for text correction within an integrated framework, thereby making the speech recognition result obtained through speech recognition and text correction more semantically reasonable. Furthermore, by jointly training the speech recognition task and the text correction task within the integrated framework, there is no need to additionally acquire parallel corpora carrying acoustic features, thus reducing training costs.

[0162] Based on any of the above embodiments, the device further includes a training unit, used for:

[0163] Based on the initial recognition module in the initialization framework, speech recognition is performed on the sample speech to obtain sample acoustic features, sample recognition result sampling features, and preliminary recognition results;

[0164] Based on the initial error correction module in the initialization framework, text error correction is performed by applying the acoustic features of the sample and the sampling features of the sample recognition result to obtain the sample recognition result;

[0165] Based on the preliminary recognition results, the sample recognition results, and the labeled text of the sample speech, a first loss value is determined;

[0166] Based on the first loss value, the parameters of the initialization framework are iterated to obtain the integrated framework.

[0167] Based on any of the above embodiments, the training unit is specifically used for:

[0168] Based on the initial error correction module, text error correction is performed on the masked text to obtain the corrected text, wherein the masked text is obtained by masking the sample text.

[0169] Based on the corrected text and the sample text, determine the alignment loss value;

[0170] Based on the preliminary recognition results, the sample recognition results, and the labeled text of the sample speech, the recognition loss value is determined;

[0171] The first loss value is determined based on the recognition loss value and the alignment loss value.

[0172] Based on any of the above embodiments, the training unit is specifically used for:

[0173] The sample recognition results are analyzed for downstream tasks to obtain the analysis results of the corresponding downstream tasks; the downstream tasks include at least one of grammar evaluation task, intent classification task and translation task;

[0174] Based on the analysis results and the downstream task labels of the labeled text, a second loss value is determined;

[0175] Based on the first loss value and the second loss value, the parameters of the initialization framework are iterated to obtain the integrated framework.

[0176] Based on any of the above embodiments, the training unit is specifically used for:

[0177] Part-of-speech tagging is performed on each word segment in the sample recognition text, and / or syntactic structure recognition is performed on the sample recognition text to obtain the analysis results of the grammar evaluation task corresponding to the sample recognition results.

[0178] Based on any of the above embodiments, when the sample speech is phrase-type data, the downstream tasks include a grammar evaluation task and an intent classification task;

[0179] In the case that the sample speech is long-form data, the downstream tasks include grammar evaluation and translation tasks.

[0180] Based on any of the above embodiments, the identification unit 520 is specifically used for:

[0181] Based on the multi-scale coding module in the speech recognition module, the speech to be recognized is acoustically coded to obtain multi-scale acoustic features including shallow hidden layer output and deep hidden layer output.

[0182] Based on the decoding module in the speech recognition module, the acoustic features are semantically decoded, and the hidden layer features generated in the semantic decoding are sampled to obtain the recognition result sampling features.

[0183] Based on any of the above embodiments, the error correction unit 530 is specifically used for:

[0184] Based on the fusion module in the integrated framework, the correlation between the acoustic features and the recognition result sampling features is applied to perform feature fusion on the acoustic features and the recognition result sampling features to obtain fused features;

[0185] Based on the error correction module, the fusion features are applied to perform text error correction, and the speech recognition result of the speech to be recognized is obtained.

[0186] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, a communication bus 640, and a microphone 650. The processor 610, communication interface 620, memory 630, and microphone 650 communicate with each other via the communication bus 640. The microphone 650 can acquire the speech to be recognized and transmit it to the processor 610 via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to realize speech recognition of the speech to be recognized through an integrated framework. The integrated framework includes a speech recognition module and an error correction module. The speech recognition module is used to acquire the acoustic features and recognition result sampling features of the speech to be recognized. The error correction module is used to apply the acoustic features and the recognition result sampling features to perform text error correction to obtain the speech recognition result of the speech to be recognized. The integrated framework is trained based on sample speech and the labeled text of the sample speech.

[0187] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0188] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the speech recognition method provided by the above methods. The method includes: acquiring speech to be recognized; performing speech recognition on the speech to be recognized based on a speech recognition module in an integrated framework to obtain acoustic features and recognition result sampling features of the speech to be recognized; and performing text error correction based on an error correction module in the integrated framework by applying the acoustic features and the recognition result sampling features to obtain a speech recognition result of the speech to be recognized. The integrated framework is trained based on sample speech and the labeled text of the sample speech.

[0189] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the speech recognition method provided by the above methods. The method includes: acquiring speech to be recognized; performing speech recognition on the speech to be recognized based on a speech recognition module in an integrated framework to obtain acoustic features and recognition result sampling features of the speech to be recognized; and performing text error correction based on an error correction module in the integrated framework, applying the acoustic features and the recognition result sampling features to obtain a speech recognition result of the speech to be recognized. The integrated framework is trained based on sample speech and the labeled text of the sample speech.

[0190] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0191] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0192] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech recognition method, characterized in that, include: Acquire the speech to be recognized; Based on the speech recognition module in the integrated framework, speech recognition is performed on the speech to be recognized to obtain the acoustic features of the speech to be recognized and the sampling features of the recognition result. Based on the error correction module in the integrated framework, the acoustic features and the recognition result sampling features are applied to perform text error correction, thereby obtaining the speech recognition result of the speech to be recognized; The integrated framework is trained based on sample speech and the labeled text of the sample speech; The training steps of the integrated framework include: Based on the initial recognition module in the initialization framework, speech recognition is performed on the sample speech to obtain sample acoustic features, sample recognition result sampling features, and preliminary recognition results; Based on the initial error correction module in the initialization framework, text error correction is performed by applying the acoustic features of the sample and the sampling features of the sample recognition result to obtain the sample recognition result; Based on the preliminary recognition results, the sample recognition results, and the labeled text of the sample speech, a first loss value is determined; Based on the first loss value, the parameters of the initialization framework are iterated to obtain the integrated framework.

2. The speech recognition method according to claim 1, characterized in that, The step of determining the first loss value based on the preliminary recognition result, the sample recognition result, and the labeled text of the sample speech includes: Based on the initial error correction module, text error correction is performed on the masked text to obtain the corrected text, wherein the masked text is obtained by masking the sample text. Based on the corrected text and the sample text, determine the alignment loss value; Based on the preliminary recognition results, the sample recognition results, and the labeled text of the sample speech, the recognition loss value is determined; The first loss value is determined based on the recognition loss value and the alignment loss value.

3. The speech recognition method according to claim 1, characterized in that, The step of iterating the parameters of the initialization framework based on the first loss value to obtain the integrated framework includes: The sample recognition results are analyzed for downstream tasks to obtain the analysis results of the corresponding downstream tasks; the downstream tasks include at least one of grammar evaluation task, intent classification task and translation task; Based on the analysis results and the downstream task labels of the labeled text, a second loss value is determined; Based on the first loss value and the second loss value, the parameters of the initialization framework are iterated to obtain the integrated framework.

4. The speech recognition method according to claim 3, characterized in that, When the downstream task includes the grammar evaluation task, the downstream task analysis of the sample recognition result to obtain the analysis result of the downstream task corresponding to the sample recognition result includes: Part-of-speech tagging is performed on each word segment in the sample recognition text, and / or syntactic structure recognition is performed on the sample recognition text to obtain the analysis results of the grammar evaluation task corresponding to the sample recognition results.

5. The speech recognition method according to claim 3, characterized in that, When the sample speech is phrase-type data, the downstream tasks include a grammatical evaluation task and an intent classification task; In the case that the sample speech is long-form data, the downstream tasks include grammar evaluation and translation tasks.

6. The speech recognition method according to any one of claims 1 to 5, characterized in that, The speech recognition module in the integrated framework performs speech recognition on the speech to be recognized, obtaining the acoustic features of the speech to be recognized and the sampling features of the recognition result, including: Based on the multi-scale coding module in the speech recognition module, the speech to be recognized is acoustically coded to obtain multi-scale acoustic features including shallow hidden layer output and deep hidden layer output. Based on the decoding module in the speech recognition module, the acoustic features are semantically decoded, and the hidden layer features generated in the semantic decoding are sampled to obtain the recognition result sampling features.

7. The speech recognition method according to any one of claims 1 to 5, characterized in that, The error correction module based on the integrated framework applies the acoustic features and the sampling features of the recognition result to perform text error correction, obtaining the speech recognition result of the speech to be recognized, including: Based on the fusion module in the integrated framework, the correlation between the acoustic features and the recognition result sampling features is applied to perform feature fusion on the acoustic features and the recognition result sampling features to obtain fused features; Based on the error correction module, the fusion features are applied to perform text error correction, and the speech recognition result of the speech to be recognized is obtained.

8. A voice recognition device, characterized in that, include: The acquisition unit is used to acquire the speech to be recognized; The recognition unit is used to perform speech recognition on the speech to be recognized based on the speech recognition module in the integrated framework, and obtain the acoustic features of the speech to be recognized and the recognition result sampling features. The error correction unit is used to perform text error correction based on the error correction module in the integrated framework, applying the acoustic features and the recognition result sampling features, to obtain the speech recognition result of the speech to be recognized; The integrated framework is trained based on sample speech and the labeled text of the sample speech; The training steps of the integrated framework include: Based on the initial recognition module in the initialization framework, speech recognition is performed on the sample speech to obtain sample acoustic features, sample recognition result sampling features, and preliminary recognition results; Based on the initial error correction module in the initialization framework, text error correction is performed by applying the acoustic features of the sample and the sampling features of the sample recognition result to obtain the sample recognition result; Based on the preliminary recognition results, the sample recognition results, and the labeled text of the sample speech, a first loss value is determined; Based on the first loss value, the parameters of the initialization framework are iterated to obtain the integrated framework.

9. An electronic device comprising a microphone, a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The microphone is used to collect the speech to be recognized. The processor executes the integrated framework in the computer program to realize the speech recognition of the speech to be recognized. The integrated framework includes a speech recognition module and an error correction module. The speech recognition module is used to acquire the acoustic features and recognition result sampling features of the speech to be recognized. The error correction module is used to apply the acoustic features and the recognition result sampling features to perform text error correction in order to obtain the speech recognition result of the speech to be recognized. The integrated framework is trained based on sample speech and the labeled text of the sample speech; The training steps of the integrated framework include: Based on the initial recognition module in the initialization framework, speech recognition is performed on the sample speech to obtain sample acoustic features, sample recognition result sampling features, and preliminary recognition results; Based on the initial error correction module in the initialization framework, text error correction is performed by applying the acoustic features of the sample and the sampling features of the sample recognition result to obtain the sample recognition result; Based on the preliminary recognition results, the sample recognition results, and the labeled text of the sample speech, a first loss value is determined; Based on the first loss value, the parameters of the initialization framework are iterated to obtain the integrated framework.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the speech recognition method as described in any one of claims 1 to 7.

Images