A speech recognition method, apparatus, electronic device, and storage medium

Abstract

The application relates to a speech recognition method and device, electronic equipment and a storage medium. The method comprises the following steps: obtaining a speech signal to be recognized and a speech recognition model, wherein the speech recognition model comprises a shared encoder, a first decoder and a second decoder; based on the speech signal, the shared encoder is used to determine an acoustic representation sequence corresponding to the speech signal; based on the acoustic representation sequence, the first decoder is used to determine a candidate text sequence set corresponding to the speech signal, and the second decoder is used to determine a target dialect identifier corresponding to the speech signal, wherein each candidate text sequence in the candidate text sequence set does not comprise a dialect identifier; the target dialect identifier is spliced with each candidate text sequence to obtain a target text sequence; and the second decoder is used to process each target text sequence to determine a speech recognition result corresponding to the speech signal based on a processing result. The application reduces the engineering deployment and operation and maintenance costs.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, specifically to a speech recognition method, device, electronic device, and storage medium. Background Technology

[0002] With the widespread application of speech recognition technology in fields such as in-vehicle interaction, the demand for multi-dialect speech recognition is becoming increasingly urgent. Currently, multi-dialect speech recognition methods mainly employ an architecture combining a shared encoder with independent dialect classification branches. This involves extracting the acoustic identifier sequence of the speech using a shared encoder, followed by parallel processing by separate speech recognition and dialect classification branches, each completing its own text recognition and dialect classification tasks. However, this approach requires deploying an additional independent dialect classification model outside the speech recognition model. This not only increases system complexity and computational resource consumption but also necessitates separate processing of the two independent models during training, maintenance, and updates, increasing engineering deployment and operational costs. Summary of the Invention

[0003] One objective of this application is to provide a speech recognition method that avoids the problems of high system complexity and large consumption of computing resources caused by independently deploying dialect classification models, thereby reducing engineering deployment and maintenance costs; another objective of this application is to provide a speech recognition device; a third objective of this application is to provide an electronic device; and a fourth objective of this application is to provide a storage medium.

[0004] To achieve the above objectives, in a first aspect, this application provides a speech recognition method, comprising: Acquire the speech signal to be recognized and the speech recognition model, wherein the speech recognition model includes a shared encoder, a first decoder and a second decoder; Based on the speech signal, the acoustic representation sequence corresponding to the speech signal is determined by the shared encoder; Based on the acoustic representation sequence, a set of candidate text sequences corresponding to the speech signal is determined by the first decoder, and a target dialect identifier corresponding to the speech signal is determined by the second decoder. Each candidate text sequence in the candidate text sequence set does not include a dialect identifier. The target dialect identifier is concatenated with each of the candidate text sequences to obtain each target text sequence; The second decoder processes each of the target text sequences to determine the speech recognition result corresponding to the speech signal from each of the target text sequences based on the processing result.

[0005] To achieve the above objectives, in a second aspect, this application provides an information display device, comprising: The acquisition module is used to acquire the speech signal to be recognized and the speech recognition model, wherein the speech recognition model includes a shared encoder, a first decoder and a second decoder; The determining module is used to determine the acoustic representation sequence corresponding to the speech signal based on the speech signal and through the shared encoder; The determining module is further configured to determine, based on the acoustic representation sequence, a set of candidate text sequences corresponding to the speech signal by the first decoder, and a target dialect identifier corresponding to the speech signal by the second decoder, wherein each candidate text sequence in the candidate text sequence set does not include a dialect identifier; The concatenation module is used to concatenate the target dialect identifier with each of the candidate text sequences to obtain each target text sequence; The recognition module is used to process each of the target text sequences through the second decoder, so as to determine the speech recognition result corresponding to the speech signal from each of the target text sequences based on the processing result.

[0006] To achieve the above objectives, in a third aspect, this application also provides an electronic device, including: a processor and a memory, wherein the processor is configured to execute a speech recognition program stored in the memory to implement the speech recognition method as described above.

[0007] To achieve the above objectives, in a fourth aspect, this application also provides a storage medium storing one or more programs that can be executed by one or more processors to implement the speech recognition method described above.

[0008] The beneficial effects of this application are as follows: This application provides a speech recognition method, which includes acquiring a speech signal to be recognized and a speech recognition model, wherein the speech recognition model includes a shared encoder, a first decoder, and a second decoder; based on the speech signal, determining the acoustic representation sequence corresponding to the speech signal through the shared encoder; based on the acoustic representation sequence, determining a set of candidate text sequences corresponding to the speech signal through the first decoder, and determining a target dialect identifier corresponding to the speech signal through the second decoder, wherein each candidate text sequence in the candidate text sequence set does not include the dialect identifier; concatenating the target dialect identifier with each candidate text sequence to obtain each target text sequence; and processing each target text sequence through the second decoder to determine the speech recognition result corresponding to the speech signal from each target text sequence based on the processing result. In this application, after extracting the acoustic representation sequence of the speech signal to be recognized by a shared encoder, a first decoder is used to generate candidate text sequences that do not include dialect identifiers, and a second decoder is used to determine the target dialect identifier. The target dialect identifier is then concatenated with each candidate text sequence, and the second decoder processes each concatenated target text sequence to determine the final speech recognition result. This integrates the dialect classification function into the second decoder within the speech recognition model, eliminating the need to deploy a separate dialect classification model outside the speech recognition model. This avoids the problems of high system complexity and high computational resource consumption caused by deploying a separate dialect classification model, and reduces engineering deployment and maintenance costs. Attached Figure Description

[0009] Figure 1 This diagram illustrates a flow chart of a speech recognition method provided in an embodiment of this application. Figure 2 This diagram illustrates a flowchart of a training method for a speech recognition model provided in an embodiment of this application. Figure 3 This diagram illustrates a flow chart of a speech recognition method provided in an embodiment of this application. Figure 4 This diagram illustrates a flowchart of a training method for another speech recognition model provided in an embodiment of this application. Figure 5 This diagram illustrates a flowchart of an inference method for a speech recognition model provided in an embodiment of this application. Figure 6 This diagram illustrates the structure of a speech recognition device provided in an embodiment of this application. Figure 7 This diagram illustrates the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0010] The embodiments of this application will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be understood that the preferred embodiments are only for illustrating this application and are not intended to limit the scope of protection of this application.

[0011] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0012] To facilitate understanding of the embodiments of this application, the following will provide further explanation and description with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of this application.

[0013] refer to Figure 1 , Figure 1 This is a flowchart illustrating a speech recognition method provided in an embodiment of this application. The speech recognition method provided in this embodiment specifically includes the following steps: S101: Obtain the speech signal to be recognized and the speech recognition model.

[0014] In this embodiment, the speech recognition model refers to a pre-trained speech recognition model. The speech recognition model adopts the end-to-end model structure in the WeNet-E2E framework, including a shared encoder, a first decoder, and a second decoder.

[0015] The shared encoder is the foundation for speech feature extraction in speech recognition models. It typically employs a Conformer or Transformer structure to extract high-dimensional acoustic representation sequences from speech signals. Its input is 80-dimensional logarithmic Mel filter bank features (FBank features), extracted from the speech signal through a sliding window. The speech features are first downsampled temporally through convolutional subsampling layers, then enter a multi-layer Conformer module. Each module includes a feedforward network, a multi-head self-attention mechanism, a convolutional module, and a layer normalization structure. The final output is a high-dimensional acoustic representation sequence containing comprehensive information such as speech content and dialect attributes, which is used by both the first and second decoders.

[0016] The first decoder is actually a CTC decoder. The CTC decoder solves the alignment problem when the number of speech signal frames far exceeds the number of text characters by automatically establishing temporal alignment relationships between speech acoustic frames and text characters. In this embodiment, the first decoder is used to generate a set of candidate text sequences that do not include dialect tokens. Since dialect tokens do not correspond to specific acoustic pronunciations and cannot participate in the temporal alignment process of the CTC decoder, the first decoder does not involve dialect tokens in the training and inference processes of the speech recognition model.

[0017] The second decoder is actually an attention decoder. Attention decoders can sift through a large amount of information to extract key content and capture content-based global dependencies. In this embodiment, the second decoder plays a dual role: first, predicting dialect representations from the acoustic representation sequence output by the shared encoder; and second, re-scoring the complete sequence formed by concatenating the dialect representation and candidate text sequences to obtain the final speech recognition result. This capability stems from the design of the training phase, where dialect identifiers are inserted at predetermined positions in the training text sequences corresponding to the speech training samples. This forces the attention decoder to predict dialect identifiers at preset positions and generate text content at other positions during the autoregressive sequence generation process, thereby learning both dialect classification and text generation capabilities simultaneously during the unified decoding process.

[0018] Specifically, the system receives the user-input speech signal to be recognized and loads a pre-trained speech recognition model. During the training phase, this speech recognition model has learned how to work collaboratively with the shared encoder, the first decoder, and the second decoder through a large number of multi-dialect speech training samples labeled with dialect identifiers.

[0019] More specifically, reference Figure 2 and Figure 4 As shown, the speech recognition model in this embodiment is trained in the following way: S201: Obtain the speech training sample set and the training text sequence corresponding to each speech training sample in the speech training sample set.

[0020] S202: Input each speech training sample in the speech training sample set into the shared encoder to obtain the training acoustic representation sequence corresponding to each speech training sample.

[0021] S203: For each speech training sample, based on the training acoustic representation sequence corresponding to the speech training sample, train the first decoder with the first text sequence as the target to obtain the first loss corresponding to the speech training sample, and perform autoregressive training on the second decoder with the training text sequence corresponding to the speech training sample as the target to obtain the second loss corresponding to the speech training sample.

[0022] S204: Based on the first loss and second loss corresponding to each speech training sample, update the parameters of the shared encoder, the first decoder and the second decoder to obtain the speech recognition model.

[0023] Regarding steps S201 to S204 above, the speech training sample set can be understood as a multi-dialect speech data set used to train the speech recognition model. This speech training sample set contains speech training samples from various dialect categories, such as Sichuan dialect, Cantonese, Mandarin, and English. Each speech training sample is labeled with a corresponding text sequence and dialect representation. The training text sequence is the sequence obtained after expanding the labeled text sequence corresponding to the speech training sample. Specifically, a corresponding dialect identifier is inserted at a preset position in the labeled text, so that the training text sequence simultaneously contains dialect information and text content. The preset position can be the inspiration position of the text sequence. For example, the labeled text sequence of a Cantonese speech is "I want to watch Transformers 5: The Last Knight". After inserting the preset dialect identifier, the training text sequence becomes... <cantonese>I want to watch Transformers: The Last Knight.

[0024] The preset dialect identifiers correspond to special symbols for each dialect category. The preset dialect identifiers include... <sichuan>(Sichuan dialect) <mandarin>(mandarin), <cantonese>(Cantonese) and <english>(English), etc. In the model vocabulary, each dialect identifier is assigned a unique Token ID, for example... <sichuan>Corresponding to 8156, <mandarin>Corresponding to 8157, <cantonese>Corresponding to 8158, <english>This corresponds to 8159. Simultaneously, these dialect identifiers are added to the non-language symbol file (non_lang_syms.txt), where regular expression rules enable the tokenizer to recognize them as complete symbols, preventing them from being split into sub-word units.

[0025] The first text sequence is obtained by removing preset dialect identifiers from the training text sequence. It exists as a labeled text sequence and does not contain any dialect identifier information. For example, the training text sequence is... <cantonese>In Manchester United, the first text sequence obtained after removing the preset dialect identifier is "In Manchester United". The first text sequence is specifically used as the training target for the first decoder.

[0026] The first loss can be understood as the recognition loss generated by the first decoder during training, which is used to measure the speech recognition effect of the first decoder. The second loss can be accumulated as the loss generated by the second decoder during training, which is used to measure the comprehensive effects of the two tasks of dialect classification and text generation of the second decoder at the same time.

[0027] Among them, a speech training sample set of multiple dialect categories is obtained, and each speech training sample is labeled with the original text sequence and the corresponding dialect category. For each speech training sample, a preset dialect identifier corresponding to the dialect category is inserted at a preset position in its original text sequence to obtain a training text sequence. For example, inserting a preset dialect identifier at the starting position in the original text sequence: the original text sequence of the Cantonese speech training sample "I want to watch Transformers: The Last Knight" is processed as <cantonese>I want to watch Transformers 5: The Last Knight; the original text sequence of the Mandarin speech training samples was processed into... (The sentence fragment "even showed a situation where transactions almost stopped" appears to be incomplete and lacks context.) <mandarin>There were even instances where transactions almost came to a standstill; the original text sequences of the English speech training samples were processed into... <english>HAM TO BE OR NOT TO BE. Therefore, each training text sequence is processed into a unified format called dialect identifier + text content, providing a foundation for subsequent joint training.

[0028] Each speech training sample in the speech training sample set is input into a shared encoder. The shared encoder extracts acoustic features from each speech training sample, processes them through a multi-layer neural network, and outputs a corresponding training acoustic representation sequence. This acoustic representation sequence is a high-dimensional abstraction of the speech signal, containing both speech content information and dialect features, and is used by both the first and second decoders.

[0029] For each speech training sample, the first decoder (CTCDecoder) and the second decoder (Attention Decoder) are trained based on their corresponding training acoustic representation sequence.

[0030] The training of the first decoder targets the first text sequence corresponding to the speech training samples. During training, a slicing operation is performed on the text tensor corresponding to the training text sequence, removing the units corresponding to dialect identifiers at preset positions and retaining only the parts corresponding to the actual text characters. Simultaneously, the text length parameter is reduced accordingly to obtain the first text sequence. For example, for a training text sequence... <cantonese>In the case of Manchester United, the original text length was 5. After removing the dialect identifier at the starting position, the first text sequence (e.g., text[:, 1:]) becomes "In Manchester United", and the text length is updated to 4 (text_lengths - 1). The first decoder decodes based on the input sequence of acoustic representations, and its output (text) is compared with the first text sequence to calculate the first loss corresponding to the speech training sample. Since the training target is the pure text sequence after removing the dialect identifier, the dialect identifier does not participate in the alignment and loss calculation of the first decoder, ensuring that the first decoder specifically learns the mapping relationship between speech frames and real pronunciation characters without being interfered by the dialect identifier.

[0031] For the training of the second decoder, autoregressive training is carried out with the training text sequence corresponding to the speech training sample (e.g., the text content is text and the text length is text_lengths) as the target. The second decoder gradually generates a predicted text sequence in an autoregressive manner based on the input sequence of acoustic representations. Since the training text sequence contains a dialect identifier at a preset position ( <tag>The remaining positions are text content. During training, the second decoder is guided to predict dialect identifiers at preset positions and text content at the remaining positions. The predicted text sequence (…) <tag>The second decoder compares the given text with the training text sequence to calculate the second loss corresponding to the speech sample. Through this training, the second decoder simultaneously masters both dialect classification and text generation capabilities in a unified sequence generation process.

[0032] After obtaining the first and second losses for each speech training sample, the parameters of the shared encoder, first decoder, and second decoder are jointly updated based on each first and second loss. By iteratively updating the parameters across the entire speech training sample set until the speech recognition model converges, a speech recognition model with both speech recognition and dialect classification capabilities is finally obtained.

[0033] In this embodiment, by inserting preset dialect identifiers at preset positions in the training text sequences corresponding to each speech training sample, the first decoder is trained using a first text sequence with the preset dialect identifiers removed, and the second decoder is trained using an autoregressive method with a complete training text sequence containing the preset dialect identifiers. Finally, the losses of the first and second decoders are used together to update the parameters of the shared encoder, the first decoder, and the second decoder. This allows the speech recognition model to learn speech recognition and dialect classification capabilities simultaneously within a unified end-to-end framework without the need for an additional independent dialect classification model. This effectively simplifies the system architecture and reduces computational resource consumption and engineering deployment costs.

[0034] In step S203 above, the first decoder is trained using the first text sequence as the target to obtain the first loss corresponding to the speech training samples, including: The first predicted text sequence is obtained by decoding the training acoustic representation sequence corresponding to the speech training sample using the first decoder. Based on the first predicted text sequence and the first text sequence, determine the first loss corresponding to the speech training sample.

[0035] Here, the first predicted text sequence can be understood as the prediction result output by the first decoder after decoding the input training acoustic representation sequence during the training process. Since the training objective of the first decoder is a first text sequence without dialect identifiers, the first predicted text sequence also does not include any dialect identifiers, and only contains pure text content predictions.

[0036] Specifically, for each speech training sample, its corresponding training acoustic representation sequence is input into the first decoder. The first decoder decodes the acoustic representation sequence and outputs the first predicted text sequence. For example, for a Cantonese speech sample, its training text sequence is... <cantonese>In the case of Manchester United, after removing the dialect identifier at the starting position, the first text sequence is "In Manchester United". The first decoder decodes based on the acoustic representation sequence of this sample and outputs the first predicted text sequence, such as "In Manchester United". Then, the first predicted text sequence is compared with the first text sequence, and the difference between the two is calculated to obtain the first loss corresponding to this speech training sample.

[0037] In the above step S203, the second decoder is autoregressively trained with the training text sequence corresponding to the speech training sample as the target to obtain the second loss corresponding to the speech training sample, including: The second decoder autoregressively decodes the training acoustic representation sequence corresponding to the speech training sample to obtain the second predicted text sequence; Based on the second predicted text sequence and the training text sequence corresponding to the speech training sample, the second loss corresponding to the speech training sample is determined.

[0038] Among them, the second predicted text sequence can be understood as the prediction result output by the second decoder after autoregressive decoding according to the input training acoustic representation sequence during the training process. There is a predicted dialect identifier at the preset position in the second predicted text sequence, and predicted text at the remaining positions except the preset position.

[0039] Autoregressive decoding can be understood as that each step of the second decoder's prediction not only depends on the input acoustic representation sequence but also on the previously generated partial sequence. This step-by-step generation method enables the second decoder to generate a dialect identifier at the preset position and text content at the remaining positions in turn according to the structure of the training text sequence, thus completing two different types of tasks in a unified sequence.

[0040] Specifically, for the same speech training sample, its training acoustic representation sequence is input into the second decoder. The second decoder decodes the acoustic representation sequence in an autoregressive manner and gradually generates the second predicted text sequence. Since the training target of the second decoder is the complete training text sequence (including the dialect identifier), it is guided to output the predicted dialect identifier at the preset position and the predicted text content at the remaining positions during the decoding process. The second predicted text sequence is compared with the training text sequence, and the difference between the two is calculated, including both the prediction error of the dialect identifier part and the prediction error of the text content part, so as to obtain the second loss corresponding to this speech training sample.

[0041] In this embodiment, by setting different training objectives and output structures for the first decoder and the second decoder, the first decoder focuses on the task of recognizing pure text without dialect identifiers, while the second decoder simultaneously predicts the dialect identifiers at preset locations and generates the text content at other locations. Finally, the calculation of their respective losses drives the two decoders to master the corresponding capabilities, so that the entire speech recognition model can achieve speech recognition and dialect classification functions simultaneously in an end-to-end framework without the need for an additional independent dialect classifier, effectively reducing the complexity of the system architecture and the cost of model maintenance.

[0042] The above-mentioned step S204 specifically includes: For each speech training sample, the first loss and the second loss corresponding to the speech training sample are weighted and summed to obtain the joint loss corresponding to the speech training sample. The parameters of the shared encoder, the first decoder, and the second decoder are updated based on the joint loss corresponding to each speech training sample. When the joint loss is less than a preset loss threshold, the parameters of the shared encoder, the first decoder, and the second decoder are stopped from being updated to obtain the speech recognition model.

[0043] The preset loss threshold can be understood as a pre-defined condition used to determine whether the model has converged during training. When the joint loss decreases to less than the preset loss threshold, it indicates that the difference between the model's prediction and the true label is small enough, and the model has stable speech recognition and dialect classification capabilities. At this point, training can be stopped.

[0044] Specifically, after obtaining the first loss and the second loss corresponding to each speech training sample, the first loss and the second loss corresponding to each speech training sample are weighted and summed to obtain the joint loss corresponding to that speech training sample. The formula for the weighted summation can be expressed as: Loss = λ × Loss1 + (1-λ) × Loss2, where Loss1 is the first loss, Loss2 is the second loss, and λ is the weight parameter.

[0045] In actual training, batch processing is performed. After obtaining the joint loss for each speech training sample, the joint losses for each speech training sample within a batch are aggregated (e.g., summed or averaged) to obtain the total batch loss for that batch. Based on the total batch loss, the parameters of the shared encoder, the first decoder, and the second decoder are updated all at once using the backpropagation algorithm. Since the shared encoder is connected to both the first and second decoders, its parameter gradients during backpropagation originate from both the first and second losses. This allows the acoustic representation learned by the shared encoder to simultaneously serve both text recognition and dialect classification tasks, achieving true joint optimization.

[0046] The above steps are iteratively executed on the entire speech training sample set, with parameter updates performed after each batch. As training progresses, the joint loss gradually decreases. When the joint loss is less than a preset loss threshold, it indicates that the overall error of the model has been reduced to an acceptable level, and the model has achieved stable speech recognition and dialect classification capabilities. At this point, updates are stopped, and the final speech recognition model is obtained.

[0047] In this embodiment, the plain text recognition loss of the first decoder is weighted and fused with the dialect classification and text generation loss of the second decoder to obtain a unified joint loss. Based on this joint loss, the joint parameters of the shared encoder, the first decoder, and the second decoder are updated. Training stops when the joint loss meets the preset convergence condition, so that the acoustic representation learned by the shared encoder can serve the tasks of the two decoders simultaneously. Without adding an independent dialect classification model, efficient collaborative training of speech recognition and dialect classification is achieved in a unified end-to-end framework.

[0048] S102: Based on the speech signal, determine the acoustic representation sequence corresponding to the speech signal through a shared encoder.

[0049] In this embodiment, after acquiring the speech signal, the speech is segmented into a continuous frame sequence using a sliding window, and feature calculation is performed on each frame to extract 80-dimensional FBank features. Subsequently, the extracted FBank features are input to the shared encoder. Upon receiving the FBank features, the shared encoder first performs temporal downsampling through a convolutional subsampling layer to reduce the sequence length and extract local features. The downsampled features then enter a multi-layer Conformer module, where each module sequentially passes through a feedforward network, a multi-head self-attention mechanism, a convolutional module, and layer normalization processing. The shared encoder ultimately outputs a high-dimensional acoustic representation sequence, which simultaneously contains the speech content and dialect features, for use by the subsequent first and second decoders.

[0050] S103: Based on the acoustic representation sequence, the candidate text sequence set corresponding to the speech signal is determined by the first decoder, and the target dialect identifier corresponding to the speech signal is determined by the second decoder.

[0051] In this embodiment, the candidate text sequence set includes multiple candidate text sequences, and each candidate text sequence does not include dialect identifiers. That is, the candidate text sequences are alternative text sequences generated by the first decoder through CTC prefix beam search.

[0052] The target dialect is identified by the dialect category symbol predicted by the second decoder.

[0053] Specifically, the first decoder takes the acoustic representation sequence as input and performs CTC prefix beam search. The first decoder predicts character probabilities on each frame and sums them over all possible alignment paths to obtain the overall probability of the sequence. The beam search algorithm retains several paths with the highest probabilities during the decoding process and finally generates a set of candidate text sequences. Since the dialect identifier does not correspond to the actual pronunciation, the dialect identifier is not involved in the training and inference of the first decoder, so all the generated candidate text sequences are plain texts without dialect identifiers. For example, for a Cantonese speech, the first decoder may generate candidate sequences such as "我要看变形金刚五”、"我要看变形金刚五” etc.

[0054] The second decoder also takes the acoustic representation sequence as input. Since in the training stage, the second decoder is trained autoregressively with the complete text sequence containing the dialect identifier as the target, and is trained to predict the dialect identifier at the preset position and generate text content at the remaining positions, so during inference, the second decoder performs autoregressive decoding based on the acoustic representation sequence and extracts the output at the preset position from the decoded output sequence, which is the predicted target dialect identifier, such as <sichuan>Once the target dialect identifier is determined, this stage of the task is complete.

[0055] It should be noted that before determining the target dialect identifier for the speech signal using the second decoder, the target application scenario for the speech signal is determined. If the target application scenario is a preset application scenario, the target dialect identifier for the speech signal is determined using the second decoder. If the target application scenario is not a preset application scenario, the candidate text sequence with the highest probability is selected from the candidate text sequence set as the final speech recognition result. The target application scenario refers to the specific usage scenario of the current speech signal, which can be determined by the currently opened application, etc. The preset application scenario can be understood as a pre-defined application scenario that requires the output of a dialect identifier. By introducing application scenario judgment before determining the target dialect identifier, the speech recognition model can call the dialect classification function as needed, outputting a complete result with a dialect identifier in scenarios requiring dialect information, and directly outputting a plain text result in scenarios not requiring dialect information, thus saving computational resources and inference latency.

[0056] S104: Concatenate the target dialect identifier with each candidate text sequence to obtain each target text sequence.

[0057] In this embodiment, the single target dialect identifier predicted by the second decoder is concatenated with each candidate text sequence generated by the first decoder. The dialect identifier is located at a pre-agreed fixed position (usually the start position of the sequence for ease of operation). For example, if the predicted target dialect identifier is... <cantonese>If the candidate text sequence is "I want to watch Transformers 5", then concatenating them will yield the target text sequence. <cantonese>I want to watch Transformers 5; if the candidate text sequence is "I want to watch Transformers 5", then the concatenation will yield... <cantonese>I want to watch Transformers 5. By performing this operation on all candidate text sequences, individual target text sequences containing dialect information are constructed.

[0058] S105: The second decoder processes each target text sequence to determine the speech recognition result corresponding to the speech signal from each target text sequence based on the processing result.

[0059] In this embodiment, the concatenated target text sequences are input into a second decoder, which processes each sequence. Since the second decoder has learned the statistical association between dialect identifiers and text content during training, it can effectively evaluate the overall matching degree between each target text sequence and the input speech. Based on the processing results of the second decoder, the final speech recognition result is determined from each target text sequence. For example, the final output result is... <cantonese>I want to watch Transformers 5. The result includes both the complete recognized text and the explicit labeling of the dialect category of the speech, achieving an integrated presentation of dialect attributes and recognized text in the same sequence.

[0060] In the above process, after obtaining the speech recognition result, the target dialect identifier is output as structured information and passed to subsequent modules to achieve automatic confirmation and continuous utilization of the user's dialect. At the same time, in conjunction with the speech synthesis module, the speech synthesis strategy of the corresponding dialect is adaptively selected according to the target dialect identifier, thereby achieving consistent matching between the user's input dialect and the system's output dialect. This solves the problem of dialect inconsistency during voice interaction and improves the personalized service capability and the naturalness and consistency of interaction in multi-dialect scenarios at the business level.

[0061] This embodiment provides a speech recognition method that extracts the acoustic representation sequence of the speech signal to be recognized by a shared encoder, generates candidate text sequences without dialect identifiers using a first decoder, determines the target dialect identifier using a second decoder, concatenates the target dialect identifier with each candidate text sequence, and then processes the concatenated target text sequences using the second decoder to determine the final speech recognition result. This integrates the dialect classification function into the second decoder within the speech recognition model, eliminating the need to deploy a separate dialect classification model outside the speech recognition model. This avoids the high system complexity and computational resource consumption caused by independently deploying a dialect classification model, and reduces engineering deployment and maintenance costs.

[0062] refer to Figure 3 , Figure 3 This is a flowchart illustrating another speech recognition method provided in an embodiment of this application. The speech recognition method provided in this application specifically includes the following steps: S301: Obtain the speech signal to be recognized and the speech recognition model.

[0063] S302: Based on the speech signal, determine the acoustic representation sequence corresponding to the speech signal through a shared encoder.

[0064] Regarding steps S301 and S302, step S301 is the same as step S101, and step S302 is the same as step S102. For details, please refer to the description of steps S101 and S102 above. This embodiment will not repeat the description here.

[0065] S303: Based on the acoustic representation sequence, a candidate text sequence set corresponding to the speech signal is determined by a first decoder, and a target dialect identifier corresponding to the speech signal is determined by a second decoder.

[0066] In this embodiment, the method for determining the candidate text sequence set in step S303 is consistent with that in step S103 described above, and can be found in step S103 for details. Step S303, based on the acoustic representation sequence, determines the target dialect identifier corresponding to the speech signal through a second decoder, specifically including: Based on the acoustic representation sequence, the acoustic representation sequence is decoded by a second decoder to obtain the decoding result; Extract the dialect identifier corresponding to the preset location from the decoding results; The dialect identifier corresponding to the preset location is determined as the target dialect identifier corresponding to the speech signal.

[0067] The decoding result can be understood as the complete sequence output by the second decoder after performing autoregressive decoding based on the input acoustic representation sequence. Since the second decoder is trained during the training phase to predict dialect identifiers at preset positions and generate text content at the remaining positions, during inference, the preset positions in its decoding result correspond to the dialect identifiers, and the remaining positions correspond to the text content. For example, if the preset position is the starting position, the decoding result might be: <sichuan>There are also examples like leapfrog. In other words, the preset position is used to represent the location of the preset dialect identifier in the training text sequence corresponding to the speech training sample. The speech training sample and the training text sequence are used to train the speech recognition model.

[0068] Specifically, the acoustic representation sequence output by the shared encoder is input into the second decoder, which decodes in an autoregressive manner, progressively generating the output sequence to obtain the decoding result. The decoding result includes dialect identifiers output at preset positions.

[0069] After obtaining the decoding result, based on the preset position agreed upon during training, the output corresponding to that preset position is extracted from the decoding result as the predicted dialect identifier (the dialect identifier corresponding to the highest probability). Since the preset position remains consistent during the training and inference phases, the extracted dialect identifier is the second decoder's judgment result on the dialect category of the input speech signal. For example, if the preset position is the starting position, the decoding result is... <sichuan>For games like Leapfrog, extract the content from the starting position. <sichuan>As a marker of the target dialect.

[0070] The dialect identifier extracted from a preset location is determined as the target dialect identifier corresponding to the speech signal, and is used for subsequent concatenation with the candidate text sequence generated by the first decoder. For example, the finally determined target dialect identifier is... <sichuan>This will be used to concatenate with the candidate text sequence "and those leapfrogs" to form the complete target text sequence. <sichuan>And there are things like leapfrog.

[0071] It should be noted that after extracting the dialect identifier corresponding to the preset position from the decoding result, if there is only one dialect identifier, the dialect identifier corresponding to the preset position is determined as the target dialect identifier corresponding to the speech signal; if there are multiple dialect identifiers, based on the prediction probability corresponding to each dialect identifier, all candidate dialect identifiers with prediction probabilities greater than a preset probability threshold are determined from each dialect identifier, and each candidate dialect identifier is determined as the target dialect identifier. Thus, during subsequent concatenation, for each target dialect identifier, the target dialect identifier is concatenated with each candidate text sequence in the candidate text sequence set. This embodiment achieves the determination of subsequent speech recognition results by extracting multiple candidate dialect identifiers output by the second decoder and concatenating them with each candidate text sequence in the candidate text sequence set, solving the problem of uncorrectable errors in single dialect prediction and improving the accuracy and robustness of speech recognition for highly similar dialect pairs.

[0072] In this embodiment, by utilizing the training mechanism of the second decoder to predict dialect identifiers at preset positions, the target dialect identifier is extracted from the preset position of the decoding result during inference. This allows the dialect classification task to be completed entirely by the second decoder in a unified decoding process, without the need to deploy an additional independent dialect classification model, thus achieving the integration of dialect classification and speech recognition.

[0073] S304: Concatenate the target dialect identifier with each candidate text sequence to obtain each target text sequence.

[0074] In this embodiment, step S304 is the same as step S104 described above. For details, please refer to step S104 described above. This embodiment will not repeat the details here.

[0075] S305: The second decoder re-scores each target text sequence to obtain the sequence score corresponding to each target text sequence.

[0076] S306: For each target text sequence, determine the target score corresponding to the target text sequence based on the sequence score corresponding to the target text sequence and the first decoder probability corresponding to the candidate text sequences included in the target text sequence.

[0077] S307: Based on the scores of each target, determine the speech recognition result corresponding to the speech signal from each target text sequence.

[0078] Regarding steps S305 to S307 above, each candidate text sequence in the candidate text sequence set corresponds to a first decoder probability. The first decoder probability can be understood as the original probability score of the candidate text sequence calculated by the first decoder when the candidate text sequence is generated.

[0079] Specifically, during the inference process of the speech recognition model, the first decoder generates a set of candidate text sequences through CTC prefix beam search. The beam search algorithm retains the paths with the highest probabilities during decoding, and each candidate text sequence corresponds to a CTC probability score during generation, which is the probability of the first decoder.

[0080] After concatenating the target dialect identifier with each candidate text sequence to obtain the target text sequence, all target text sequences are re-inputted into the second decoder. The second decoder uses an attention mechanism to re-score each target text sequence. The re-scorening process is as follows: the second decoder calculates the probability of generating the complete target text sequence given the acoustic representation sequence output by the shared encoder, and uses this probability as the sequence score of the target text sequence. Since the second decoder learns the association between the dialect identifier and the text content during training, it can evaluate the degree of matching between the target text sequence and the input speech signal from an overall semantic perspective.

[0081] For each target text sequence, its corresponding sequence score is fused with the first decoder probability corresponding to the candidate text sequences included in the target text sequence to obtain the target score for that sequence. Then, based on each target score, the final speech recognition result is determined from each target text sequence.

[0082] Through the above methods, this embodiment introduces a rescoring mechanism to fuse the acoustic probability of the first decoder with the semantic score of the second decoder to obtain the target score. Based on the target score, the final speech recognition result is determined from each target text sequence, so that the selected target text sequence is highly matched with the input speech in both acoustic alignment and overall semantic dimensions. This improves the accuracy and reliability of speech recognition without adding an independent dialect classification model.

[0083] The above-mentioned step S306 specifically includes: Determine the first preset weight corresponding to the first decoder and the second preset weight corresponding to the second decoder; Using the first preset weight and the second preset weight, the probability of the first decoder and the sequence score are weighted and summed to determine the target score corresponding to the target text sequence.

[0084] The first preset weight can be understood as a weight coefficient assigned to the probability of the first decoder, used to control the proportion of the acoustic judgment of the first decoder in the overall score. The second preset weight can be understood as a weight coefficient assigned to the sequence score, used to control the proportion of the semantic judgment of the second decoder in the overall score.

[0085] Specifically, before weighted fusion, a first preset weight corresponding to the first decoder and a second preset weight corresponding to the second decoder are determined. The first and second preset weights can be flexibly set according to the actual application scenario. The sum of the first and second preset weights is 1.

[0086] For each target text sequence, using the determined first and second preset weights, the first decoder probability and sequence score corresponding to the target text sequence are weighted and summed to obtain the target score of the sequence. The weighted summation formula can be expressed as: S(W)=ɑ×S_ctc(W)+(1-ɑ)×S_att(W), where S(W) represents the target score, ɑ represents the first preset weight, 1-ɑ represents the second preset weight, S_ctc(W) represents the first decoder probability, and S_att(W) represents the sequence score.

[0087] It should be noted that the first and second preset weights can be dynamically updated. After obtaining the target dialect identifier, the second preset weight is updated using the prediction probability corresponding to the target dialect identifier, and the first preset weight is updated using (1 - the updated second preset weight). Then, the first decoder probability and the sequence score are weighted and summed using the updated first and second preset weights to determine the target score corresponding to the target text sequence. Thus, without increasing any additional computation and resource consumption, the inherent defect of fixed weights being unable to adapt to different dialect prediction reliability scenarios is solved, and the recognition accuracy of low-confidence dialects and high-similarity dialects is improved.

[0088] The above S307 steps specifically include: The highest target score is determined from all target scores; The target text sequence corresponding to the highest target score is determined as the speech recognition result corresponding to the speech signal.

[0089] After obtaining the scores for each target, the target scores of all target text sequences are compared, and the highest target score is determined. The target text sequence corresponding to the highest target score is the optimal speech recognition result. For example, if... <cantonese>My target rating for Transformers 5 is 0.87. <cantonese>If my target score for Transformers 5 is 0.45, then I will select the sequence with a target score of 0.87. <cantonese>I want to watch Transformers 5 as the final speech recognition result. This speech recognition result includes both the recognized text and dialect category, achieving integrated output of dialect attributes and text content.

[0090] In this embodiment, the acoustic probability of the first decoder and the semantic score of the second decoder are weighted and summed to obtain the target score of each target text sequence. The target text sequence with the highest target score is selected as the final speech recognition result. This allows the final decision to flexibly balance the influence of acoustic alignment and semantic matching, and further improves the accuracy and reliability of speech recognition results without adding an independent dialect classification model.

[0091] This embodiment provides a speech recognition method that extracts the acoustic representation sequence of the speech signal to be recognized by a shared encoder, generates candidate text sequences without dialect identifiers using a first decoder, determines the target dialect identifier using a second decoder, concatenates the target dialect identifier with each candidate text sequence, and then processes the concatenated target text sequences using the second decoder to determine the final speech recognition result. This integrates the dialect classification function into the second decoder within the speech recognition model, eliminating the need to deploy a separate dialect classification model outside the speech recognition model. This avoids the high system complexity and computational resource consumption caused by independently deploying a dialect classification model, and reduces engineering deployment and maintenance costs.

[0092] The following is an example for reference. Figure 5 Let me explain the entire inference process of speech recognition in detail: The speech signal to be recognized (Input) is input into the shared encoder in the speech recognition model.

[0093] The shared encoder performs high-level feature extraction on the input speech signal and outputs a unified acoustic representation sequence. This acoustic representation sequence is simultaneously input to the first decoder and the second decoder to achieve feature sharing.

[0094] The first decoder is the CTC decoder, which first performs frame-by-frame probability calculation on the acoustic representation sequence to obtain the CTC likelihood, which is the probability distribution of each token in the vocabulary corresponding to each frame of speech.

[0095] The CTC prefix beam search algorithm is used to decode the CTC likelihood and generate multiple candidate text sequences, forming an N-Best candidate text sequence set (N-Best). All candidate text sequences in the candidate text sequence set do not contain any dialect identifiers and only contain the text content corresponding to the actual speech.

[0096] The second decoder is an attention decoder that performs autoregressive decoding based on the acoustic representation sequence output by the shared encoder. Since the model has learned the rule of outputting dialect tokens at preset positions during the training phase, the second decoder will predict the target dialect token at the preset position of the output sequence (such as the beginning position of the sequence).

[0097] The target dialect token predicted by the second decoder is concatenated with each candidate text sequence in the N-Best candidate text sequence set generated by the first decoder to obtain multiple complete target text sequences (dialect token + N-Best) containing dialect tokens and text content.

[0098] All concatenated complete target text sequences are input into the same second decoder for attention rescoring, resulting in a sequence score for each complete target text sequence. The CTC probabilities of each candidate text sequence output from the first decoder are then combined and weighted according to preset weights. The complete target text sequence with the highest score is then selected as the final recognition result (Output).

[0099] refer to Figure 6 , Figure 6 This is a schematic diagram of a speech recognition device provided in an embodiment of this application. The speech recognition device provided in this application includes: an acquisition module 10, a determination module 20, a splicing module 30, and a recognition module 40. The acquisition module 10 is used to acquire a speech signal to be recognized and a speech recognition model, wherein the speech recognition model includes a shared encoder, a first decoder, and a second decoder. The determination module 20 is used to determine the acoustic representation sequence corresponding to the speech signal based on the speech signal using the shared encoder. The determination module 20 is also used to determine a set of candidate text sequences corresponding to the speech signal based on the acoustic representation sequence using the first decoder, and to determine a target dialect identifier corresponding to the speech signal using the second decoder. Each candidate text sequence in the candidate text sequence set does not include a dialect identifier. The splicing module 30 is used to splice the target dialect identifier with each of the candidate text sequences to obtain each target text sequence. The recognition module 40 is used to process each of the target text sequences using the second decoder to determine the speech recognition result corresponding to the speech signal from each of the target text sequences based on the processing result.

[0100] In this embodiment, each candidate text sequence in the candidate text sequence set corresponds to a first decoder probability; the recognition module 40 is further configured to: The second decoder is used to re-score each of the target text sequences to obtain the sequence score corresponding to each of the target text sequences; For each target text sequence, a target score is determined based on the sequence score corresponding to the target text sequence and the first decoder probability corresponding to the candidate text sequences included in the target text sequence; Based on the scores of each target, the speech recognition result corresponding to the speech signal is determined from each target text sequence.

[0101] In this embodiment, the identification module 40 is further configured to: Determine the first preset weight corresponding to the first decoder and the second preset weight corresponding to the second decoder; Using the first preset weight and the second preset weight, the first decoder probability and the sequence score are weighted and summed to determine the target score corresponding to the target text sequence; The highest target score is determined from all the target scores mentioned; The target text sequence corresponding to the highest target score is determined as the speech recognition result corresponding to the speech signal.

[0102] In this embodiment, the determining module 20 is further configured to: Based on the acoustic representation sequence, the second decoder decodes the acoustic representation sequence to obtain the decoding result; The dialect identifier corresponding to the preset position is extracted from the decoding result. The preset position is used to represent the position of the preset dialect identifier in the training text sequence corresponding to the speech training sample. The speech training sample and the training text sequence are used to train the speech recognition model. The dialect identifier corresponding to the preset location is determined as the target dialect identifier corresponding to the speech signal.

[0103] The speech recognition method provided in this embodiment also includes a training module, which is used for: Obtain a speech training sample set and a training text sequence corresponding to each speech training sample in the speech training sample set, wherein a preset dialect identifier corresponding to the speech training sample is set at a preset position in the training text sequence. Each of the speech training samples in the speech training sample set is input into the shared encoder to obtain the training acoustic representation sequence corresponding to each speech training sample; For each of the speech training samples, based on the training acoustic representation sequence corresponding to the speech training sample, the first decoder is trained with the first text sequence as the target to obtain the first loss corresponding to the speech training sample, and the second decoder is trained with the training text sequence corresponding to the speech training sample as the target to obtain the second loss corresponding to the speech training sample. The first text sequence is obtained by removing the preset dialect identifier from the training text sequence. Based on the first loss and the second loss corresponding to each of the speech training samples, the parameters of the shared encoder, the first decoder and the second decoder are updated to obtain the speech recognition model.

[0104] In this embodiment, the training module is also used for: The first decoder decodes the training acoustic representation sequence corresponding to the speech training sample to obtain a first predicted text sequence, which does not include dialect identifiers. Based on the first predicted text sequence and the first text sequence, determine the first loss corresponding to the speech training sample; The second decoder performs autoregressive decoding on the training acoustic representation sequence corresponding to the speech training sample to obtain a second predicted text sequence. The preset position in the second predicted text sequence corresponds to a predicted dialect identifier, and the remaining positions other than the preset position correspond to predicted text. Based on the second predicted text sequence and the training text sequence corresponding to the speech training sample, a second loss corresponding to the speech training sample is determined.

[0105] In this embodiment, the training module is also used for: For each of the speech training samples, the first loss and the second loss corresponding to the speech training sample are weighted and summed to obtain the joint loss corresponding to the speech training sample; Based on the joint loss corresponding to each of the aforementioned speech training samples, the parameters of the shared encoder, the first decoder, and the second decoder are updated; When the joint loss is less than a preset loss threshold, the updating of the parameters of the shared encoder, the first decoder, and the second decoder is stopped to obtain the speech recognition model.

[0106] This embodiment provides a speech recognition device that extracts the acoustic representation sequence of the speech signal to be recognized by a shared encoder, generates candidate text sequences without dialect identifiers using a first decoder, determines the target dialect identifier using a second decoder, concatenates the target dialect identifier with each candidate text sequence, and then processes the concatenated target text sequences using the second decoder to determine the final speech recognition result. This integrates the dialect classification function into the second decoder within the speech recognition model, eliminating the need to deploy a separate dialect classification model outside the speech recognition model. This avoids the high system complexity and computational resource consumption caused by independently deploying a dialect classification model, and reduces engineering deployment and maintenance costs.

[0107] refer to Figure 7 As shown, Figure 7 This is a schematic diagram of an electronic device provided in an embodiment of this application. The electronic device 700 in this embodiment may include: at least one processor 701, a memory 702, at least one network interface 704, and other user interfaces 703. The various components in the electronic device 700 are coupled together via a bus system 705. It is understood that the bus system 705 is used to implement communication between these components. In addition to a data bus, the bus system 705 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are referred to as bus system 705.

[0108] The user interface 703 may include a display, keyboard, or clicking device (e.g., mouse, trackball, touchpad, or touchscreen).

[0109] It is understood that the memory 702 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 702 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0110] In some implementations, memory 702 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 7021 and application program 7022.

[0111] The operating system 7021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 7022 includes various applications, such as a media player and a browser, used to implement various application functions. The program implementing the method of the embodiments of this application can be included in the application program 7022.

[0112] In the embodiments of this application, the processor 701 executes the methods provided in the various method embodiments by calling the program or instructions stored in the memory 702, specifically the program or instructions stored in the application program 7022.

[0113] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 701. Processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 701 or by instructions in the form of software. The processor 701 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 702. Processor 701 reads the information in memory 702 and uses its hardware to complete the above method.

[0114] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0115] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0116] This application also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; it may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; and it may also include combinations of the above types of memory.

[0117] When one or more programs in a storage medium can be executed by one or more processors, the above-described method can be implemented when the storage medium is used in an electronic device. The processor executes the programs stored in the memory to implement the above-described execution method.

[0118] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0119] It should be noted that the terms "one implementation," "example," "exemplary embodiment," and "some embodiments" used in the specification indicate that the described embodiments may include specific features, structures, or characteristics, but not every embodiment necessarily includes that specific feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Moreover, when describing a specific feature, structure, or characteristic in conjunction with embodiments, implementing such a feature, structure, or characteristic in conjunction with other embodiments, whether explicitly described or not, is within the knowledge scope of those skilled in the art.

[0120] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or article that comprises a list of elements, or not only those elements, but also other elements not expressly listed, or elements inherent to such a process, method, or article. Unless otherwise specified, an element defined by the phrase "comprising one…" does not exclude the presence of other identical elements in the process, method, or article that includes said element.

[0121] The above embodiments are merely preferred embodiments provided to fully illustrate this application, and the scope of protection of this application is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on this application are all within the scope of protection of this application.< / cantonese> < / cantonese> < / cantonese> < / sichuan> < / sichuan> < / sichuan> < / sichuan> < / sichuan> < / cantonese> < / cantonese> < / cantonese> < / cantonese> < / sichuan> < / cantonese> < / tag> < / tag> < / cantonese> < / english> < / mandarin> < / cantonese> < / cantonese> < / english> < / cantonese> < / mandarin> < / sichuan> < / english> < / cantonese> < / mandarin> < / sichuan> < / cantonese>

Claims

1. A speech recognition method, characterized in that, include: Acquire the speech signal to be recognized and the speech recognition model, wherein the speech recognition model includes a shared encoder, a first decoder and a second decoder; Based on the speech signal, the acoustic representation sequence corresponding to the speech signal is determined by the shared encoder; Based on the acoustic representation sequence, a set of candidate text sequences corresponding to the speech signal is determined by the first decoder, and a target dialect identifier corresponding to the speech signal is determined by the second decoder. Each candidate text sequence in the candidate text sequence set does not include a dialect identifier. The target dialect identifier is concatenated with each of the candidate text sequences to obtain each target text sequence; The second decoder processes each of the target text sequences to determine the speech recognition result corresponding to the speech signal from each of the target text sequences based on the processing result.

2. The speech recognition method according to claim 1, characterized in that, Each candidate text sequence in the candidate text sequence set corresponds to a first decoder probability; The step of processing each of the target text sequences using the second decoder to determine the speech recognition result corresponding to the speech signal from each of the target text sequences based on the processing result includes: The second decoder is used to re-score each of the target text sequences to obtain the sequence score corresponding to each of the target text sequences; For each target text sequence, a target score is determined based on the sequence score corresponding to the target text sequence and the first decoder probability corresponding to the candidate text sequences included in the target text sequence; Based on the scores of each target, the speech recognition result corresponding to the speech signal is determined from each target text sequence.

3. The speech recognition method according to claim 2, characterized in that, The step of determining the target score corresponding to the target text sequence based on the sequence score corresponding to the target text sequence and the first decoder probability corresponding to the candidate text sequences included in the target text sequence includes: Determine the first preset weight corresponding to the first decoder and the second preset weight corresponding to the second decoder; Using the first preset weight and the second preset weight, the first decoder probability and the sequence score are weighted and summed to determine the target score corresponding to the target text sequence; The step of determining the speech recognition result corresponding to the speech signal from each of the target text sequences based on each of the target scores includes: The highest target score is determined from all the target scores mentioned; The target text sequence corresponding to the highest target score is determined as the speech recognition result corresponding to the speech signal.

4. The speech recognition method according to claim 1, characterized in that, The step of determining the target dialect identifier corresponding to the speech signal through the second decoder based on the acoustic representation sequence includes: Based on the acoustic representation sequence, the second decoder decodes the acoustic representation sequence to obtain the decoding result; The dialect identifier corresponding to the preset position is extracted from the decoding result. The preset position is used to represent the position of the preset dialect identifier in the training text sequence corresponding to the speech training sample. The speech training sample and the training text sequence are used to train the speech recognition model. The dialect identifier corresponding to the preset location is determined as the target dialect identifier corresponding to the speech signal.

5. The speech recognition method according to claim 1, characterized in that, The speech recognition model is trained in the following manner: Obtain a speech training sample set and a training text sequence corresponding to each speech training sample in the speech training sample set, wherein a preset dialect identifier corresponding to the speech training sample is set at a preset position in the training text sequence. Each of the speech training samples in the speech training sample set is input into the shared encoder to obtain the training acoustic representation sequence corresponding to each speech training sample; For each of the speech training samples, based on the training acoustic representation sequence corresponding to the speech training sample, the first decoder is trained with the first text sequence as the target to obtain the first loss corresponding to the speech training sample, and the second decoder is trained with the training text sequence corresponding to the speech training sample as the target to obtain the second loss corresponding to the speech training sample. The first text sequence is obtained by removing the preset dialect identifier from the training text sequence. Based on the first loss and the second loss corresponding to each of the speech training samples, the parameters of the shared encoder, the first decoder and the second decoder are updated to obtain the speech recognition model.

6. The speech recognition method according to claim 5, characterized in that, The step of training the first decoder with the first text sequence as the target to obtain the first loss corresponding to the speech training sample includes: The first decoder decodes the training acoustic representation sequence corresponding to the speech training sample to obtain a first predicted text sequence, which does not include dialect identifiers. Based on the first predicted text sequence and the first text sequence, determine the first loss corresponding to the speech training sample; The step of performing autoregressive training on the second decoder using the training text sequence corresponding to the speech training sample as the target to obtain the second loss corresponding to the speech training sample includes: The second decoder performs autoregressive decoding on the training acoustic representation sequence corresponding to the speech training sample to obtain a second predicted text sequence. The preset position in the second predicted text sequence corresponds to a predicted dialect identifier, and the remaining positions other than the preset position correspond to predicted text. Based on the second predicted text sequence and the training text sequence corresponding to the speech training sample, a second loss corresponding to the speech training sample is determined.

7. The speech recognition method according to claim 5, characterized in that, The step of updating the parameters of the shared encoder, the first decoder, and the second decoder based on the first loss and the second loss corresponding to each of the speech training samples to obtain the speech recognition model includes: For each of the speech training samples, the first loss and the second loss corresponding to the speech training sample are weighted and summed to obtain the joint loss corresponding to the speech training sample; Based on the joint loss corresponding to each of the aforementioned speech training samples, the parameters of the shared encoder, the first decoder, and the second decoder are updated; When the joint loss is less than a preset loss threshold, the updating of the parameters of the shared encoder, the first decoder, and the second decoder is stopped to obtain the speech recognition model.

8. A voice recognition device, characterized in that, include: The acquisition module is used to acquire the speech signal to be recognized and the speech recognition model, wherein the speech recognition model includes a shared encoder, a first decoder and a second decoder; The determining module is used to determine the acoustic representation sequence corresponding to the speech signal based on the speech signal and through the shared encoder; The determining module is further configured to determine, based on the acoustic representation sequence, a set of candidate text sequences corresponding to the speech signal by the first decoder, and a target dialect identifier corresponding to the speech signal by the second decoder, wherein each candidate text sequence in the candidate text sequence set does not include a dialect identifier; The concatenation module is used to concatenate the target dialect identifier with each of the candidate text sequences to obtain each target text sequence; The recognition module is used to process each of the target text sequences through the second decoder, so as to determine the speech recognition result corresponding to the speech signal from each of the target text sequences based on the processing result.

9. An electronic device, characterized in that, include: A processor and a memory, wherein the processor is configured to execute a speech recognition program stored in the memory to implement the speech recognition method according to any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the speech recognition method of any one of claims 1 to 7.

Images