Speech recognition method, speech recognition device, storage medium and electronic device

Abstract

The embodiments of the present disclosure relate to a speech recognition method, a speech recognition device, a storage medium and an electronic device, and relate to the field of artificial intelligence technology. The speech recognition method comprises: extracting speech features corresponding to multiple speech frames of speech data to be recognized; encoding the speech features to obtain encoded latent features of the speech features; decoding the encoded latent features into a syllable sequence; and re-decoding the encoded latent features and the syllable sequence to obtain a text sequence corresponding to the speech data to be recognized. The present disclosure can not only improve the accuracy of speech recognition, but also can first perform hot word syllable recognition from the syllable dimension and then perform hot word recognition from the text dimension during the recognition process, thereby increasing the possibility of including hot words in the final recognition result, and significantly improving the recall rate and accuracy of hot words in the recognition result.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of artificial intelligence technology. More specifically, the embodiments of the present disclosure relate to a speech recognition method, a speech recognition device, a computer-readable storage medium, and an electronic device. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims, and no statement herein is admitted to be prior art by inclusion in this section.

[0003] Voice is the primary means of communication in the human world. Automatic speech recognition (ASR), which uses computers to convert sound signals into text, has long been a key research area in the field of artificial intelligence. Current speech recognition systems are generally limited to modeling and recognizing speech data based on Chinese characters. Summary of the Invention

[0004] In this context, embodiments of the present disclosure are intended to provide a speech recognition method, a speech recognition device, a computer-readable storage medium, and an electronic device.

[0005] According to a first aspect of an embodiment of the present disclosure, a speech recognition method is provided, comprising: extracting speech features corresponding to multiple speech frames of speech data to be recognized; encoding the speech features to obtain encoded latent features of the speech features; decoding the encoded latent features into a syllable sequence; and re-decoding the encoded latent features and the syllable sequence to obtain a text sequence corresponding to the speech data to be recognized.

[0006] In an optional embodiment, decoding the encoded latent features into a syllable sequence includes: inputting the encoded latent features into a pre-trained syllable classifier to generate a syllable probability distribution corresponding to each of the speech frames, the syllable probability distribution being used to characterize multiple predicted syllables corresponding to each of the speech frames and the probability that the speech frame belongs to each of the predicted syllables; selecting candidate syllables from the multiple predicted syllables according to the syllable probability distribution to obtain the syllable sequence.

[0007] In an optional embodiment, the selecting of candidate syllables from the multiple predicted syllables according to the syllable probability distribution to obtain the syllable sequence includes: selecting candidate syllables of the initial frame from the multiple predicted syllables corresponding to the initial frame according to the syllable probability distribution corresponding to the initial frame; the initial frame is the first frame of the speech data to be recognized; combining the candidate syllables of the initial frame with the multiple predicted syllables corresponding to the next frame, and selecting a candidate combination result from the multiple syllable combination results obtained; and combining the candidate combination result with the multiple predicted syllables corresponding to subsequent speech frames in sequence to obtain the syllable sequence.

[0008] In an optional embodiment, the selecting of candidate combination results from the obtained multiple syllable combination results includes: obtaining a score corresponding to each of the syllable combination results, the score being calculated based on the probability of multiple predicted syllables contained in the syllable combination result; and selecting the candidate combination result from the multiple syllable combination results in descending order of the scores.

[0009] In an optional embodiment, the method further includes: matching each of the syllable combination results with a preset syllable library to determine whether the syllable combination result contains a preset syllable sequence; the preset syllable library includes syllable sequences corresponding to multiple hot words, and the hot words are determined based on the frequency of use of vocabulary in the target field; each of the preset syllable sequences corresponds to a preset reward value; if the preset syllable sequence is included, then updating the score corresponding to the syllable combination result according to the preset reward value corresponding to the preset syllable sequence.

[0010] In an optional embodiment, the encoding latent features and the syllable sequence are re-decoded to obtain a text sequence corresponding to the voice data to be recognized, including: inputting the encoding latent features and the syllable sequence into a pre-trained text decoder to obtain text features corresponding to the voice data to be recognized; inputting the text features into a pre-trained text classifier to generate a text probability distribution corresponding to each syllable contained in the syllable sequence; the text probability distribution is used to characterize multiple predicted texts corresponding to each syllable and the probability that the syllable belongs to each predicted text; selecting candidate texts from the multiple predicted texts according to the text probability distribution to obtain a candidate text sequence; and screening the candidate text sequence to obtain a text sequence corresponding to the voice data to be recognized.

[0011] In an optional embodiment, the selecting candidate text from the multiple predicted texts according to the text probability distribution to obtain a candidate text sequence includes: selecting the candidate text of the starting syllable from the multiple predicted texts corresponding to the starting syllable of the syllable sequence according to the text probability distribution corresponding to the starting syllable; combining the candidate text of the starting syllable with the multiple predicted texts corresponding to the next syllable, and selecting a candidate combination result from the multiple text combination results obtained; and combining the candidate combination result with the multiple predicted texts corresponding to the subsequent syllables in sequence to obtain the candidate text sequence.

[0012] In an optional embodiment, the selecting of candidate combination results from the obtained multiple text combination results includes: obtaining a score corresponding to each of the text combination results, the score being calculated based on the probability of the multiple texts included in the text combination result; and selecting the candidate combination result from the multiple text combination results in descending order of the scores.

[0013] In an optional embodiment, the method further includes: matching each of the text combination results with a preset vocabulary library to determine whether the text combination result contains preset vocabulary; the preset vocabulary library includes a plurality of vocabulary determined based on the frequency of use of vocabulary in the target field, and each of the preset vocabulary corresponds to a preset reward value; if the preset vocabulary is included, updating the score corresponding to the text combination result according to the preset reward value corresponding to the preset vocabulary.

[0014] In an optional embodiment, the screening of the candidate text sequences to obtain the text sequences corresponding to the voice data to be recognized includes: obtaining a first score corresponding to each of the candidate text sequences, and obtaining a second score for the syllable sequence corresponding to each of the candidate text sequences; determining a comprehensive score based on the first score and the second score; and determining the candidate text sequence having the comprehensive score greater than a preset score threshold as the text sequence corresponding to the voice data to be recognized.

[0015] According to a second aspect of the present disclosure, a method for training a speech recognition model is provided, comprising: obtaining a training sample, the training sample comprising sample speech data and a label corresponding to the sample speech data; the label comprising a syllable label and a text label; inputting the training sample into a speech recognition model to be trained to obtain a speech recognition processing result; determining a loss value of the speech recognition model to be trained based on the degree of difference between the speech recognition processing result and the label; and iteratively training the speech recognition model to be trained based on the loss value until the loss value meets a preset convergence condition, thereby obtaining a trained speech recognition model.

[0016] In an optional embodiment, the speech recognition processing result includes a syllable recognition result and a text recognition result; determining the loss value of the speech recognition model to be trained based on the degree of difference between the speech recognition processing result and the label includes: determining a first loss value output by the speech recognition model to be trained based on the degree of difference between the syllable recognition result and the syllable label; determining a second loss value output by the speech recognition model to be trained based on the degree of difference between the text recognition result and the text label; and determining the loss value of the speech recognition model to be trained based on the first loss value and the second loss value.

[0017] According to a third aspect of the present disclosure, a speech recognition device is provided, comprising: a feature extraction module for extracting speech features corresponding to multiple speech frames of speech data to be recognized; an encoding module for encoding the speech features to obtain encoded latent features of the speech features; a syllable decoding module for decoding the encoded latent features into a syllable sequence; and a text decoding module for re-decoding the encoded latent features and the syllable sequence to obtain a text sequence corresponding to the speech data to be recognized.

[0018] In an optional embodiment, the syllable decoding module is configured to: input the encoded latent features into a pre-trained syllable classifier to generate a syllable probability distribution corresponding to each of the speech frames, wherein the syllable probability distribution is used to characterize the multiple predicted syllables corresponding to each of the speech frames and the probability that the speech frame belongs to each of the predicted syllables; select candidate syllables from the multiple predicted syllables according to the syllable probability distribution to obtain the syllable sequence.

[0019] In an optional embodiment, the syllable decoding module is configured to: select candidate syllables of the initial frame from multiple predicted syllables corresponding to the initial frame according to the syllable probability distribution corresponding to the initial frame; the initial frame is the first frame of the speech data to be recognized; combine the candidate syllables of the initial frame with multiple predicted syllables corresponding to the next frame, and select a candidate combination result from the multiple syllable combination results obtained; and combine the candidate combination results with multiple predicted syllables corresponding to subsequent speech frames in turn to obtain the syllable sequence.

[0020] In an optional embodiment, the syllable decoding module is configured to: obtain a score corresponding to each of the syllable combination results, wherein the score is calculated based on the probability of multiple predicted syllables contained in the syllable combination result; and select the candidate combination result from the multiple syllable combination results in descending order of the scores.

[0021] In an optional embodiment, the syllable decoding module is configured to: match each of the syllable combination results with a preset syllable library to determine whether the syllable combination result contains a preset syllable sequence; the preset syllable library includes syllable sequences corresponding to multiple hot words, and the hot words are determined based on the frequency of use of vocabulary in the target field; each of the preset syllable sequences corresponds to a preset reward value; if the preset syllable sequence is included, the score corresponding to the syllable combination result is updated according to the preset reward value corresponding to the preset syllable sequence.

[0022] In an optional embodiment, the text decoding module is configured to: input the encoded latent features and the syllable sequence into a pre-trained text decoder to obtain text features corresponding to the speech data to be recognized; input the text features into a pre-trained text classifier to generate a text probability distribution corresponding to each syllable contained in the syllable sequence; the text probability distribution is used to characterize the multiple predicted texts corresponding to each syllable and the probability that the syllable belongs to each predicted text; select candidate texts from the multiple predicted texts according to the text probability distribution to obtain a candidate text sequence; and screen the candidate text sequence to obtain a text sequence corresponding to the speech data to be recognized.

[0023] In an optional embodiment, the text decoding module is configured to: select a candidate text for the starting syllable from multiple predicted texts corresponding to the starting syllable of the syllable sequence based on the text probability distribution corresponding to the starting syllable; combine the candidate text of the starting syllable with multiple predicted texts corresponding to the next syllable, and select a candidate combination result from the multiple text combination results obtained; and combine the candidate combination result with multiple predicted texts corresponding to subsequent syllables in sequence to obtain the candidate text sequence.

[0024] In an optional embodiment, the text decoding module is configured to: obtain a score corresponding to each of the text combination results, where the score is calculated based on the probability of multiple texts included in the text combination result; and select the candidate combination result from the multiple text combination results in descending order of the scores.

[0025] In an optional embodiment, the text decoding module is configured to: match each of the text combination results with a preset vocabulary library to determine whether the text combination result contains a preset vocabulary; the preset vocabulary library includes a plurality of vocabulary determined based on the frequency of use of vocabulary in the target field, and each of the preset vocabulary corresponds to a preset reward value; if the preset vocabulary is included, then the score corresponding to the text combination result is updated according to the preset reward value corresponding to the preset vocabulary.

[0026] In an optional embodiment, the text decoding module is configured to: obtain a first score corresponding to each of the candidate text sequences, and obtain a second score of the syllable sequence corresponding to each of the candidate text sequences; determine a comprehensive score based on the first score and the second score; and determine the candidate text sequence whose comprehensive score is greater than a preset score threshold as the text sequence corresponding to the speech data to be recognized.

[0027] According to a fourth aspect of the present disclosure, a training device for a speech recognition model is provided, comprising: a sample acquisition module for acquiring training samples, wherein the training samples include sample speech data and labels corresponding to the sample speech data; the labels include syllable labels and text labels; a processing module for inputting the training samples into the speech recognition model to be trained to obtain a speech recognition processing result; a loss determination module for determining the loss value of the speech recognition model to be trained based on the degree of difference between the speech recognition processing result and the label; and a model training module for iteratively training the speech recognition model to be trained based on the loss value until the loss value meets a preset convergence condition, thereby obtaining a trained speech recognition model.

[0028] In an optional embodiment, the speech recognition processing result includes a syllable recognition result and a text recognition result; the loss determination module is configured to: determine a first loss value output by the speech recognition model to be trained based on the degree of difference between the syllable recognition result and the syllable label; determine a second loss value output by the speech recognition model to be trained based on the degree of difference between the text recognition result and the text label; and determine the loss value of the speech recognition model to be trained based on the first loss value and the second loss value.

[0029] According to a fifth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method described in any one of the above aspects when executed by a processor.

[0030] According to a sixth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the method described in any one of the above aspects by executing the executable instructions.

[0031] According to the speech recognition method, speech recognition device, computer-readable storage medium, and electronic device of the embodiment of the present disclosure, on the one hand, speech features corresponding to multiple speech frames of the speech data to be recognized are extracted, the speech features are encoded, and the encoded latent features of the speech features are obtained. The encoded latent features are decoded into a syllable sequence, so that the syllable features of the speech data to be recognized can be obtained before text recognition. On the other hand, the above-mentioned encoded latent features and syllable sequence are re-decoded to obtain a text sequence corresponding to the speech data to be recognized, so that the speech data to be recognized can be re-recognized from two dimensions: syllables and Chinese characters, solving the technical problem of low accuracy caused by only being able to recognize at the single Chinese character level in the related art, thereby improving the accuracy of the recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings, in which several embodiments of the present disclosure are shown by way of example and not limitation, wherein:

[0033] Figure 1 A flow chart of a speech recognition method according to an embodiment of the present disclosure is shown;

[0034] Figure 2 A flowchart of decoding encoded latent features into syllable sequences according to an embodiment of the present disclosure is shown;

[0035] Figure 3 A flow chart of determining a syllable sequence according to an embodiment of the present disclosure is shown;

[0036] Figure 4 A flow chart of selecting candidate combination results from syllable combination results according to an embodiment of the present disclosure is shown;

[0037] Figure 5 A flowchart of updating the corresponding score of syllable combination results according to an embodiment of the present disclosure is shown;

[0038] Figure 6 A flowchart of determining a text sequence according to an embodiment of the present disclosure is shown;

[0039] Figure 7 A flowchart of determining a candidate text sequence according to an embodiment of the present disclosure is shown;

[0040] Figure 8 A flow chart of selecting candidate combination results from text combination results according to an embodiment of the present disclosure is shown;

[0041] Figure 9 A flowchart of updating the corresponding score of text combination results according to an embodiment of the present disclosure is shown;

[0042] Figure 10 A flowchart of determining a text sequence corresponding to speech data to be recognized according to an embodiment of the present disclosure is shown;

[0043] Figure 11 shows an overall flow chart of a speech recognition method according to an embodiment of the present disclosure;

[0044] Figure 12 A flowchart of a method for training a speech recognition model according to an embodiment of the present disclosure is shown;

[0045] Figure 13 A flowchart of determining the loss value of a speech recognition model to be trained according to an embodiment of the present disclosure is shown;

[0046] Figure 14 A schematic diagram of a speech recognition device according to an embodiment of the present disclosure is shown;

[0047] Figure 15 A schematic diagram showing another speech recognition device according to an embodiment of the present disclosure; and

[0048] Figure 16 A schematic diagram of a storage medium according to an embodiment of the present disclosure is shown;

[0049] Figure 17 A structural diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0050] In the drawings, the same or corresponding reference numerals denote the same or corresponding parts. DETAILED DESCRIPTION

[0051] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided solely to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. Rather, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0052] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present disclosure may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software.

[0053] According to an embodiment of the present disclosure, a speech recognition method, a speech recognition device, a computer-readable storage medium, and an electronic device are provided.

[0054] In this document, any number of elements in the drawings is for illustration and not for limitation, and any naming is for distinction only and does not have any limiting meaning.

[0055] The principles and spirit of the present disclosure are described in detail below with reference to several representative embodiments of the present disclosure. SUMMARY OF THE INVENTION

[0057] The inventors have discovered that related speech recognition methods can generally only perform speech recognition at the level of a single Chinese character, and the recognition accuracy is low.

[0058] In view of the above, the basic idea of ​​the present disclosure is to provide a speech recognition method, a speech recognition device, a computer-readable storage medium and an electronic device. On the one hand, the speech features corresponding to multiple speech frames of the speech data to be recognized are extracted, the speech features are encoded, and the encoded latent features of the speech features are obtained. The encoded latent features are decoded into a syllable sequence, so that the syllable features of the speech data to be recognized can be obtained before text recognition. On the other hand, the above-mentioned encoded latent features and syllable sequence are re-decoded to obtain the text sequence corresponding to the speech data to be recognized, and the speech data to be recognized can be re-recognized from two dimensions: syllables and Chinese characters, solving the technical problem of low accuracy caused by the related technology that only recognition can be performed at the single Chinese character level, thereby improving the accuracy of the recognition results.

[0059] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0060] Application Scenario Overview

[0061] It should be noted that the following application scenarios are only provided to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0062] The embodiments of the present disclosure support converting the input voice data to be recognized into a text sequence and outputting it. Specifically, after receiving the voice data to be recognized, feature extraction can be performed on the voice data to be recognized, and the voice features can be converted into coded latent features. Then, the above-mentioned coded latent features can be decoded into a syllable sequence. Furthermore, the coded latent features and the syllable sequence can be re-decoded to output the text sequence corresponding to the above-mentioned voice data to be recognized.

[0063] Exemplary Methods

[0064] An exemplary embodiment of the present disclosure first provides a speech recognition method. Figure 1The flowchart of the speech recognition method according to the embodiment of the present disclosure may include the following steps S110 to S140:

[0065] Step S110 , extracting speech features corresponding to a plurality of speech frames of the speech data to be recognized.

[0066] In this step, the voice data to be recognized is the voice data to be recognized. For example, the voice data to be recognized can be obtained by reading a pre-stored voice signal, collecting a voice signal through a built-in or external microphone of a terminal device, receiving a voice signal transmitted from another device via a network, or obtaining a voice signal using a short-range communication technology such as Bluetooth or infrared. Each of these methods can be set based on actual circumstances and is not specifically limited in this disclosure.

[0067] The above-mentioned voice data to be recognized may include multiple voice frames. The voice frames are several small audio segments obtained after the above-mentioned voice data to be recognized is cut into units of preset time length. For example: taking the voice data to be recognized as 1 minute and the preset time length as 1 second as an example, the voice data to be recognized can be divided into 60 audio segments, each of which is the above-mentioned voice frame.

[0068] The process of extracting speech features is the process of converting speech data into a multidimensional vector that can be recognized and processed by a computer. For example, the speech features corresponding to each speech frame can be extracted by the following methods: discrete wavelet transform, linear prediction coefficients (LPC), perceptual linear prediction (PLP), Mel frequency cepstrum (MFCC), linear prediction cepstrum (LPCC), etc., which can be set according to actual conditions and are not specifically limited in this disclosure.

[0069] For example, the speech information to be recognized includes T speech frames, and the speech feature corresponding to each speech frame is an N-dimensional vector. Then, the speech features corresponding to the above-mentioned multiple speech frames can be expressed as X={x1, x2, x3, ... x T}, where x i is an N-dimensional vector (i=1, 2, 3...T).

[0070] Step S120: Encode the speech features to obtain encoded latent features of the speech features.

[0071] In this step, an autoencoder can be used to encode the above-mentioned speech features to obtain the encoded hidden features of the speech features.

[0072] Exemplarily, assume that the above-mentioned autoencoder totally includes m layers of encoding networks. Then the input layer is the first layer, the output layer is the m-th layer, the output of the (n - 1)-th layer is the input of the n-th layer, and the output of the n-th layer is the input of the (n + 1)-th layer. After inputting the speech features into the above-mentioned autoencoder, following the above-mentioned transfer method and so on, through m layers of transfer, the final encoded hidden features are output from the m-th layer. Exemplarily, the finally output encoded hidden features can be high-dimensional features h x .

[0073] Step S130: Decode the encoded hidden features into a syllable sequence.

[0074] In this step, decoding is a process of restoring the above-mentioned encoded hidden features into syllable information and is the reverse process of encoding.

[0075] A syllable is a speech unit composed of one or several phonemes combined according to certain rules. Chinese syllables are centered around vowels and consist of three parts: initials, finals, and tones; English syllables are centered around vowel phonemes and are combined with one or more consonant phonemes. Chinese usually includes 22 initials and 38 finals, more than 400 non-tonal syllables, and about 1,300 tonal syllables. For example, "yu", "xiao", etc. are non-tonal syllables.

[0076] A syllable sequence is a sequence composed of multiple syllables. Exemplarily, the syllable sequence corresponding to "你" can be "ni3", and the syllable sequence corresponding to "你好" can be "ni3hao3".

[0077] Step S140: Decode the encoded hidden features and the syllable sequence again to obtain the text sequence corresponding to the speech data to be recognized.

[0078] In this step, the re-decoding process is a process of outputting the text sequence according to the encoded hidden features and the syllable sequence. Among them, the text sequence is a sequence composed of multiple texts. Exemplarily, the text sequence corresponding to "ni3hao3" can be "你好".

[0079] Based on the above technical solutions, the present disclosure can re-recognize the speech data to be recognized from two dimensions of syllables and Chinese characters, solve the technical problem of low accuracy caused by only being able to recognize at the single Chinese character level in the related art, and improve the accuracy of the recognition result.

[0080] The following explains the specific implementation manner of "decoding the encoded hidden features into a syllable sequence" in step S130:

[0081] Specifically, refer to Figure 2 , Figure 2 A flowchart of decoding encoded latent features into syllable sequences according to an embodiment of the present disclosure is shown, including steps S201 and S202:

[0082] In step S201, the encoded latent features are input into a pre-trained syllable classifier to generate a syllable probability distribution corresponding to each speech frame.

[0083] In this step, the above-mentioned encoded latent features can be input into a pre-trained syllable classifier, and the syllable probability distribution corresponding to each speech frame can be obtained according to the output of the syllable classifier.

[0084] The syllable classifier is used to predict the predicted syllables corresponding to the above-mentioned speech frames, and the probability (ie, possibility) of each speech frame belonging to each predicted syllable. It should be noted that the sum of the probabilities of each speech frame belonging to each predicted syllable is 100%.

[0085] In step S202, candidate syllables are selected from a plurality of predicted syllables according to the syllable probability distribution to obtain a syllable sequence.

[0086] In step S202, a beam search algorithm can be used to search for multiple predicted syllables corresponding to each speech frame to obtain a syllable sequence. Beam search is a heuristic graph search algorithm. When the solution space of the graph is relatively large, in order to reduce the space and time occupied by the search, some nodes with relatively poor quality are cut off at each step of depth expansion, and some nodes with higher quality are retained. The specific process is: use the breadth-first strategy to establish a search tree at each level of the tree, sort the nodes according to the heuristic cost, and then only leave a predetermined number of nodes (Beam Width-beam width). Only these nodes continue to expand at the next level, and the other nodes are cut off. Therefore, the present disclosure can reduce the memory requirements and increase the speed of generating syllable sequences.

[0087] Specifically, refer to Figure 3 , Figure 3 A flowchart of determining a syllable sequence according to an embodiment of the present disclosure is shown, comprising steps S301 to S303:

[0088] In step S301 , a candidate syllable of the initial frame is selected from a plurality of predicted syllables corresponding to the initial frame according to the syllable probability distribution corresponding to the initial frame.

[0089] In this step, the initial frame is the first frame of the speech data to be recognized. Therefore, for the initial frame, candidate syllables can be selected from multiple predicted syllables based on its syllable probability distribution. For example, two candidate syllables can be selected from the predicted syllables. The number of candidate syllables selected can be set based on actual circumstances and is not specifically limited in this disclosure.

[0090] In step S302, the candidate syllables of the initial frame are combined with a plurality of predicted syllables corresponding to the next frame, and a candidate combination result is selected from the obtained plurality of syllable combination results.

[0091] In this step, after determining the candidate syllables of the initial frame, the candidate syllables can be combined with multiple predicted syllables corresponding to the next frame (i.e., the second frame of the speech data to be processed) to obtain multiple syllable combination results. Figure 4 , Figure 4 A flowchart of selecting candidate combination results from syllable combination results according to an embodiment of the present disclosure is shown, including steps S401 and S402:

[0092] In step S401, the score corresponding to each syllable combination result is obtained.

[0093] In this step, referring to the relevant explanation of the above-mentioned step S302, after combining the candidate syllables of the initial frame with the multiple predicted syllables corresponding to the next frame, for example, taking the candidate syllables of the initial frame as s1 (90%) and s2 (10%), and the multiple predicted syllables corresponding to the second frame as s3 (60%), s4 (30%), and s5 (10%), the obtained syllable combination results are: s1s3, s1s4, s1s5, s2s3, s2s4, and s2s5.

[0094] Therefore, taking the example of directly determining the score corresponding to each syllable combination result based on the probability product of multiple predicted syllables contained in the syllable combination result, the score corresponding to s1s3 is 90%*60%=54%, the score corresponding to s1s4 is 90%*30%=27%, the score corresponding to s1s5 is 90%*10%=9%, the score corresponding to s2s3 is 10%*60%=6%, the score corresponding to s2s4 is 10%*30%=3%, and the score corresponding to s2s5 is 10%*10%=1%.

[0095] It should be noted that different preset weights can be assigned to the multiple predicted syllables contained in the syllable combination results. Then, according to the above preset weights, the probabilities of the multiple predicted syllables contained in the syllable combination results are weighted to obtain the scores corresponding to each syllable combination result. The scores can be set according to the actual situation, and the present disclosure does not make any special restrictions on this.

[0096] In an optional embodiment, after preliminarily determining the scores corresponding to the syllable combination results, it is also possible to detect whether the syllable combination results contain a preset syllable sequence. When it contains the preset syllable sequence, the scores corresponding to the syllable combination results can be updated. Figure 5 , Figure 5 A flowchart of updating the corresponding score of syllable combination results according to an embodiment of the present disclosure is shown, including steps S501 and S502:

[0097] In step S501, each syllable combination result is matched with a preset syllable library to determine whether the syllable combination result contains a preset syllable sequence.

[0098] In this step, the preset syllable library contains syllable sequences corresponding to multiple hot words, wherein the criteria for determining hot words may be: the frequency of use of the vocabulary in the target field. For example, the hot words corresponding to the e-commerce field may be "package", "express delivery", "payment", etc., the hot words corresponding to the education field may be "online class", "teaching", etc., and the hot words corresponding to the medical field may be "blood pressure", "pulse", etc.

[0099] Specifically, taking the e-commerce field as an example, crawler technology can be used to crawl all web pages, databases, etc. related to the e-commerce field to obtain massive information related to the field. Then, data analysis can be performed on the massive information to count the usage frequency of each word involved, and the syllable sequences corresponding to multiple words with a usage frequency greater than the frequency threshold are stored in the above-mentioned preset syllable library. For example, the preset syllable sequence "bao1guo3" corresponding to "package" and the preset syllable sequence "kuai4di4" corresponding to "express delivery" can be stored in the above-mentioned preset syllable library.

[0100] For example, a preset reward value may be pre-assigned to each preset syllable sequence. For example, the preset reward value corresponding to the preset syllable sequence "bao1guo3" may be set to 30%, and the preset reward value corresponding to the preset syllable sequence "kuai4di4" may be set to "20%." These values ​​may be set based on actual conditions, and the present disclosure does not impose any special restrictions on this. The purpose of the preset reward value is that when the recognized syllable combination result includes a preset syllable sequence, the score of the syllable recognition result may be increased according to the preset reward value, thereby increasing the possibility of including hot words in the final recognition result and improving the recall rate and accuracy of the hot words.

[0101] In step S502 , if a preset syllable sequence is included, the score corresponding to the syllable combination result is updated according to the preset reward value corresponding to the preset syllable sequence.

[0102] In this step, if it is detected that the syllable combination result contains the above-mentioned preset syllable sequence, the preset reward value corresponding to the preset syllable sequence can be read from the above-mentioned preset syllable library, and the score corresponding to the syllable combination result can be updated according to the read preset reward value.

[0103] Referring to the relevant explanation of the above step S401, if it is detected that the syllable combination result "s1s4" contains the above-mentioned preset syllable sequence "kuai4di4", the preset reward value corresponding to "kuai4di4" can be read from the preset syllable library as "20%", and thus, the score corresponding to the syllable combination result "s1s4" can be updated to: 27% + 20% = 47%.

[0104] In step S402, candidate combination results are selected from the plurality of syllable combination results in descending order of scores.

[0105] In this step, after obtaining the scores corresponding to the syllable combination results, the scores can be sorted in descending order, and then the syllable combination results whose scores meet the threshold conditions are selected from the sorted sequence as candidate combination results. For example, the syllable combination results with the top two scores can be used as the candidate combination results (for example, "s2s5" and "s1s4"). The number of selected candidate combination results can be set according to actual circumstances, and this disclosure does not impose any special restrictions on this.

[0106] Next reference Figure 3 In step S303, the candidate combination results are sequentially combined with multiple predicted syllables corresponding to subsequent speech frames to obtain a syllable sequence.

[0107] In this step, after obtaining the candidate combination results corresponding to the second frame, the above candidate combination results can be combined with the multiple predicted syllables corresponding to the third frame, and the candidate combination results can be screened out and then combined with the multiple predicted syllables corresponding to the fourth frame... Repeat the above steps until the candidate combination results corresponding to the T-1 frame are combined with the multiple predicted syllables corresponding to the T frame to obtain the above syllable sequence. It can be seen that, on the one hand, by selecting the candidate combination results with higher scores at each intermediate stage and combining them with the predicted syllables corresponding to the subsequent speech frames in turn to generate a speech sequence, the present disclosure can improve the accuracy of the generated syllable sequence. On the other hand, the present disclosure can also reduce the memory requirements of the system and improve the generation speed of the syllable sequence.

[0108] After obtaining the above-mentioned syllable sequence, in an optional embodiment, all obtained syllable sequences and the above-mentioned latent encoding features can be used as input and then re-decoded to obtain a text sequence corresponding to the speech data to be recognized. In another optional embodiment, syllable sequences with scores that meet a scoring threshold and the above-mentioned latent encoding features can be selected from the obtained syllable sequences and then re-decoded to obtain a text sequence corresponding to the speech data to be recognized. This can be set according to actual circumstances and is not specifically limited in this disclosure.

[0109] The following is an explanation of the specific implementation of "re-decoding the encoded latent features and the syllable sequence to obtain a text sequence corresponding to the speech data to be recognized" in step S140:

[0110] refer to Figure 6 , Figure 6 A flowchart of determining a text sequence according to an embodiment of the present disclosure is shown, comprising steps S601 to S604:

[0111] In step S601, the encoded latent features and the syllable sequence are input into a pre-trained text decoder to obtain text features corresponding to the speech data to be recognized.

[0112] In this step, the encoded latent features and the above-mentioned syllable sequence can be input into a pre-trained text decoder to obtain text features corresponding to the speech data to be recognized.

[0113] The function of the text decoder is to convert the encoded latent features and the syllable sequence into text features. The text features can be a vector obtained by fusing the above-mentioned encoded latent features and the syllable sequence.

[0114] In step S602 , the text features are input into a pre-trained text classifier to generate a text probability distribution corresponding to each syllable contained in the syllable sequence.

[0115] In this step, the above text features may be input into a trained text classifier to generate a text probability distribution corresponding to each syllable contained in the syllable sequence.

[0116] Among them, the text classifier is used to classify the above-mentioned text features to predict the predicted text and its probability corresponding to each syllable contained in each syllable sequence, that is, the text probability distribution output by the above-mentioned text classifier is used to characterize the multiple predicted texts corresponding to each syllable and the probability of each syllable belonging to each predicted text. It should be noted that the sum of the probabilities of each syllable belonging to each predicted text is 100%.

[0117] In step S603, candidate texts are selected from multiple predicted texts according to the text probability distribution to obtain a candidate text sequence.

[0118] In step S603 , a beam search algorithm may be used to search multiple predicted texts corresponding to each syllable to obtain a candidate text sequence.

[0119] Specifically, refer to Figure 7 , Figure 7 A flowchart for determining a candidate text sequence according to an embodiment of the present disclosure is shown, comprising steps S701 to S703:

[0120] In step S701 , a candidate text of the initial syllable is selected from a plurality of predicted texts corresponding to the initial syllable according to the text probability distribution corresponding to the initial syllable of the syllable sequence.

[0121] In this step, for the above-mentioned initial syllable, candidate texts can be selected from multiple predicted texts based on their corresponding text probability distribution. For example, 2 candidate texts can be selected from their predicted texts. The number of selected candidate texts can be set according to actual conditions, and this disclosure does not make any special restrictions on this.

[0122] In step S702, the candidate text of the initial syllable is combined with multiple predicted texts corresponding to the next syllable, and a candidate combination result is selected from the multiple text combination results obtained.

[0123] In this step, after determining the candidate text of the initial syllable, the candidate text may be combined with multiple predicted texts corresponding to the next syllable (ie, the second syllable in the syllable sequence) to obtain multiple text combination results.

[0124] Specifically, refer to Figure 8 , Figure 8 A flowchart of selecting candidate combination results from text combination results according to an embodiment of the present disclosure is shown, including steps S801 and S802:

[0125] In step S801 , the score corresponding to each text combination result is obtained.

[0126] In this step, referring to the relevant explanation of the above-mentioned step S302, after combining the candidate text of the initial syllable with the multiple predicted texts corresponding to the next syllable, illustratively, taking the candidate text of the initial syllable as: y1 (90%), y2 (10%) as an example, and taking the multiple predicted texts corresponding to the second syllable as: y3 (60%), y4 (30%), y5 (10%) as an example, the obtained text combination results are: y1y3, y1y4, y1y5, y2y3, y2y4, y2y5.

[0127] Therefore, taking the example of directly determining the score corresponding to each text combination result based on the probability product of multiple predicted texts contained in the text combination result, the score corresponding to y1y3 is 90%*60%=54%, the score corresponding to y1y4 is 90%*30%=27%, the score corresponding to y1y5 is 90%*10%=9%, the score corresponding to y2y3 is 10%*60%=6%, the score corresponding to y2y4 is 10%*30%=3%, and the score corresponding to y2y5 is 10%*10%=1%.

[0128] It should be noted that different preset weights can also be assigned to the multiple predicted texts included in the text combination results. Then, according to the above preset weights, the probabilities of the multiple predicted texts included in the text combination results are weighted to obtain the scores corresponding to each text combination result. The scores can be set according to the actual situation, and this disclosure does not make any special restrictions on this.

[0129] In an optional embodiment, after initially determining the scores corresponding to the text combination results, you can refer to Figure 9 , Figure 9 A flowchart of updating the corresponding score of the text combination result according to an embodiment of the present disclosure is shown, including steps S901 and S902:

[0130] In step S901 , each text combination result is matched with a preset vocabulary library to determine whether the text combination result contains a preset vocabulary.

[0131] In this step, the text combination results obtained at each stage of the decoding process may be matched with a preset vocabulary library to determine whether each text combination result matches the preset vocabulary.

[0132] Among them, the preset vocabulary library includes multiple words determined based on the frequency of use of words in the target field. For example, the hot words corresponding to the e-commerce field may be "package", "express delivery", "payment", etc., the hot words corresponding to the education field may be "online class", "teaching", etc., and the hot words corresponding to the medical field may be "blood pressure", "pulse", etc.

[0133] For example, a preset reward value may be pre-assigned to each preset word. For example, the preset reward value corresponding to the preset word "package" may be set to 30%, and the preset reward value corresponding to the preset word "express delivery" may be set to "20%". These values ​​may be set according to actual circumstances, and the present disclosure does not impose any special restrictions on this. The purpose of the above-mentioned preset reward value is that when the recognized text combination result contains the preset word, the score of the text recognition result may be increased according to the preset reward value, so as to increase the possibility that the hot word is included in the final recognition result, thereby significantly improving the recall rate and accuracy of the hot word in the recognition result.

[0134] In step S902, if the preset vocabulary is included, the score corresponding to the text combination result is updated according to the preset reward value corresponding to the preset vocabulary.

[0135] In this step, if it is detected that the text combination result contains the above-mentioned preset vocabulary, the preset reward value corresponding to the preset vocabulary can be read from the above-mentioned preset vocabulary library, and the score corresponding to the text combination result can be updated according to the read preset reward value.

[0136] Referring to the relevant explanation of the above step S401, if it is detected that the text combination result "y1y4" contains the above-mentioned preset vocabulary "express delivery", the preset reward value corresponding to "express delivery" can be read from the preset vocabulary library as "20%", and thus, the score corresponding to the text combination result "y1y4" can be updated to: 27% + 20% = 47%.

[0137] Next reference Figure 8 In step S802, candidate combination results are selected from multiple text combination results in descending order of scores.

[0138] In this step, after obtaining the scores corresponding to the respective text combination results, the scores can be sorted in descending order, and then the text combination results whose scores meet the threshold conditions are selected from the sorted sequence as candidate combination results. For example, the text combination results with the top two scores can be selected as the candidate combination results (for example, "y2y5" and "y1y4"). The number of selected candidate combination results can be set according to actual circumstances, and this disclosure does not impose any special restrictions on this.

[0139] Next reference Figure 7 In step S703, the candidate combination results are sequentially combined with multiple predicted texts corresponding to subsequent syllables to obtain a candidate text sequence.

[0140] In this step, after obtaining the candidate combination result corresponding to the second syllable, the above candidate combination result can be combined with multiple predicted texts corresponding to the third syllable, and the candidate combination result can be screened out and then combined with multiple predicted texts corresponding to the fourth syllable... Repeat the above steps until the multiple predicted texts corresponding to the ending syllable are combined to obtain the above candidate text sequence. It can be seen that, on the one hand, by selecting the candidate combination result with a higher score at each intermediate stage and combining it with the predicted texts corresponding to the subsequent syllables in turn to generate a candidate text sequence, the present disclosure can improve the accuracy of the generated candidate text sequence. On the other hand, the present disclosure can also reduce the memory requirements of the system and improve the generation speed of the candidate text sequence.

[0141] Next reference Figure 6 In step S604, the candidate text sequences are screened to obtain the text sequence corresponding to the speech data to be recognized.

[0142] In this step, after obtaining the above candidate text sequence, you can refer to Figure 10 , Figure 10 A flowchart of determining a text sequence corresponding to speech data to be recognized according to an embodiment of the present disclosure is shown, comprising steps S1001 to S1003:

[0143] In step S1001 , a first score corresponding to each candidate text sequence is obtained, and a second score of a syllable sequence corresponding to each candidate text sequence is obtained.

[0144] In this step, referring to the relevant explanations of the above steps, it can be seen that the first score corresponding to each candidate text sequence is the product of the probabilities of the multiple predicted texts included in the candidate text sequence.

[0145] Before decoding to obtain the candidate text sequence, the input syllable sequence is the syllable sequence corresponding to the candidate text sequence, and thus the second score is the product of the probabilities of the multiple predicted syllables contained in the syllable sequence.

[0146] In step S1002 , a comprehensive score is determined based on the first score and the second score.

[0147] In this step, the first score and the second score can be weighted and summed to obtain a comprehensive score. For example, taking the first score of any candidate text sequence as 0.8 and the second score of its syllable sequence as 0.6 as an example, in an optional embodiment, the comprehensive score can be In another optional implementation, the weight corresponding to the first score may be set to 0.4, and the weight corresponding to the second score may be set to 0.6, so that the comprehensive score may be 0.8*0.4+0.6*0.6=0.68.

[0148] The calculation method of the above-mentioned weight values ​​and comprehensive scores can be set according to actual conditions, and this disclosure does not impose any special restrictions on this.

[0149] In step S1003 , the candidate text sequences whose comprehensive scores are greater than a preset score threshold are determined as the text sequences corresponding to the speech data to be recognized.

[0150] In this step, for example, the candidate text sequence having the above-mentioned comprehensive score greater than a preset score threshold (which can be set or modified according to actual circumstances and is not specifically limited in this disclosure) can be determined as the text sequence corresponding to the above-mentioned speech data to be recognized. For example, the candidate text sequence with the highest comprehensive score can also be directly determined as the text sequence corresponding to the above-mentioned speech data to be recognized. This can be set or modified according to actual circumstances and is not specifically limited in this disclosure.

[0151] In an optional embodiment, after obtaining the above text sequence, it can also be processed based on a natural language processing (NLP) algorithm to obtain a final output result, thereby ensuring the accuracy and readability of the output result and making it more in line with human word formation habits.

[0152] refer to Figure 11 , Figure 11 The overall flow chart of the speech recognition method according to the embodiment of the present disclosure is shown, including steps S1101 to S1108:

[0153] In step S1101, speech features are input into an encoder to obtain encoded latent features.

[0154] In this step, the speech features corresponding to the speech data to be recognized can be extracted first, and then the speech features are input into the autoencoder to encode the speech features through the m-layer encoding network of the autoencoder to output an encoded latent feature.

[0155] In step S1102, the encoded latent features are input into a syllable classifier to obtain a syllable probability distribution.

[0156] In this step, after obtaining the encoded latent features, the encoded latent features can be input into the syllable classifier, and the syllable corresponding to each speech frame is predicted by the syllable classifier, and the syllable probability distribution is output. The syllable probability distribution is used to characterize the probability that each speech frame belongs to each predicted syllable.

[0157] In step S1103, a beam search is performed based on the above syllable probability distribution to obtain a syllable sequence.

[0158] In this step, a beam search can be performed based on the above-mentioned syllable probability distribution. Specifically, candidate syllables can be selected from the predicted syllables corresponding to the first frame based on the syllable probability distribution corresponding to the first frame, and the candidate syllables can be combined with the predicted syllables corresponding to the second frame to obtain a syllable combination result. According to the score of the syllable combination result (the score of the syllable combination result is the product of the probabilities of the multiple predicted syllables contained therein), a candidate combination result is selected therefrom, and the candidate combination result is sequentially combined with the predicted syllables corresponding to subsequent speech frames to obtain a syllable sequence.

[0159] After obtaining the syllable combination results of each intermediate stage, hot word syllable recognition can be performed on the syllable combination results. If the syllable combination results contain hot word syllables, the corresponding scores are updated.

[0160] In step S1104, the encoded latent features and the syllable sequence are input into a text decoder to obtain text features.

[0161] In this step, the encoded latent features obtained in step S1101 and the syllable sequence obtained in the above step S1104 can be input into a text decoder to output a text feature.

[0162] In step S1105 , the text features are input into a text classifier to obtain a text probability distribution.

[0163] In this step, after obtaining the text features, the text features can be input into a text classifier, which predicts the text corresponding to each syllable and outputs a text probability distribution, which is used to characterize the probability that each syllable belongs to each predicted text.

[0164] In step S1106 , a beam search is performed based on the above text probability distribution to obtain a candidate text sequence.

[0165] In this step, a beam search can be performed based on the above-mentioned text probability distribution. Specifically, a candidate text can be selected from the predicted text corresponding to the starting syllable based on the text probability distribution corresponding to the starting syllable, and the candidate text can be combined with the predicted text corresponding to the next syllable to obtain a text combination result. According to the score of the text combination result (the score of the text combination result is the product of the probabilities of the multiple predicted texts contained therein), a candidate combination result can be selected from it, and the candidate combination result can be combined with the predicted text corresponding to each subsequent syllable in turn to obtain a candidate text sequence.

[0166] After obtaining the text combination result of each intermediate stage, hot word recognition can be performed on the text combination result. If the text combination result contains a hot word, its corresponding score is updated.

[0167] By first performing hot word syllable recognition from the syllable dimension and then performing hot word recognition from the text dimension during the speech recognition process, the possibility of including hot words in the final recognition results is increased, and the recall rate and accuracy of hot words in the recognition results are significantly improved.

[0168] In step S1107 , a text sequence corresponding to the speech data to be recognized is selected from the candidate text sequences.

[0169] In this step, the candidate text sequence with the highest score can be determined as the text sequence corresponding to the voice data to be recognized and output, or the candidate text sequences with the top scores can be determined as the text sequence corresponding to the voice data to be recognized and output. It can be set according to actual conditions, and this disclosure does not make any special restrictions on this.

[0170] The present disclosure also provides a method for training a speech recognition model, referring to Figure 12 , Figure 12 A flowchart of a method for training a speech recognition model according to an embodiment of the present disclosure is shown, comprising steps S1210 to S1240:

[0171] In step S1210, a training sample is obtained.

[0172] In this step, a training sample of the model may be obtained. The training sample may include sample speech data and labels corresponding to the sample speech data. The labels are annotation information corresponding to the sample speech data. For example, the labels include syllable labels and text labels.

[0173] For example, if the speech contained in a certain piece of sample speech data is "hello", then its corresponding syllable label may be "ni3hao3" and the text label may be "hello".

[0174] In step S1220, the training sample is input into the speech recognition model to be trained to obtain a speech recognition processing result.

[0175] In this step, the above training samples can be input into the speech recognition model to be trained, and the speech recognition processing results can be obtained according to the output of the above speech recognition model. The above speech recognition processing results can include two parts, namely syllable recognition results and text recognition results.

[0176] Specifically, the speech recognition model to be trained may include the following parts: a feature extraction network, an encoder, a syllable classifier, a text decoder, and a text classifier. Thus, after the training sample is input into the speech recognition model to be trained, the speech features corresponding to the training sample can be extracted by the feature extraction network first, and then the speech features can be input into the encoder and encoded by the encoder to obtain the encoded latent features h. x , then, we can use the syllable classifier to predict the syllable probability distribution corresponding to the encoded latent features, and perform a beam search on the syllable probability distribution to obtain the syllable recognition result CTC(h x ).

[0177] Furthermore, the above-mentioned encoded latent features and syllable recognition results can be decoded again by a text decoder to obtain text features, and the text classifier can be used to predict the text probability distribution corresponding to the text features, and a beam search can be performed on the text probability distribution to obtain the text recognition result softmax(s y ). Among them, softmax is used in the classification process to implement multi-classification. Simply put, it maps some output neurons to real numbers between (0-1) and normalizes them to ensure that the sum is 1, so that the sum of the probabilities of multiple classifications is also exactly 1, that is, the final output is the probability of each classification being taken.

[0178] In step S1230, the loss value of the speech recognition model to be trained is determined according to the degree of difference between the speech recognition processing result and the label.

[0179] In this step, you can refer to Figure 13 , Figure 13A flowchart for determining the loss value of a speech recognition model to be trained according to an embodiment of the present disclosure is shown, including steps S1301 to S1303:

[0180] In step S1301, a first loss value output by the speech recognition model to be trained is determined based on the degree of difference between the syllable recognition result and the syllable label.

[0181] In this step, the first loss value output by the speech recognition model to be trained can be determined based on the degree of difference between the syllable recognition result and the syllable label. Specifically, the first loss value can be determined based on the following formula 1:

[0182] loss ctc =ctc_loss(Softmax(Linear1(h x ), S_Y)) Formula 1

[0183] Among them, loss ctc represents the first loss value mentioned above; ctc_loss() is a function that calculates the CTC (Connectionisttemporal classification, CTC for short) loss; Softmax() is a softmax function that calculates the probability distribution of each frame on the syllable dictionary (formed by numbering all syllables that appeared during the training phase); Linear1() is a linear layer that converts h x Mapping from the dimension of attention to the dimension of syllable dictionary length; h x It is the high-dimensional encoded latent feature output by the encoder; S_Y represents the above syllable label.

[0184] In step S1302, a second loss value output by the speech recognition model to be trained is determined based on the degree of difference between the text recognition result and the text label.

[0185] In this step, the second loss value of the speech recognition model to be trained can be determined based on the difference between the obtained text recognition result and the text label. Specifically, the second loss value can be determined based on the following formula 2:

[0186] loss_ce=ce_loss(Softmax(Linear2(s y ), Y)) Formula 2

[0187] Among them, loss_ce represents the second loss value mentioned above, ce_loss represents the function of calculating cross entropy loss; s y is the output of the text decoder, Linear2() is the linear layer, which is used to convert s yMapping from the attention dimension to the length dimension of the text dictionary (formed by numbering all text occurrences during the training phase); Softmax() is the Softmax function that calculates the probability distribution of each syllable on the text dictionary; Y represents the above text label.

[0188] In step S1303, the loss value of the speech recognition model to be trained is determined according to the first loss value and the second loss value.

[0189] In this step, after obtaining the above-mentioned first loss value and second loss value, illustratively, a preset weight α (a real number greater than 0 and less than 1) can be set for the first loss value, and the weight corresponding to the second loss value can be expressed as 1-α. Furthermore, the loss value of the speech recognition model to be trained can be determined according to the following formula 3:

[0190] loss=α*loss_ctc+(1-α)*loss_ce Formula 3

[0191] Among them, the above loss represents the loss value of the speech recognition model to be trained.

[0192] In step S1240, the speech recognition model to be trained is iteratively trained according to the loss value until the loss value meets a preset convergence condition, thereby obtaining a trained speech recognition model.

[0193] In this step, after determining the loss value of the above-mentioned speech recognition model to be trained, the speech recognition model to be trained can be iteratively trained according to the above-mentioned loss value (that is, the speech recognition model to be trained is repeatedly trained multiple times using different sample speech data in the above-mentioned training samples) until the loss value meets the preset convergence condition (for example: the above-mentioned loss value is less than the preset value, the preset value can be set according to the actual situation, and the present disclosure does not make any special restrictions on this), and the trained speech recognition model is obtained.

[0194] Afterwards, the relevant steps of the above steps S110 to S140 may be executed based on the above trained speech recognition model to implement the recognition process of the speech data to be recognized.

[0195] Based on the speech recognition model obtained in the present disclosure, the present disclosure can perform dual recognition of two dimensions (i.e., syllables and Chinese characters) on the speech data to be recognized, thereby ensuring the accuracy of speech recognition.

[0196] Exemplary devices

[0197] After introducing the speech recognition method of the exemplary embodiment of the present disclosure, next, Figure 14 The speech recognition device according to the embodiment of the present disclosure is described. Figure 14The voice recognition device shown can be used to execute the relevant steps of the above-mentioned voice recognition method.

[0198] Figure 14 A schematic diagram of a speech recognition device according to an embodiment of the present disclosure is shown. The speech recognition device 1400 includes:

[0199] The feature extraction module 1410 is used to extract speech features corresponding to multiple speech frames of the speech data to be recognized;

[0200] The encoding module 1420 is used to encode the speech feature to obtain the encoded latent feature of the speech feature;

[0201] a syllable decoding module 1430 for decoding the encoded latent features into a syllable sequence;

[0202] The text decoding module 1440 is used to re-decode the encoded latent features and the syllable sequence to obtain a text sequence corresponding to the speech data to be recognized.

[0203] In an optional embodiment, the syllable decoding module 1430 is configured to:

[0204] Inputting the encoded latent features into a pre-trained syllable classifier to generate a syllable probability distribution corresponding to each speech frame, wherein the syllable probability distribution is used to represent multiple predicted syllables corresponding to each speech frame and the probability that the speech frame belongs to each predicted syllable;

[0205] A candidate syllable is selected from the plurality of predicted syllables according to the syllable probability distribution to obtain the syllable sequence.

[0206] In an optional embodiment, the syllable decoding module 1430 is configured to:

[0207] Selecting a candidate syllable of the initial frame from a plurality of predicted syllables corresponding to the initial frame according to a syllable probability distribution corresponding to the initial frame; the initial frame being the first frame of the speech data to be recognized;

[0208] Combining the candidate syllable of the initial frame with multiple predicted syllables corresponding to the next frame, and selecting a candidate combination result from the multiple syllable combination results obtained;

[0209] The candidate combination results are sequentially combined with a plurality of predicted syllables corresponding to subsequent speech frames to obtain the syllable sequence.

[0210] In an optional implementation, the syllable decoding module 1430 is configured to:

[0211] Obtaining a score corresponding to each of the syllable combination results, where the score is calculated based on the probabilities of multiple predicted syllables included in the syllable combination result;

[0212] The candidate combination result is selected from the multiple syllable combination results in descending order of the scores.

[0213] In an optional implementation, the syllable decoding module 1430 is configured to:

[0214] Matching each of the syllable combination results with a preset syllable library to determine whether the syllable combination result contains a preset syllable sequence; the preset syllable library includes syllable sequences corresponding to multiple hot words, and the hot words are determined based on the frequency of use of vocabulary in the target field; each of the preset syllable sequences corresponds to a preset reward value;

[0215] If the preset syllable sequence is included, the score corresponding to the syllable combination result is updated according to the preset reward value corresponding to the preset syllable sequence.

[0216] In an optional implementation, the text decoding module 1440 is configured to:

[0217] Inputting the encoded latent features and the syllable sequence into a pre-trained text decoder to obtain text features corresponding to the speech data to be recognized;

[0218] Inputting the text features into a pre-trained text classifier to generate a text probability distribution corresponding to each syllable contained in the syllable sequence; the text probability distribution is used to represent multiple predicted texts corresponding to each syllable and the probability that the syllable belongs to each predicted text;

[0219] Selecting candidate texts from the plurality of predicted texts according to the text probability distribution to obtain a candidate text sequence;

[0220] The candidate text sequences are screened to obtain text sequences corresponding to the speech data to be recognized.

[0221] In an optional implementation, the text decoding module 1440 is configured to:

[0222] Selecting a candidate text for the starting syllable from a plurality of predicted texts corresponding to the starting syllable of the syllable sequence according to a text probability distribution corresponding to the starting syllable of the syllable sequence;

[0223] Combining the candidate text of the starting syllable with multiple predicted texts corresponding to the next syllable, and selecting a candidate combination result from the multiple text combination results obtained;

[0224] The candidate combination results are sequentially combined with multiple predicted texts corresponding to subsequent syllables to obtain the candidate text sequence.

[0225] In an optional implementation, the text decoding module 1440 is configured to:

[0226] Obtaining a score corresponding to each of the text combination results, where the score is calculated based on the probability of multiple texts included in the text combination result;

[0227] The candidate combination result is selected from the multiple text combination results in descending order of the scores.

[0228] In an optional implementation, the text decoding module 1440 is configured to:

[0229] Matching each of the text combination results with a preset vocabulary library to determine whether the text combination result contains a preset vocabulary; the preset vocabulary library includes a plurality of vocabulary determined based on the frequency of use of vocabulary in the target field, and each of the preset vocabulary corresponds to a preset reward value;

[0230] If the preset vocabulary is included, the score corresponding to the text combination result is updated according to the preset reward value corresponding to the preset vocabulary.

[0231] In an optional implementation, the text decoding module 1440 is configured to:

[0232] Obtaining a first score corresponding to each candidate text sequence, and obtaining a second score of a syllable sequence corresponding to each candidate text sequence;

[0233] determining a comprehensive score based on the first score and the second score;

[0234] The candidate text sequence whose comprehensive score is greater than a preset score threshold is determined as the text sequence corresponding to the speech data to be recognized.

[0235] It should be noted that, since the functional modules of the speech recognition device in the embodiment of the present disclosure are the same as those in the embodiment of the speech recognition method described above, they will not be described in detail here.

[0236] Next, refer to Figure 15 The training device for the speech recognition model according to the embodiment of the present disclosure is described. The training device for the speech recognition model can be used to execute the relevant steps of the training method for the speech recognition model described above.

[0237] Figure 15A schematic diagram of a speech recognition model training device according to an embodiment of the present disclosure is shown. The speech recognition model training device 1500 includes:

[0238] The sample acquisition module 1510 is used to acquire training samples, where the training samples include sample speech data and labels corresponding to the sample speech data; the labels include syllable labels and text labels.

[0239] The processing module 1520 is used to input the training samples into the speech recognition model to be trained to obtain a speech recognition processing result.

[0240] The loss determination module 1530 is used to determine the loss value of the speech recognition model to be trained according to the degree of difference between the speech recognition processing result and the label.

[0241] The model training module 1540 is used to iteratively train the speech recognition model to be trained according to the loss value until the loss value meets a preset convergence condition, thereby obtaining a trained speech recognition model.

[0242] In an optional embodiment, the speech recognition processing result includes a syllable recognition result and a text recognition result; the loss determination module 1530 is configured to:

[0243] Determining a first loss value output by the speech recognition model to be trained according to a degree of difference between the syllable recognition result and the syllable label;

[0244] Determining a second loss value output by the to-be-trained speech recognition model according to a degree of difference between the text recognition result and the text label;

[0245] Determine the loss value of the speech recognition model to be trained based on the first loss value and the second loss value.

[0246] It should be noted that since the various functional modules of the speech recognition model training device in the embodiment of the present disclosure are the same as those in the embodiment of the speech recognition model training method described above, they will not be described in detail here.

[0247] Exemplary Storage Media

[0248] Reference below Figure 16 A storage medium according to an exemplary embodiment of the present disclosure is described.

[0249] In this exemplary embodiment, the above method can be implemented by a program product 1600, such as a portable compact disc read-only memory (CD-ROM) that includes program code and can be executed on a device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0250] The program product can be implemented in any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0251] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0252] The program code contained on the readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber cable, RE, etc., or any suitable combination of the foregoing.

[0253] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (FAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0254] Exemplary electronic devices

[0255] refer to Figure 17 An electronic device according to an exemplary embodiment of the present disclosure will be described.

[0256] Figure 17 The electronic device 1700 shown is merely an example and should not limit the functionality and scope of use of the embodiments of the present disclosure.

[0257] like Figure 17 As shown, electronic device 1700 is implemented as a general-purpose computing device. Components of electronic device 1700 may include, but are not limited to, at least one processing unit 1710, at least one storage unit 1720, a bus 1730 connecting various system components (including storage unit 1720 and processing unit 1710), and a display unit 1740.

[0258] The storage unit stores program codes, which can be executed by the processing unit 1710, so that the processing unit 1710 performs the steps according to various exemplary embodiments of the present disclosure described in the "Exemplary Method" section above. For example, the processing unit 1710 can perform the following steps: Figure 1 The method steps shown, etc.

[0259] The storage unit 1720 may include a volatile storage unit, such as a random access memory unit (RAM) 1721 and / or a cache memory unit 1722 , and may further include a read-only memory unit (ROM) 1723 .

[0260] The storage unit 1720 may also include a program / utility 1724 having a set (at least one) of program modules 1725, such program modules 1725 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0261] The bus 1730 may include a data bus, an address bus, and a control bus.

[0262] Electronic device 1700 can also communicate with one or more external devices 1800 (e.g., a keyboard, pointing device, Bluetooth device, etc.), and such communication can be performed via input / output (I / O) interface 1750. Electronic device 1700 also includes a display unit 1740, which is connected to input / output (I / O) interface 1750 for display. Furthermore, electronic device 1700 can also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 1760. As shown, network adapter 1760 communicates with other modules of electronic device 1700 via bus 1730. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 1700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0263] It should be noted that although several modules or submodules of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more units / modules described above can be embodied in a single unit / module. Conversely, the features and functions of a single unit / module described above can be further divided and embodied by multiple units / modules.

[0264] Furthermore, although the operations of the disclosed method are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0265] Although the spirit and principles of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the specific embodiments disclosed, and the division into various aspects does not mean that the features in these aspects cannot be combined to benefit. Such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the appended claims.

Claims

1. A speech recognition method, characterized in that: include: Extracting speech features corresponding to multiple speech frames of speech data to be recognized; Encoding the speech feature to obtain an encoded latent feature of the speech feature; Decoding the encoded latent features into a syllable sequence; Decoding the encoded latent features into a syllable sequence includes: Inputting the encoded latent features into a pre-trained syllable classifier to generate a syllable probability distribution corresponding to each speech frame, wherein the syllable probability distribution is used to represent multiple predicted syllables corresponding to each speech frame and the probability that the speech frame belongs to each predicted syllable; Selecting candidate syllables from the plurality of predicted syllables according to the syllable probability distribution based on a beam search algorithm to obtain the syllable sequence; Inputting the encoded latent features and the syllable sequence into a pre-trained text decoder to obtain text features corresponding to the speech data to be recognized; Inputting the text features into a pre-trained text classifier to generate a text probability distribution corresponding to each syllable contained in the syllable sequence; the text probability distribution is used to represent multiple predicted texts corresponding to each syllable and the probability that the syllable belongs to each predicted text; Selecting candidate texts from the plurality of predicted texts according to the text probability distribution to obtain a candidate text sequence; The candidate text sequences are screened to obtain text sequences corresponding to the speech data to be recognized.

2. The method according to claim 1, characterized in that The step of selecting a candidate syllable from the plurality of predicted syllables according to the syllable probability distribution to obtain the syllable sequence includes: Selecting a candidate syllable of the initial frame from a plurality of predicted syllables corresponding to the initial frame according to a syllable probability distribution corresponding to the initial frame; the initial frame being the first frame of the speech data to be recognized; Combining the candidate syllable of the initial frame with multiple predicted syllables corresponding to the next frame, and selecting a candidate combination result from the multiple syllable combination results obtained; The candidate combination results are sequentially combined with a plurality of predicted syllables corresponding to subsequent speech frames to obtain the syllable sequence.

3. The method according to claim 2, characterized in that The step of selecting a candidate combination result from the obtained multiple syllable combination results includes: Obtaining a score corresponding to each of the syllable combination results, where the score is calculated based on the probabilities of multiple predicted syllables included in the syllable combination result; The candidate combination result is selected from the multiple syllable combination results in descending order of the scores.

4. The method according to claim 3, characterized in that The method further comprises: Matching each of the syllable combination results with a preset syllable library to determine whether the syllable combination result contains a preset syllable sequence; the preset syllable library includes syllable sequences corresponding to multiple hot words, and the hot words are determined based on the frequency of use of vocabulary in the target field; each of the preset syllable sequences corresponds to a preset reward value; If the preset syllable sequence is included, the score corresponding to the syllable combination result is updated according to the preset reward value corresponding to the preset syllable sequence.

5. The method according to claim 1, wherein The step of selecting candidate texts from the plurality of predicted texts according to the text probability distribution to obtain a candidate text sequence includes: Selecting a candidate text for the starting syllable from a plurality of predicted texts corresponding to the starting syllable of the syllable sequence according to a text probability distribution corresponding to the starting syllable of the syllable sequence; Combining the candidate text of the starting syllable with multiple predicted texts corresponding to the next syllable, and selecting a candidate combination result from the multiple text combination results obtained; The candidate combination results are sequentially combined with multiple predicted texts corresponding to subsequent syllables to obtain the candidate text sequence.

6. The method according to claim 5, characterized in that The step of selecting a candidate combination result from the obtained multiple text combination results includes: Obtaining a score corresponding to each of the text combination results, where the score is calculated based on the probability of multiple texts included in the text combination result; The candidate combination result is selected from the multiple text combination results in descending order of the scores.

7. The method according to claim 6, characterized in that The method further comprises: Matching each of the text combination results with a preset vocabulary library to determine whether the text combination result contains a preset vocabulary; the preset vocabulary library includes a plurality of vocabulary determined based on the frequency of use of vocabulary in the target field, and each of the preset vocabulary corresponds to a preset reward value; If the preset vocabulary is included, the score corresponding to the text combination result is updated according to the preset reward value corresponding to the preset vocabulary.

8. The method according to claim 1, characterized in that The step of screening the candidate text sequences to obtain a text sequence corresponding to the speech data to be recognized includes: Obtaining a first score corresponding to each candidate text sequence, and obtaining a second score of a syllable sequence corresponding to each candidate text sequence; determining a comprehensive score based on the first score and the second score; The candidate text sequence whose comprehensive score is greater than a preset score threshold is determined as the text sequence corresponding to the speech data to be recognized.

9. A method for training a speech recognition model, characterized in that: include: Acquire a training sample, wherein the training sample includes sample speech data and a label corresponding to the sample speech data; The labels include syllable labels and text labels; Inputting the training sample into the speech recognition model to be trained to obtain a speech recognition processing result; the speech recognition processing result includes a syllable recognition result and a text recognition result; Determining a loss value of the speech recognition model to be trained based on a degree of difference between the syllable recognition result and the syllable label, and a degree of difference between the speech recognition processing result and the text label; Iteratively training the speech recognition model to be trained according to the loss value until the loss value meets a preset convergence condition, thereby obtaining a trained speech recognition model; The trained speech recognition model is used to perform the following process: Extracting speech features corresponding to multiple speech frames of speech data to be recognized; Encoding the speech feature to obtain an encoded latent feature of the speech feature; Inputting the encoded latent features into a syllable classifier to generate a syllable probability distribution corresponding to each speech frame, wherein the syllable probability distribution is used to represent multiple predicted syllables corresponding to each speech frame and the probability that the speech frame belongs to each predicted syllable; Selecting candidate syllables from the plurality of predicted syllables according to the syllable probability distribution based on a beam search algorithm to obtain the syllable sequence; Inputting the encoded latent features and the syllable sequence into a text decoder to obtain text features corresponding to the speech data to be recognized; Inputting the text features into a text classifier to generate a text probability distribution corresponding to each syllable contained in the syllable sequence; the text probability distribution is used to represent multiple predicted texts corresponding to each syllable and the probability that the syllable belongs to each predicted text; Selecting candidate texts from the plurality of predicted texts according to the text probability distribution to obtain a candidate text sequence; The candidate text sequences are screened to obtain text sequences corresponding to the speech data to be recognized.

10. The method according to claim 9, characterized in that The determining the loss value of the speech recognition model to be trained according to the difference between the syllable recognition result and the syllable label and the difference between the speech recognition processing result and the text label includes: Determining a first loss value output by the speech recognition model to be trained according to a degree of difference between the syllable recognition result and the syllable label; Determining a second loss value output by the to-be-trained speech recognition model according to a degree of difference between the text recognition result and the text label; Determine the loss value of the speech recognition model to be trained based on the first loss value and the second loss value.

11. A speech recognition device, characterized in that: include: A feature extraction module, used to extract speech features corresponding to multiple speech frames of speech data to be recognized; An encoding module, configured to encode the speech feature to obtain an encoded latent feature of the speech feature; a syllable decoding module, configured to decode the encoded latent features into a syllable sequence; The syllable decoding module is configured to: Inputting the encoded latent features into a pre-trained syllable classifier to generate a syllable probability distribution corresponding to each speech frame, wherein the syllable probability distribution is used to represent multiple predicted syllables corresponding to each speech frame and the probability that the speech frame belongs to each predicted syllable; Selecting candidate syllables from the plurality of predicted syllables according to the syllable probability distribution based on a beam search algorithm to obtain the syllable sequence; A text decoding module, configured to input the encoded latent features and the syllable sequence into a pre-trained text decoder to obtain text features corresponding to the speech data to be recognized; Inputting the text features into a pre-trained text classifier to generate a text probability distribution corresponding to each syllable contained in the syllable sequence; the text probability distribution is used to represent multiple predicted texts corresponding to each syllable and the probability that the syllable belongs to each predicted text; Selecting candidate texts from the plurality of predicted texts according to the text probability distribution to obtain a candidate text sequence; The candidate text sequences are screened to obtain text sequences corresponding to the speech data to be recognized.

12. The device according to claim 11, characterized in that The syllable decoding module is configured to: Selecting a candidate syllable of the initial frame from a plurality of predicted syllables corresponding to the initial frame according to a syllable probability distribution corresponding to the initial frame; the initial frame being the first frame of the speech data to be recognized; Combining the candidate syllable of the initial frame with multiple predicted syllables corresponding to the next frame, and selecting a candidate combination result from the multiple syllable combination results obtained; The candidate combination results are sequentially combined with a plurality of predicted syllables corresponding to subsequent speech frames to obtain the syllable sequence.

13. The device according to claim 12, characterized in that The syllable decoding module is configured to: Obtaining a score corresponding to each of the syllable combination results, where the score is calculated based on the probabilities of multiple predicted syllables included in the syllable combination result; The candidate combination result is selected from the multiple syllable combination results in descending order of the scores.

14. The device according to claim 13, characterized in that The syllable decoding module is configured to: Matching each of the syllable combination results with a preset syllable library to determine whether the syllable combination result contains a preset syllable sequence; the preset syllable library includes syllable sequences corresponding to multiple hot words, and the hot words are determined based on the frequency of use of vocabulary in the target field; each of the preset syllable sequences corresponds to a preset reward value; If the preset syllable sequence is included, the score corresponding to the syllable combination result is updated according to the preset reward value corresponding to the preset syllable sequence.

15. The device according to claim 11, characterized in that The text decoding module is configured to: Selecting a candidate text for the starting syllable from a plurality of predicted texts corresponding to the starting syllable of the syllable sequence according to a text probability distribution corresponding to the starting syllable of the syllable sequence; Combining the candidate text of the starting syllable with multiple predicted texts corresponding to the next syllable, and selecting a candidate combination result from the multiple text combination results obtained; The candidate combination results are sequentially combined with multiple predicted texts corresponding to subsequent syllables to obtain the candidate text sequence.

16. The device according to claim 15, characterized in that The text decoding module is configured to: Obtaining a score corresponding to each of the text combination results, where the score is calculated based on the probability of multiple texts included in the text combination result; The candidate combination result is selected from the multiple text combination results in descending order of the scores.

17. The device according to claim 16, characterized in that The text decoding module is configured to: Matching each of the text combination results with a preset vocabulary library to determine whether the text combination result contains a preset vocabulary; the preset vocabulary library includes a plurality of vocabulary determined based on the frequency of use of vocabulary in the target field, and each of the preset vocabulary corresponds to a preset reward value; If the preset vocabulary is included, the score corresponding to the text combination result is updated according to the preset reward value corresponding to the preset vocabulary.

18. The device according to claim 11, characterized in that The text decoding module is configured to: Obtaining a first score corresponding to each candidate text sequence, and obtaining a second score of a syllable sequence corresponding to each candidate text sequence; determining a comprehensive score based on the first score and the second score; The candidate text sequence whose comprehensive score is greater than a preset score threshold is determined as the text sequence corresponding to the speech data to be recognized.

19. A training device for a speech recognition model, characterized in that: include: A sample acquisition module is used to acquire training samples, wherein the training samples include sample speech data and labels corresponding to the sample speech data; the labels include syllable labels and text labels; A processing module, configured to input the training sample into a speech recognition model to be trained to obtain a speech recognition processing result; the speech recognition processing result includes a syllable recognition result and a text recognition result; a loss determination module, configured to determine a loss value of the speech recognition model to be trained based on a degree of difference between the syllable recognition result and the syllable label, and a degree of difference between the text recognition result and the text label; A model training module is used to iteratively train the speech recognition model to be trained according to the loss value until the loss value meets a preset convergence condition, thereby obtaining a trained speech recognition model; The trained speech recognition model is used to perform the following process: Extracting speech features corresponding to multiple speech frames of speech data to be recognized; Encoding the speech feature to obtain an encoded latent feature of the speech feature; Inputting the encoded latent features into a syllable classifier to generate a syllable probability distribution corresponding to each speech frame, wherein the syllable probability distribution is used to represent multiple predicted syllables corresponding to each speech frame and the probability that the speech frame belongs to each predicted syllable; Selecting candidate syllables from the plurality of predicted syllables according to the syllable probability distribution based on a beam search algorithm to obtain the syllable sequence; Inputting the encoded latent features and the syllable sequence into a text decoder to obtain text features corresponding to the speech data to be recognized; Inputting the text features into a text classifier to generate a text probability distribution corresponding to each syllable contained in the syllable sequence; the text probability distribution is used to represent multiple predicted texts corresponding to each syllable and the probability that the syllable belongs to each predicted text; Selecting candidate texts from the plurality of predicted texts according to the text probability distribution to obtain a candidate text sequence; The candidate text sequences are screened to obtain text sequences corresponding to the speech data to be recognized.

20. The device according to claim 19, characterized in that The loss determination module is configured to: Determining a first loss value output by the speech recognition model to be trained according to a degree of difference between the syllable recognition result and the syllable label; Determining a second loss value output by the to-be-trained speech recognition model according to a degree of difference between the text recognition result and the text label; Determine the loss value of the speech recognition model to be trained based on the first loss value and the second loss value.

21. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

22. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to perform the method according to any one of claims 1 to 10 by executing the executable instructions.

Images