Speech recognition model training method, speech recognition method and device

Abstract

The invention relates to a speech recognition model training method, a speech recognition method, a speech recognition device, computer equipment, a storage medium and a computer program product. The method comprises the steps of obtaining a mixed text sample, and obtaining a voice sample and a tagged text sample based on the mixed text sample; performing feature extraction and coding on the voice sample to obtain a coding matrix; respectively inputting the coding matrix into a text recognition decoder and a conversion recognition decoder in a to-be-trained speech recognition model to obtain a text recognition result and a text conversion recognition result, and calculating a first model loss based on the text recognition result and the mixed text sample; calculating a second model loss based on the text conversion recognition result and the tagged text sample; and training a to-be-trained speech recognition model based on the first model loss and the second model loss to obtain a speech recognition model. The speech recognition model trained by the method can improve the recognition accuracy of the Chinese pinyin mixed speech.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech recognition model training method, a speech recognition method, an apparatus, a computer device, a storage medium and a computer program product. Background Art

[0002] Existing speech recognition technology is mainly aimed at speech recognition of mainstream languages ​​such as Chinese and English, and has not yet been studied for Chinese Pinyin recognition, especially Chinese Pinyin mixed recognition. Since Chinese Pinyin is the modern standard for Chinese phonetic notation, many Chinese characters and Chinese Pinyin have a very high similarity, and the amount of Chinese Pinyin mixed speech data is limited. The speech recognition model trained by the traditional speech recognition model training method has the problem of low recognition accuracy for Chinese Pinyin mixed speech. Summary of the invention

[0003] Based on this, it is necessary to provide a speech recognition model training method, speech recognition method, device, computer equipment, computer readable storage medium and computer program product to address the above technical problems. Using the trained speech recognition model for speech recognition can improve the recognition accuracy of Chinese pinyin mixed speech.

[0004] In a first aspect, the present application provides a speech recognition model training method, comprising:

[0005] Obtain a mixed text sample, where the mixed text sample includes Chinese character text and pinyin text;

[0006] Synthesize speech samples based on mixed text samples, add switching labels at the conversion point between Chinese text and Pinyin text, and obtain labeled text samples;

[0007] Extract features from speech samples to obtain speech features, encode the speech features to obtain a coding matrix;

[0008] Input the encoding matrix into the text recognition decoder in the speech recognition model to be trained to obtain a text recognition result, and calculate the first model loss based on the text recognition result and the mixed text sample;

[0009] Input the encoding matrix into the conversion recognition decoder in the speech recognition model to be trained to obtain the text conversion recognition result, and calculate the second model loss based on the text conversion recognition result and the labeled text sample;

[0010] Based on the first model loss and the second model loss, the speech recognition model to be trained is trained to obtain a trained speech recognition model.

[0011] In a second aspect, the present application also provides a speech recognition model training device, comprising:

[0012] A first acquisition module is used to acquire a mixed text sample, where the mixed text sample includes Chinese character text and pinyin text;

[0013] A synthesis module is used to synthesize speech samples based on mixed text samples, add switching labels at the conversion point between Chinese text and Pinyin text, and obtain labeled text samples;

[0014] The feature processing module is used to extract features from the speech sample to obtain speech features, encode the speech features to obtain a coding matrix;

[0015] A first decoding module, used for inputting the encoding matrix into a text recognition decoder in the speech recognition model to be trained to obtain a text recognition result, and calculating a first model loss based on the text recognition result and the mixed text sample;

[0016] A second decoding module is used to input the encoding matrix into a conversion recognition decoder in the speech recognition model to be trained to obtain a text conversion recognition result, and calculate a second model loss based on the text conversion recognition result and the labeled text sample;

[0017] The training module is used to train the speech recognition model to be trained based on the first model loss and the second model loss to obtain a trained speech recognition model.

[0018] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0019] Obtain a mixed text sample, where the mixed text sample includes Chinese character text and pinyin text;

[0020] Synthesize speech samples based on mixed text samples, add switching labels at the conversion point between Chinese text and Pinyin text, and obtain labeled text samples;

[0021] Extract features from speech samples to obtain speech features, encode the speech features to obtain a coding matrix;

[0022] Input the encoding matrix into the text recognition decoder in the speech recognition model to be trained to obtain a text recognition result, and calculate the first model loss based on the text recognition result and the mixed text sample;

[0023] Input the encoding matrix into the conversion recognition decoder in the speech recognition model to be trained to obtain the text conversion recognition result, and calculate the second model loss based on the text conversion recognition result and the labeled text sample;

[0024] Based on the first model loss and the second model loss, the speech recognition model to be trained is trained to obtain a trained speech recognition model.

[0025] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0026] Obtain a mixed text sample, where the mixed text sample includes Chinese character text and pinyin text;

[0027] Synthesize speech samples based on mixed text samples, add switching labels at the conversion point between Chinese text and Pinyin text, and obtain labeled text samples;

[0028] Extract features from speech samples to obtain speech features, encode the speech features to obtain a coding matrix;

[0029] Input the encoding matrix into the text recognition decoder in the speech recognition model to be trained to obtain a text recognition result, and calculate the first model loss based on the text recognition result and the mixed text sample;

[0030] Input the encoding matrix into the conversion recognition decoder in the speech recognition model to be trained to obtain the text conversion recognition result, and calculate the second model loss based on the text conversion recognition result and the labeled text sample;

[0031] Based on the first model loss and the second model loss, the speech recognition model to be trained is trained to obtain a trained speech recognition model.

[0032] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0033] Obtain a mixed text sample, where the mixed text sample includes Chinese character text and pinyin text;

[0034] Synthesize speech samples based on mixed text samples, add switching labels at the conversion point between Chinese text and Pinyin text, and obtain labeled text samples;

[0035] Extract features from speech samples to obtain speech features, encode the speech features to obtain a coding matrix;

[0036] Input the encoding matrix into the text recognition decoder in the speech recognition model to be trained to obtain a text recognition result, and calculate the first model loss based on the text recognition result and the mixed text sample;

[0037] Input the encoding matrix into the conversion recognition decoder in the speech recognition model to be trained to obtain the text conversion recognition result, and calculate the second model loss based on the text conversion recognition result and the labeled text sample;

[0038] Based on the first model loss and the second model loss, the speech recognition model to be trained is trained to obtain a trained speech recognition model.

[0039] The speech recognition model training method, device, computer equipment, storage medium and computer program product obtain a mixed text sample containing Chinese text and pinyin text, synthesize a speech sample based on the mixed text sample, add a switching label at the conversion point of Chinese text and pinyin text, and obtain a labeled text sample. Since the language sample and the labeled text sample are obtained based on the mixed text sample, the data volume of the speech sample is expanded, which is conducive to improving the speech recognition accuracy of the speech recognition model. The encoding matrix is ​​obtained by feature extraction and encoding of the speech sample, and the encoding matrix is ​​respectively input into the text recognition decoder and the conversion recognition decoder in the speech recognition model to be trained to obtain the text recognition result and the text conversion recognition result. Based on the text recognition result and the mixed text sample, the first model loss is calculated, and based on the text conversion recognition result and the labeled text sample, the second model loss is calculated, so that the speech recognition model to be trained is trained using the two model losses to obtain a trained speech recognition model. This speech recognition model training method uses two decoders to decode the encoding matrix, and the two decoders respectively learn the corresponding relationship between speech and text and the conversion of Chinese text and pinyin text, which is conducive to improving the recognition accuracy of Chinese-pinyin mixed speech.

[0040] In a sixth aspect, the present application also provides a speech recognition method, comprising:

[0041] Acquire the mixed speech to be recognized; extract features of the mixed speech to be recognized to obtain features of the speech to be recognized, encode the features of the speech to be recognized to obtain an encoding matrix to be recognized;

[0042] Through the text recognition decoder in the trained speech recognition model, text recognition is performed on the coding matrix to be recognized, and a text recognition result corresponding to the coding matrix to be recognized is obtained;

[0043] Through the conversion recognition decoder in the trained speech recognition model, text conversion recognition is performed on the coding matrix to be recognized, and a text conversion recognition result corresponding to the coding matrix to be recognized is obtained;

[0044] The final result of text recognition is determined based on the text recognition result corresponding to the coding matrix to be recognized and the text conversion recognition result corresponding to the coding matrix to be recognized.

[0045] In a seventh aspect, the present application further provides a speech recognition device, comprising:

[0046] The second acquisition module is used to acquire the mixed speech to be recognized; extract the features of the mixed speech to be recognized to obtain the features of the speech to be recognized, encode the features of the speech to be recognized to obtain the encoding matrix to be recognized;

[0047] The first recognition module is used to perform text conversion recognition on the coding matrix to be recognized through the conversion recognition decoder in the trained speech recognition model to obtain the text conversion recognition result corresponding to the coding matrix to be recognized;

[0048] The second recognition module is used to perform text conversion recognition on the coding matrix to be recognized through the conversion recognition decoder in the trained speech recognition model to obtain the text conversion recognition result corresponding to the coding matrix to be recognized;

[0049] The determination module is used to determine the final result of text recognition based on the text recognition result corresponding to the coding matrix to be recognized and the text conversion recognition result corresponding to the coding matrix to be recognized.

[0050] In an eighth aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0051] Acquire the mixed speech to be recognized; extract features of the mixed speech to be recognized to obtain features of the speech to be recognized, encode the features of the speech to be recognized to obtain an encoding matrix to be recognized;

[0052] Through the text recognition decoder in the trained speech recognition model, text recognition is performed on the coding matrix to be recognized, and a text recognition result corresponding to the coding matrix to be recognized is obtained;

[0053] Through the conversion recognition decoder in the trained speech recognition model, text conversion recognition is performed on the coding matrix to be recognized, and a text conversion recognition result corresponding to the coding matrix to be recognized is obtained;

[0054] The final result of text recognition is determined based on the text recognition result corresponding to the coding matrix to be recognized and the text conversion recognition result corresponding to the coding matrix to be recognized.

[0055] In a ninth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0056] Acquire the mixed speech to be recognized; extract features of the mixed speech to be recognized to obtain features of the speech to be recognized, encode the features of the speech to be recognized to obtain an encoding matrix to be recognized;

[0057] Through the text recognition decoder in the trained speech recognition model, text recognition is performed on the coding matrix to be recognized, and a text recognition result corresponding to the coding matrix to be recognized is obtained;

[0058] Through the conversion recognition decoder in the trained speech recognition model, text conversion recognition is performed on the coding matrix to be recognized, and a text conversion recognition result corresponding to the coding matrix to be recognized is obtained;

[0059] The final result of text recognition is determined based on the text recognition result corresponding to the coding matrix to be recognized and the text conversion recognition result corresponding to the coding matrix to be recognized.

[0060] In a tenth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0061] Acquire the mixed speech to be recognized; extract features of the mixed speech to be recognized to obtain features of the speech to be recognized, encode the features of the speech to be recognized to obtain an encoding matrix to be recognized;

[0062] Through the text recognition decoder in the trained speech recognition model, text recognition is performed on the coding matrix to be recognized, and a text recognition result corresponding to the coding matrix to be recognized is obtained;

[0063] Through the conversion recognition decoder in the trained speech recognition model, text conversion recognition is performed on the coding matrix to be recognized, and a text conversion recognition result corresponding to the coding matrix to be recognized is obtained;

[0064] The final result of text recognition is determined based on the text recognition result corresponding to the coding matrix to be recognized and the text conversion recognition result corresponding to the coding matrix to be recognized.

[0065] The above-mentioned speech recognition method, apparatus, computer equipment, storage medium and computer program product respectively perform text recognition and text conversion recognition on the coding matrix to be recognized corresponding to the mixed speech to be recognized through the text recognition decoder and conversion recognition decoder in the trained speech recognition model, obtain the text recognition result and text conversion recognition result corresponding to the coding matrix to be recognized, and determine the final text recognition result based on the text recognition result corresponding to the coding matrix to be recognized and the text conversion recognition result corresponding to the coding matrix to be recognized. Since the two decoders included in the trained speech recognition model have respectively learned the correspondence between speech and text and the conversion between Chinese text and pinyin text, two decoders are used to perform text recognition and text conversion recognition on the coding matrix to be recognized, and the final text recognition result is obtained by combining the two output results, which is conducive to improving the recognition accuracy of Chinese and pinyin mixed speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 A diagram showing an application environment of a speech recognition model training method in one embodiment;

[0068] Figure 2 A flowchart of a speech recognition model training method in one embodiment;

[0069] Figure 3 The figure is a schematic diagram of the overall process of a speech recognition model training method in one embodiment;

[0070] Figure 4 A flowchart of a speech recognition model training method in another embodiment;

[0071] Figure 5 is a structural block diagram of a speech recognition model training device in one embodiment;

[0072] Figure 6 A flowchart of a speech recognition method in one embodiment;

[0073] Figure 7 is a structural block diagram of a speech recognition device in one embodiment;

[0074] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0076] The speech recognition model training method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers. The speech recognition model training method provided in the embodiment of the present application can be executed independently by the terminal 102 or the server 104, or can be executed collaboratively by the terminal 102 and the server 104. Taking the execution by the server 104 alone as an example: the server 104 obtains a mixed text sample, and the mixed text sample includes Chinese text and pinyin text; synthesizes a speech sample based on the mixed text sample, adds a switching label at the conversion point of the Chinese text and the pinyin text, and obtains a labeled text sample; performs feature extraction on the speech sample to obtain speech features, and encodes the speech features to obtain a coding matrix; inputs the coding matrix into the text recognition decoder in the speech recognition model to be trained to obtain a text recognition result, and calculates the first model loss based on the text recognition result and the mixed text sample; inputs the coding matrix into the conversion recognition decoder in the speech recognition model to be trained to obtain a text conversion recognition result, and calculates the second model loss based on the text conversion recognition result and the labeled text sample; trains the speech recognition model to be trained based on the first model loss and the second model loss to obtain a trained speech recognition model. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0077] In an exemplary embodiment, Figure 2 As shown, a speech recognition model training method is provided, and the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate the method, which includes the following steps 201 to 206. Among them:

[0078] Step 201, obtaining a mixed text sample, the mixed text sample includes Chinese character text and pinyin text.

[0079] The mixed text sample refers to a mixed text containing Chinese text and Pinyin text. The server obtains mixed text samples in various application scenarios, for example, the application scenarios may include teacher teaching scenarios, or journal / book scenarios. The server may also obtain audio or video data and extract mixed text samples from the audio or video data. The server may also obtain mixed text samples from an existing sample data set. The server may also generate mixed text samples by generating a model.

[0080] Step 202: Synthesize a speech sample based on the mixed text sample, and add a switching tag at the conversion point between the Chinese text and the pinyin text to obtain a tagged text sample.

[0081] Among them, the speech sample is speech data mixed with Chinese speech and pinyin speech. The server can convert each text data in the mixed text sample into corresponding speech data, so as to realize speech synthesis. In an actual scenario, the amount of data of the speech data mixed with Chinese speech and pinyin speech is small. The embodiment of the present application proposes a method for synthesizing a speech sample based on the mixed text sample, which is beneficial to increasing the amount of data of the speech sample.

[0082] The switching tag is a conversion mark between the Chinese text and the pinyin text in the mixed text sample. For example, if the mixed text sample is "study together aoe", a switching tag can be added between "习" and "a", and the resulting tagged text sample is "study together <switching tag>aoe".

[0083] In some embodiments, the method for adding the switching tag can be: the server identifies each character in the mixed text sample, determines that the character category to which each character belongs is a Chinese character category or a pinyin category, and adds a switching tag between any two adjacent characters whose character categories are different, so as to obtain a tagged text sample.

[0084] Step 203: Extract features from the speech sample to obtain speech features, and encode the speech features to obtain an encoding matrix.

[0085] Among them, feature extraction is used to convert the speech sample into feature data that can be processed by a neural network model. Specifically, the server performs pre-emphasis on the speech sample, frames the pre-emphasized speech sample, and windows each obtained speech frame to avoid spectral leakage. Fourier transform is performed on each windowed speech frame respectively, so as to convert the speech frame in the time domain into a frequency domain signal. A filter is used to filter the obtained frequency domain signal, and the filtering result is used as the speech feature. In some embodiments, the filter used can be a Mel filter bank. Using the Mel filter bank can simulate the perception of human ears for speech at different frequencies, take the logarithm of the filtering result, and perform discrete cosine transform on the result after taking the logarithm to obtain cepstral coefficients. The speech feature vector of each speech frame is composed of cepstral coefficients, and the speech feature includes speech feature vectors of multiple speech frames.

[0086] The server encodes the speech features using a pre-trained encoder to obtain encoded features. In some embodiments, the encoder may use a self-attention mechanism encoder to capture the speech information in the speech sample. In a specific embodiment, the structure of the encoder may consist of a two-dimensional convolutional downsampling layer and a self-attention-based Transformer Encoder layer.

[0087] Step 204: input the encoding matrix into the text recognition decoder in the speech recognition model to be trained to obtain a text recognition result, and calculate the first model loss based on the text recognition result and the mixed text sample.

[0088] The speech recognition model to be trained is a common neural network model in the field of speech recognition, for example, a Transformer (a transformer model based on attention intelligence) or a Conformer (an end-to-end speech recognition model).

[0089] The speech recognition model to be trained proposed in the present application includes a text recognition decoder, which is used to decode the encoding matrix to obtain the text recognition result corresponding to the encoding matrix. For example, the text recognition decoder can be a CTC (Connectionist Temporal Classification) decoder. The use of the CTC decoder is conducive to frame-level speech recognition and can better decode the original variable-length acoustics.

[0090] In some embodiments, the text recognition result includes text data and multiple prediction probability vectors. The text data includes multiple characters, and the multiple characters correspond to multiple prediction probability vectors one by one. The mixed text sample includes multiple characters and sample probability vectors corresponding to the multiple characters.

[0091] The server calculates the first model loss based on the prediction probability vectors corresponding to the multiple characters in the text recognition result and the sample probability vectors corresponding to the multiple characters in the mixed text sample.

[0092] Step 205: input the encoding matrix into the conversion recognition decoder in the speech recognition model to be trained to obtain the text conversion recognition result, and calculate the second model loss based on the text conversion recognition result and the labeled text sample.

[0093] The speech recognition model to be trained further includes a conversion recognition decoder, which is used to decode the encoding matrix to obtain a text conversion recognition result corresponding to the encoding matrix. The conversion recognition decoder can have the same network structure as the text recognition decoder, or a different network structure. For example, the conversion recognition decoder can be an attention decoder, and the use of an attention decoder can effectively process context modeling of long speech sequences.

[0094] The text conversion recognition result includes text data and multiple predicted probability vectors. The text data includes multiple characters and predicted conversion labels, and multiple characters or predicted conversion labels correspond to multiple predicted probability vectors one by one. The labeled text sample includes multiple characters, sample conversion labels, and their corresponding sample probability vectors.

[0095] The server calculates the second model loss based on the predicted probability vectors corresponding to the multiple characters and predicted conversion labels in the text conversion recognition result and the sample probability vectors corresponding to the multiple characters and sample conversion labels in the labeled text sample.

[0096] Step 206: Train the speech recognition model to be trained based on the first model loss and the second model loss to obtain a trained speech recognition model.

[0097] The server trains the two decoders in the speech recognition model to be trained according to the first model loss and the second model loss, respectively, stops training when the training stop condition is met, and obtains a trained speech recognition model. The trained speech recognition model is used to recognize Chinese-Pinyin mixed speech to obtain a Chinese-Pinyin mixed text result.

[0098] In the above-mentioned speech recognition model training method, a mixed text sample containing Chinese text and pinyin text is obtained, and a speech sample is synthesized based on the mixed text sample, and a switching label is added at the conversion point of Chinese text and pinyin text to obtain a labeled text sample. Since the language sample and the labeled text sample are obtained based on the mixed text sample, the data volume of the speech sample is expanded, which is conducive to improving the speech recognition accuracy of the speech recognition model. The encoding matrix is ​​obtained by feature extraction and encoding of the speech sample, and the encoding matrix is ​​respectively input into the text recognition decoder and the conversion recognition decoder in the speech recognition model to be trained to obtain the text recognition result and the text conversion recognition result. Based on the text recognition result and the mixed text sample, the first model loss is calculated, and based on the text conversion recognition result and the labeled text sample, the second model loss is calculated, so that the speech recognition model to be trained is trained using the two model losses to obtain a trained speech recognition model. This speech recognition model training method uses two decoders to decode the encoding matrix, and the two decoders respectively learn the corresponding relationship between speech and text and the conversion of Chinese text and pinyin text, which is conducive to improving the recognition accuracy of Chinese pinyin mixed speech.

[0099] In an exemplary embodiment, synthesizing a speech sample based on a mixed text sample includes: performing speech synthesis on the Chinese character text in the mixed text sample to obtain Chinese speech; querying a preset pinyin-speech relationship to obtain the pinyin speech corresponding to the pinyin text; determining the arrangement order of the Chinese character text and the pinyin text in the mixed text sample, and splicing the Chinese speech and the pinyin speech according to the arrangement order to obtain a speech sample.

[0100] Among them, each character in the mixed text in the server is compared with a preset character in a preset character set respectively to determine that the character category to which each character belongs is a Chinese character category or a pinyin category, so as to distinguish the Chinese character text and the pinyin text in the mixed text sample.

[0101] In some embodiments, the mixed text sample can be constructed by using a text generation model. For example, simulating a teacher's class scenario, using a text generation model to generate a mixed text sample. The mixed text sample includes pinyin text, Chinese text, and Chinese-pinyin mixed text. For example, the mixed text sample includes the following types:

[0102] 1) Single Chinese character pinyin, such as "a", "o", "e", "w", "y";

[0103] 2) Repeated single Chinese character pinyin, such as "a a a", "a a", "o o o";

[0104] 3) Divided according to the structure and characteristics of Chinese character pinyin, including single vowels, compound vowels, special vowels, nasal vowels, front nasal vowels, back nasal vowels, medial vowels, initials, and whole recognition, such as "vowels a o e", "medial vowels i, u, ü", "initials d t n l", etc.;

[0105] 4) Spelled Chinese character pinyin, such as "lái";

[0106] 5) Spelling type Chinese-pinyin mixed data, such as "méi梅h u ā花梅花";

[0107] 6) General knowledge type Chinese-pinyin mixed data, such as "大y小i", "有ā在先给大哥ā戴标调āo";

[0108] 7) Formula type Chinese-pinyin mixed data, such as "ü像一条大鲤鱼,嘴里吐泡üüü";

[0109] 8) Pronunciation type Chinese-pinyin mixed data, such as "ān: First pronounce the sound of ā, then gradually raise the tip of the tongue and press against the upper gum to pronounce the sound of n";

[0110] 9) Scenario-based Chinese pinyin-mixed data, simulating the context of a teacher's class, such as "āo is composed of the two single vowels ā and o that we have learned", and randomly adding filler words such as "嗯", "噢", "啊", etc., such as "Pay attention to distinguishing the whole recognition syllable wu, the big w with the little u behind it啊".

[0111] The server uses speech synthesis technology to perform speech synthesis on the Chinese character text in the mixed text sample to obtain Chinese speech. The speech synthesis technology used can include end-to-end speech synthesis technology, such as Tacotron (a deep learning model for speech synthesis) or FastSpeech (a fully convolutional neural network model), etc., personalized speech synthesis technology, such as VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech, an end-to-end text-to-speech model with adversarial learning) or Dict-tts (a variable container text-to-speech model), etc., speech cloning technology, such as MockingBrid (a real-time speech cloning software) or VALL-E (a neural codec language model), etc.

[0112] The preset pinyin-speech relationship is used to represent the mapping relationship between pinyin text and speech. The construction method of the preset pinyin-speech relationship is as follows: The server obtains pinyin-speech data, roughly screens the obtained pinyin-speech data through a phoneme-level accent evaluation system, and associates the roughly screened pinyin-speech data with the corresponding pinyin text labels, thereby obtaining the preset pinyin-speech relationship.

[0113] Pinyin text labels include initials, finals, finals with tones, and whole recognition syllables. Among them,

[0114] There are 23 initials: b, p, m, f, d, t, n, l, g, k, h, j, q, x, zh, ch, sh, r, z, c, s, y, w.

[0115] There are 24 finals: a, o, e, i, u, ü, ai, ei, ui, ao, ou, iu, ie, üe, er, an, en, in, un, ün, ang, eng, ing, ong.

[0116] Contains 96 finals with tones: ā, á, ǎ, à, ō, ó, ǒ, ò, ē, é, ě, è, ī, í, ǐ, ì, ū, ú, ǔ, ù, ǖ, ǘ, ǚ, ǜ, āi, ái, ǎi, ài, ē i, éi, ěi, èi, uī, uí, uǐ, uì, āo, áo, ǎo, ào, ōu, óu, ǒu, òu, iū, iú, iǔ, iù, iē, ié, iě, iè, üē ,üé,üě,üè,ēr,ér,ěr,èr,ān,án,ǎn,àn,ēn,én,ěn,èn,īn,ín,ǐn,ìn,ūn,ún,ǔn,ùn,ǖn, ǘn, ǚn, ǜn, āng, áng, ǎng, àng, ēng, éng, ěng, èng, īng, íng, ǐng, ìng, ōng, óng, ǒng, òng.

[0117] Recognize and read 16 syllables as a whole: zhi, chi, shi, ri, zi, ci, si, yi, wu, yu, ye, yue, yin, yun, yuɑn, ying.

[0118] Each Chinese character text and Pinyin text has a fixed arrangement order in the mixed text sample. According to the arrangement order, the Chinese voice and the Pinyin voice are spliced, and the spliced ​​result is used as the voice sample.

[0119] In this embodiment, the Chinese text in the mixed text sample is synthesized into Chinese speech through speech synthesis technology, the pinyin speech corresponding to the pinyin text is queried, and the speech corresponding to each text is spliced ​​to obtain a speech sample. Since the amount of speech data mixed with pinyin and Chinese is relatively small in real scenarios, the use of this synthesis method is conducive to expanding the number of speech samples, thereby facilitating the training of a speech recognition model with better performance.

[0120] In an exemplary embodiment, a text recognition decoder includes a connected temporal classification decoder and an attention decoder; the encoding matrix is ​​input into the text recognition decoder in the speech recognition model to be trained to obtain a text recognition result, including: inputting the encoding matrix into the connected temporal classification decoder to obtain a first text recognition result; inputting the encoding matrix into the attention decoder to obtain a second text recognition result, and the text recognition results include a first text recognition result and a second text recognition result.

[0121] The connection time classification decoder is used to perform frame-level speech recognition on the coding matrix to obtain a first text recognition result corresponding to the coding matrix. The attention decoder is used to perform word-level speech recognition on the coding matrix to obtain a second text recognition result. In some embodiments, the connection time classification decoder adopts a CTC decoder, and the attention decoder adopts a Transformer decoder. The attention decoder includes a Transformer Decoder layer and a linear mapping layer.

[0122] In this embodiment, by connecting the temporal classification decoder to perform frame-level speech recognition on the coding matrix, and using the attention decoder to perform word-level speech recognition on the coding matrix, it is beneficial to improve the accuracy of the decoder output results, thereby improving the speech recognition accuracy of the trained speech recognition model.

[0123] In an exemplary embodiment, a speech sample includes multiple speech frames, and a coding matrix includes coding vectors corresponding to each of the multiple speech frames; the coding matrix is ​​input into a connected temporal classification decoder to obtain a first text recognition result, including: extracting coding vectors corresponding to each of the speech frames from the coding matrix; for each coding vector, decoding the coding vector through a connected temporal classification decoder to obtain the probability that the speech frames corresponding to the coding vector are multiple preset characters; determining a target text whose probability meets a preset probability condition from the multiple preset characters; performing deduplication processing on the target text corresponding to each of the multiple coding vectors, and using the deduplication result as the first text recognition result.

[0124] Among them, since the coding matrix is ​​composed of coding vectors corresponding to multiple speech frames, the connected temporal classification decoder decodes the coding vector corresponding to each speech frame respectively, thereby realizing frame-level speech recognition.

[0125] The connection time classification decoder can predict the probability of multiple preset texts for the speech frame corresponding to each coding vector to obtain multiple probabilities. The preset probability condition can be to use the first preset text corresponding to the maximum probability among the multiple probabilities as the target text, or to use the second preset text corresponding to the target probability greater than the preset probability among the multiple probabilities as the target text.

[0126] Multiple encoding vectors correspond to multiple target texts one by one. Since a voice word in actual voice data often lasts for multiple voice frames, the target texts corresponding to multiple encoding vectors have text duplication problems. The server removes duplicate texts from the multiple target texts and uses the result after duplication as the first text recognition result.

[0127] In this embodiment, the coding vector corresponding to each speech frame is decoded by connecting the temporal classification decoder to achieve speech recognition at the speech sample frame level, which is beneficial to improving the speech recognition accuracy of the trained speech recognition model.

[0128] In an exemplary embodiment, a speech sample includes multiple speech frames, and a coding matrix includes coding vectors corresponding to each of the multiple speech frames; the coding matrix is ​​input into an attention decoder to obtain a second text recognition result, including: extracting the coding vectors corresponding to each of the speech frames from the coding matrix; determining the first target text corresponding to each coding vector from multiple first texts contained in a first dictionary; and linearly mapping the multiple first target texts to obtain a second text recognition result.

[0129] The server presets a first dictionary, which contains multiple first characters. The Transformer Decoder layer in the attention decoder predicts the probability of each speech frame corresponding to each encoding vector being the multiple first characters, obtains multiple probabilities, and determines the first target character whose probability meets the preset probability condition from the multiple first characters.

[0130] The linear mapping layer in the attention decoder linearly maps the multiple first target characters to obtain a second text recognition result.

[0131] In this embodiment, the first target text corresponding to each speech frame is determined through the Transformer Decoder layer in the attention decoder, and then multiple first target texts are mapped to second text recognition results through the linear mapping layer, thereby realizing word-level recognition of speech samples.

[0132] In an exemplary embodiment, a first model loss is calculated based on text recognition results and mixed text samples, including: obtaining a first loss based on the first text recognition results and the mixed text samples; obtaining a second loss based on the second text recognition results and the mixed text samples; and performing weighted summation of the first loss and the second loss to obtain the first model loss.

[0133] The first text recognition result includes first text data and a plurality of first prediction probability vectors. The first text data includes a plurality of first character data, and the plurality of first character data corresponds to the plurality of first prediction probability vectors one by one.

[0134] The second text recognition result includes second text data and a plurality of second prediction probability vectors. The second text data includes a plurality of second character data and prediction conversion labels, and the plurality of second character data or prediction conversion labels correspond to the plurality of second prediction probability vectors one by one.

[0135] The mixed text sample includes third text data and a plurality of third prediction probability vectors. The third text data includes a plurality of third character data, and the plurality of third character data corresponds one to one with the plurality of third prediction probability vectors.

[0136] The first prediction probability vector is composed of a plurality of first probabilities, and each first probability is the probability that the corresponding first text data is predicted to be a plurality of preset texts.

[0137] The second prediction probability vector is composed of a plurality of second probabilities, and each second probability is a probability that the corresponding second character data is predicted to be a plurality of first characters in the first dictionary.

[0138] The third prediction probability vector is composed of a plurality of third probabilities, and each third probability is the probability that the corresponding third character data is a plurality of first characters in the first dictionary.

[0139] The server brings the first prediction probability vector and the third prediction probability vector into the first loss function to obtain the first loss, brings the second prediction probability vector and the third prediction probability vector into the second loss function to obtain the second loss, performs weighted summation on the first loss and the second loss, and obtains the summation result as the first model loss.

[0140] For example, the first dictionary includes 10 first characters, and the first text recognition result includes 2 first character data and first prediction probability vectors x1 and x2 corresponding to the 2 first character data.

[0141] x1={0.1,0.2,0.4,0.1,0.18,0.03,0.03,0.02,0.02,0.02},

[0142] x2={0.05,0.25,0.1,0.4,0.12,0.02,0.02,0.02,0.01,0.01}.

[0143] The second text recognition result includes two second character data and second prediction probability vectors y1 and y2 corresponding to the two second character data respectively.

[0144] The mixed text sample includes two third character data and third prediction probability vectors z1 and z2 corresponding to the two third character data, wherein p1 = {1, 0, 0, 0, 0, 0, 0, 0} and p2 = {0, 0, 1, 0, 0, 0, 0, 0, 0}.

[0145] The server substitutes the first predicted probability vector x1, x2, and the third predicted probability vector z1 and z2 into the first loss function to obtain the first loss, substitutes the second predicted probability vector y1, y2, and the third predicted probability vector z1 and z2 into the second loss function to obtain the second loss, and weightedly sums the first loss and the second loss to obtain the first model loss.

[0146] In this embodiment, the first model loss is determined by combining the text recognition results of the two decoders, which is beneficial to improving the recognition accuracy of the trained speech recognition model for text.

[0147] In an exemplary embodiment, a speech sample includes a plurality of speech frames, and a coding matrix includes coding vectors corresponding to each of the plurality of speech frames; the coding matrix is ​​input into a conversion recognition decoder in a speech recognition model to be trained to obtain a text conversion recognition result, including: extracting the coding vectors corresponding to each of the speech frames from the coding matrix; determining a second target text or a target conversion label corresponding to each coding vector among a plurality of second texts and a plurality of conversion labels contained in a second dictionary; and linearly mapping the plurality of second target texts and the target conversion labels to obtain a text conversion recognition result.

[0148] A second dictionary is preset in the server, and the second dictionary contains multiple second characters and multiple conversion labels. The conversion recognition decoder includes a Transformer Decoder layer and a linear mapping layer. The Transformer Decoder layer predicts the probability of the speech frame corresponding to each encoding vector being multiple second characters or multiple conversion labels, obtains multiple probabilities, and determines the second target character or target conversion label whose probability meets the preset probability condition from the multiple second characters or multiple conversion labels.

[0149] The linear mapping layer in the conversion recognition decoder performs linear mapping on the plurality of second target characters and the target conversion labels to obtain a second text recognition result.

[0150] In this embodiment, the Transformer Decoder layer in the transformation recognition decoder is used to determine the second target text or target conversion label corresponding to each speech frame, and then the multiple second target texts and target conversion labels are mapped into second text recognition results through a linear mapping layer, thereby realizing word-level recognition of speech samples.

[0151] In an exemplary embodiment, a speech recognition model to be trained is trained based on a first model loss and a second model loss to obtain a trained speech recognition model, including: weighting the first model loss and the second model loss to obtain a total loss; and updating model parameters of the speech recognition model to be trained according to the total loss to obtain a trained speech recognition model.

[0152] Among them, a first weighting coefficient of the first model loss and a second weighting coefficient of the second model loss are determined, the first model loss is multiplied by the first weighting coefficient, and the second model loss is multiplied by the second weighting coefficient. The two product values ​​are summed to obtain the total loss. The model parameters are continuously adjusted according to the total loss until the total loss meets the preset loss condition, and the updating of the model parameters is stopped to obtain a trained speech recognition model.

[0153] In this embodiment, the text recognition decoder and the conversion recognition decoder are trained through two kinds of losses. The text recognition decoder and the conversion recognition decoder respectively learn the correspondence between speech and text and the conversion between Chinese text and Pinyin text, which is beneficial to improving the recognition accuracy of Chinese-Pinyin mixed speech.

[0154] In an exemplary embodiment, the speech recognition model training method also includes: obtaining multiple mixed text data, synthesizing multiple mixed speech data based on the multiple mixed text data, and the mixed speech data and the mixed text data correspond one to one; using the speech recognition model to recognize the multiple mixed speech data respectively to obtain the recognition texts corresponding to the multiple mixed speech data; cleaning the multiple mixed text data according to the multiple recognition texts, and using the cleaned results as target mixed text data; determining the target mixed speech data corresponding to the target mixed text data from the multiple mixed speech data; and optimizing the speech recognition model based on the target mixed speech data and the target mixed text data to obtain the optimized speech recognition model.

[0155] Among them, since the mixed text sample is generated based on the text generation model, there is a lot of dirty data in the mixed text sample, and these dirty data will affect the recognition accuracy of the speech recognition model. In order to solve the impact of dirty data and further improve the recognition accuracy of the speech recognition model, the embodiment of the present application proposes to use a trained speech recognition model to clean the mixed text data generated by the text generation model, and then iteratively train the speech recognition model based on the cleaned results, so as to optimize the speech recognition accuracy of the speech recognition model and obtain an optimized speech recognition model.

[0156] Specifically, the server obtains a plurality of mixed text data generated by the text generation model, and synthesizes the plurality of mixed text data into a plurality of mixed voice data, and the synthesis method may adopt the method of “synthesizing voice samples based on mixed text samples” as described in step 202 .

[0157] The multiple mixed speech data are respectively input into the speech recognition model to obtain the recognition texts corresponding to the multiple mixed speech data, which include Chinese recognition text, Pinyin recognition text and switch recognition label.

[0158] Since the mixed speech data and the mixed text data correspond one to one, the multiple mixed text data are compared with the corresponding recognized texts, and the data inconsistent with the corresponding recognized texts in the multiple mixed text data are removed. The cleaned result is the target mixed text data.

[0159] The target mixed text data and the target mixed speech data are respectively the results of the speech recognition model cleaning the mixed text data and the mixed speech data. The speech recognition model is optimized and trained based on the cleaned results to obtain an optimized speech recognition model, which is beneficial to improving the accuracy of speech recognition.

[0160] In this embodiment, the mixed text data and mixed voice data are cleaned by a speech recognition model, and the speech recognition model is optimized and trained using the cleaned results to obtain an optimized speech recognition model, which is beneficial to improving the accuracy of speech recognition.

[0161] To explain the speech recognition model training method and effect in detail, the following is a most detailed example:

[0162] In order to realize the recognition ability of the speech recognition model for Chinese and Pinyin mixed speech, and solve the problem that Pinyin cannot be recognized, the speech recognition model training method proposed in the embodiment of the present application can be used for speech recognition in various application scenarios, and the teacher's class scenario is used as an example for explanation.

[0163] like Figure 3 The figure shows the overall process diagram of the speech recognition model training method. First, the preset pinyin-speech relationship is obtained, and a mixed text sample is constructed using a text generation model. Then, a speech sample is constructed based on speech synthesis and speech splicing technology. Subsequently, model training is performed based on the speech recognition model to be trained to obtain a speech recognition model. Furthermore, the trained speech recognition model is used to clean and filter the newly generated multiple mixed text data and mixed speech data. Finally, the speech recognition model is continued to be trained based on the cleaned and filtered data, and the speech recognition model is iteratively optimized in turn to obtain an optimized speech recognition model.

[0164] Among them, (1) the preset pinyin-speech relationship is obtained to construct a mixed text sample.

[0165] Obtain pinyin voice data and build a preset pinyin voice relationship. For the teacher's class scenario, use the text generation model to construct a mixed text sample. The server uses the text generation model to generate a mixed text sample. The mixed text sample contains Chinese text and pinyin text. Add a switching label at the conversion point between Chinese text and pinyin text to obtain a labeled text sample.

[0166] (2) Construct speech samples based on speech synthesis and speech splicing technology.

[0167] The server performs speech synthesis on the Chinese text in the mixed text sample to obtain Chinese speech, queries a preset pinyin-speech relationship to obtain the pinyin speech corresponding to the pinyin text, determines the arrangement order of the Chinese text and the pinyin text in the mixed text sample, and splices the Chinese speech and the pinyin speech according to the arrangement order to obtain a speech sample.

[0168] (3) Model training is performed based on the speech recognition model to be trained to obtain a speech recognition model.

[0169] The speech recognition model to be trained includes a text recognition decoder and a conversion recognition decoder. The text recognition decoder includes a connection time classification decoder and an attention decoder. Figure 4 The figure shows a schematic diagram of a training method for a speech recognition model in one embodiment, wherein the connection time classification decoder adopts a CTC decoder, the attention decoder includes a TransformerDecoder layer and a linear mapping layer, and the transformation recognition decoder includes a Transformer Decoder layer and a linear mapping layer.

[0170] Feature extraction is performed on the speech sample to obtain speech features, and the speech features are encoded to obtain a coding matrix. The coding matrix is ​​input into the connection time classification decoder to obtain a first text recognition result, and the coding matrix is ​​input into the attention decoder to obtain a second text recognition result. The text recognition result includes a first text recognition result and a second text recognition result. According to the first text recognition result and the mixed text sample, a first loss is obtained. According to the second text recognition result and the mixed text sample, a second loss is obtained. The first loss and the second loss are weighted summed to obtain a first model loss.

[0171] The encoding matrix is ​​input into the conversion recognition decoder in the speech recognition model to be trained to obtain the text conversion recognition result. Based on the text conversion recognition result and the labeled text sample, the second model loss is calculated. The first model loss and the second model loss are weighted and summed to obtain the total loss. The model parameters of the speech recognition model to be trained are updated according to the total loss to obtain the trained speech recognition model.

[0172] (4) Use the trained speech recognition model to clean and filter the newly generated multiple mixed text data and mixed speech data.

[0173] The server generates multiple mixed text data using a text generation model, and synthesizes multiple mixed voice data based on the multiple mixed text data, with the mixed voice data corresponding to the mixed text data one by one. The multiple mixed voice data are respectively recognized using a voice recognition model to obtain recognition texts corresponding to the multiple mixed voice data, and the multiple mixed text data are cleaned according to the multiple recognition texts, and the cleaned results are used as target mixed text data, and target mixed voice data corresponding to the target mixed text data is determined from the multiple mixed voice data.

[0174] (5) Iterative optimization of speech recognition model

[0175] Based on the target mixed speech data and the target mixed text data, the speech recognition model is optimized and trained to obtain an optimized speech recognition model.

[0176] The above-mentioned speech recognition model training method obtains a mixed text sample containing Chinese text and pinyin text, synthesizes a speech sample based on the mixed text sample, adds a switching label at the conversion point of Chinese text and pinyin text, and obtains a labeled text sample. Since the language sample and the labeled text sample are obtained based on the mixed text sample, the data volume of the speech sample is expanded, which is conducive to improving the speech recognition accuracy of the speech recognition model. The encoding matrix is ​​obtained by feature extraction and encoding of the speech sample, and the encoding matrix is ​​respectively input into the text recognition decoder and the conversion recognition decoder in the speech recognition model to be trained to obtain the text recognition result and the text conversion recognition result. Based on the text recognition result and the mixed text sample, the first model loss is calculated, and based on the text conversion recognition result and the labeled text sample, the second model loss is calculated, so that the speech recognition model to be trained is trained using the two model losses to obtain a trained speech recognition model. This speech recognition model training method uses two decoders to decode the encoding matrix, and the two decoders respectively learn the corresponding relationship between speech and text and the conversion of Chinese text and pinyin text, which is conducive to improving the recognition accuracy of Chinese-pinyin mixed speech. At the same time, in order to solve the problem of missing Chinese-Pinyin mixed speech data, a training data generation method based on text generation technology and speech synthesis technology is proposed. Based on the conversion recognition decoder and the Chinese-Pinyin switching label, the problem of high semantic similarity between Chinese characters and Pinyin is solved, which is conducive to improving the recognition accuracy of the speech recognition model for Chinese-Pinyin mixed speech.

[0177] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0178] Based on the same inventive concept, the embodiment of the present application also provides a speech recognition model training device for implementing the speech recognition model training method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more speech recognition model training device embodiments provided below can refer to the limitations on the speech recognition model training method above, and will not be repeated here.

[0179] In an exemplary embodiment, Figure 5 As shown, a speech recognition model training device 500 is provided, comprising: a first acquisition module 510, a synthesis module 520, a feature processing module 530, a first decoding module 540, a second decoding module 550 and a training module 560, wherein:

[0180] A first acquisition module 510 is used to acquire a mixed text sample, where the mixed text sample includes Chinese text and Pinyin text;

[0181] A synthesis module 520 is used to synthesize a speech sample based on the mixed text sample, and add a switching label at the conversion point between the Chinese text and the pinyin text to obtain a labeled text sample;

[0182] The feature processing module 530 is used to extract features from the speech sample to obtain speech features, encode the speech features to obtain a coding matrix;

[0183] A first decoding module 540 is used to input the encoding matrix into a text recognition decoder in the speech recognition model to be trained to obtain a text recognition result, and calculate a first model loss based on the text recognition result and the mixed text sample;

[0184] A second decoding module 550 is used to input the encoding matrix into a conversion recognition decoder in the speech recognition model to be trained to obtain a text conversion recognition result, and calculate a second model loss based on the text conversion recognition result and the labeled text sample;

[0185] The training module 560 is used to train the speech recognition model to be trained based on the first model loss and the second model loss to obtain a trained speech recognition model.

[0186] The speech recognition model training device obtains a mixed text sample containing Chinese text and pinyin text, synthesizes a speech sample based on the mixed text sample, adds a switching label at the conversion point of Chinese text and pinyin text, and obtains a labeled text sample. Since the language sample and the labeled text sample are obtained based on the mixed text sample, the data volume of the speech sample is expanded, which is conducive to improving the speech recognition accuracy of the speech recognition model. The encoding matrix is ​​obtained by feature extraction and encoding of the speech sample, and the encoding matrix is ​​respectively input into the text recognition decoder and the conversion recognition decoder in the speech recognition model to be trained to obtain the text recognition result and the text conversion recognition result. Based on the text recognition result and the mixed text sample, the first model loss is calculated, and based on the text conversion recognition result and the labeled text sample, the second model loss is calculated, so that the speech recognition model to be trained is trained using the two model losses to obtain a trained speech recognition model. This speech recognition model training method uses two decoders to decode the encoding matrix, and the two decoders respectively learn the corresponding relationship between speech and text and the conversion of Chinese text and pinyin text, which is conducive to improving the recognition accuracy of Chinese and pinyin mixed speech.

[0187] In one embodiment, based on the mixed text sample, the speech sample is synthesized, and the synthesis module 520 is also used to: perform speech synthesis on the Chinese text in the mixed text sample to obtain Chinese speech; query the preset pinyin-speech relationship to obtain the pinyin speech corresponding to the pinyin text; determine the arrangement order of the Chinese text and the pinyin text in the mixed text sample, and splice the Chinese speech and the pinyin speech according to the arrangement order to obtain a speech sample.

[0188] In one embodiment, the text recognition decoder includes a connection time classification decoder and an attention decoder; the encoding matrix is ​​input into the text recognition decoder in the speech recognition model to be trained to obtain a text recognition result. The first decoding module 540 is also used to: input the encoding matrix into the connection time classification decoder to obtain a first text recognition result; input the encoding matrix into the attention decoder to obtain a second text recognition result. The text recognition results include a first text recognition result and a second text recognition result.

[0189] In one embodiment, a speech sample includes multiple speech frames, and a coding matrix includes coding vectors corresponding to each of the multiple speech frames; the coding matrix is ​​input into a connected temporal classification decoder to obtain a first text recognition result, including: extracting coding vectors corresponding to each of the speech frames from the coding matrix; for each coding vector, decoding the coding vector through a connected temporal classification decoder to obtain the probability that the speech frames corresponding to the coding vector are multiple preset characters; determining a target text whose probability meets a preset probability condition from the multiple preset characters; performing deduplication processing on the target text corresponding to each of the multiple coding vectors, and using the deduplication result as the first text recognition result.

[0190] In one embodiment, the speech sample includes multiple speech frames, and the encoding matrix includes encoding vectors corresponding to each of the multiple speech frames; the encoding matrix is ​​input into the attention decoder to obtain a second text recognition result, and the first decoding module 540 is also used to: extract the encoding vectors corresponding to each of the speech frames from the encoding matrix; determine the first target text corresponding to each encoding vector from the multiple first texts contained in the first dictionary; and perform linear mapping on the multiple first target texts to obtain a second text recognition result.

[0191] In one embodiment, a first model loss is calculated based on the text recognition result and the mixed text sample, and the first decoding module 540 is also used to: obtain a first loss based on the first text recognition result and the mixed text sample; obtain a second loss based on the second text recognition result and the mixed text sample; and perform weighted summation of the first loss and the second loss to obtain the first model loss.

[0192] In one embodiment, the speech sample includes multiple speech frames, and the encoding matrix includes encoding vectors corresponding to each of the multiple speech frames; the encoding matrix is ​​input into the conversion recognition decoder in the speech recognition model to be trained to obtain a text conversion recognition result, and the second decoding module 550 is also used to: extract the encoding vector corresponding to each speech frame from the encoding matrix; determine the second target text or target conversion label corresponding to each encoding vector among the multiple second texts and multiple conversion labels contained in the second dictionary; linearly map the multiple second target texts and target conversion labels to obtain a text conversion recognition result.

[0193] In one embodiment, the speech recognition model to be trained is trained based on the first model loss and the second model loss to obtain a trained speech recognition model. The training module 560 is also used to: weight the first model loss and the second model loss to obtain a total loss; update the model parameters of the speech recognition model to be trained according to the total loss to obtain a trained speech recognition model.

[0194] In one embodiment, the speech recognition training device 500 also includes an optimization model, and the optimization module is used to: obtain multiple mixed text data, synthesize multiple mixed speech data based on the multiple mixed text data, and the mixed speech data and the mixed text data correspond one to one; use the speech recognition model to recognize the multiple mixed speech data respectively to obtain the recognition texts corresponding to the multiple mixed speech data; clean the multiple mixed text data according to the multiple recognition texts, and use the cleaned results as target mixed text data; determine the target mixed speech data corresponding to the target mixed text data from the multiple mixed speech data; optimize the speech recognition model based on the target mixed speech data and the target mixed text data to obtain the optimized speech recognition model.

[0195] Each module in the above-mentioned speech recognition model training device can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0196] In an exemplary embodiment, Figure 6 As shown, a speech recognition method is provided, wherein the trained speech recognition model is obtained by using the training method of the speech recognition model in each of the above embodiments. The speech recognition method comprises:

[0197] Step 602, obtaining a mixed speech to be recognized; extracting features of the mixed speech to be recognized to obtain features of the speech to be recognized, encoding the features of the speech to be recognized to obtain an encoding matrix to be recognized;

[0198] Step 604, performing text recognition on the coding matrix to be recognized by using the text recognition decoder in the trained speech recognition model to obtain a text recognition result corresponding to the coding matrix to be recognized;

[0199] Step 606, performing text conversion recognition on the coding matrix to be recognized by using the conversion recognition decoder in the trained speech recognition model to obtain a text conversion recognition result corresponding to the coding matrix to be recognized;

[0200] Step 608: Determine a final text recognition result based on the text recognition result corresponding to the coding matrix to be recognized and the text conversion recognition result corresponding to the coding matrix to be recognized.

[0201] The speech recognition method can be applied to a variety of speech recognition scenarios, and the application scenario of a teacher teaching is used as an example for explanation. The speech recognition method can be executed by a terminal or a server.

[0202] The mixed speech to be recognized is the mixed speech data collected in the teacher's class application scenario. The mixed speech to be recognized may include Chinese or Pinyin speech. The trained speech recognition model includes a text recognition decoder and a conversion recognition decoder. The text recognition decoder includes a connection time series classification decoder and an attention decoder.

[0203] The coding matrix to be recognized corresponding to the mixed speech to be recognized is input into the trained speech recognition model, and the temporal classification decoder is connected to perform the first text recognition on the coding matrix to be recognized to obtain the first text recognition result corresponding to the coding matrix to be recognized. The attention decoder performs the second text recognition on the coding matrix to be recognized to obtain the second text recognition result corresponding to the coding matrix to be recognized. The text conversion recognition result corresponding to the coding matrix to be recognized includes the first text recognition result and the second text recognition result.

[0204] The conversion recognition decoder performs text conversion recognition on the coding matrix to be recognized, and obtains a text conversion recognition result corresponding to the coding matrix to be recognized.

[0205] The first text recognition result includes multiple first character recognition data and multiple first recognition probability vectors. The first recognition probability vector includes multiple first recognition probabilities. Each first recognition probability is used to characterize the probability that the corresponding first character recognition data is predicted to be multiple preset characters.

[0206] The second text recognition result includes multiple second character recognition data and multiple second recognition probability vectors. The second recognition probability vector includes multiple second recognition probabilities. Each second recognition probability is used to characterize the probability that the corresponding second character recognition data is predicted to be multiple first characters in the first dictionary.

[0207] The text conversion recognition result includes a plurality of third character recognition data, a plurality of recognition labels and a plurality of third recognition probability vectors. The third recognition probability vector includes a plurality of third recognition probabilities, each of which is used to characterize the probability that the corresponding third character recognition data or recognition label is predicted to be a plurality of second characters or conversion labels in the second dictionary.

[0208] A plurality of first recognition probability vectors, a plurality of second recognition probability vectors, and a plurality of third recognition probability vectors are weighted and summed to obtain a plurality of weighted vectors. Each weighted vector includes a plurality of weighted probabilities, and for each weighted vector, a target weighted probability that satisfies a preset probability condition is determined from the plurality of weighted probabilities. The second character corresponding to the target weighted probability is used as the target character, and the target characters corresponding to the plurality of weighted vectors constitute a text conversion recognition result.

[0209] The above-mentioned speech recognition method, through the text recognition decoder and conversion recognition decoder in the trained speech recognition model, respectively performs text recognition and text conversion recognition on the coding matrix to be recognized corresponding to the mixed speech to be recognized, obtains the text recognition result and text conversion recognition result corresponding to the coding matrix to be recognized, and determines the final text recognition result based on the text recognition result corresponding to the coding matrix to be recognized and the text conversion recognition result corresponding to the coding matrix to be recognized. Since the two decoders included in the trained speech recognition model have respectively learned the correspondence between speech and text and the conversion between Chinese text and pinyin text, two decoders are used to perform text recognition and text conversion recognition on the coding matrix to be recognized, and the final text recognition result is obtained by combining the two output results, which is conducive to improving the recognition accuracy of Chinese and pinyin mixed speech.

[0210] In an exemplary embodiment, Figure 7 As shown, a speech recognition device 700 is provided, comprising: a second acquisition module 720, a first recognition module 740, a second recognition module 760 and a determination module 780, wherein:

[0211] The second acquisition module 720 is used to acquire the mixed speech to be recognized; extract the features of the mixed speech to be recognized to obtain the features of the speech to be recognized, and encode the features of the speech to be recognized to obtain the encoding matrix to be recognized;

[0212] The first recognition module 740 is used to perform text conversion recognition on the coding matrix to be recognized through the conversion recognition decoder in the trained speech recognition model to obtain a text conversion recognition result corresponding to the coding matrix to be recognized;

[0213] The second recognition module 760 is used to perform text conversion recognition on the coding matrix to be recognized through the conversion recognition decoder in the trained speech recognition model to obtain the text conversion recognition result corresponding to the coding matrix to be recognized;

[0214] The determination module 780 is used to determine the final result of text recognition based on the text recognition result corresponding to the coding matrix to be recognized and the text conversion recognition result corresponding to the coding matrix to be recognized.

[0215] The above-mentioned speech recognition device, through the text recognition decoder and conversion recognition decoder in the trained speech recognition model, respectively performs text recognition and text conversion recognition on the coding matrix to be recognized corresponding to the mixed speech to be recognized, obtains the text recognition result and text conversion recognition result corresponding to the coding matrix to be recognized, and determines the final text recognition result based on the text recognition result corresponding to the coding matrix to be recognized and the text conversion recognition result corresponding to the coding matrix to be recognized. Since the two decoders included in the trained speech recognition model have respectively learned the correspondence between speech and text and the conversion between Chinese text and pinyin text, two decoders are used to perform text recognition and text conversion recognition on the coding matrix to be recognized, and the final text recognition result is obtained by combining the two output results, which is conducive to improving the recognition accuracy of Chinese and pinyin mixed speech.

[0216] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store mixed text samples, voice samples, labeled text samples or mixed voices to be recognized, etc. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a speech recognition model training method and a speech recognition method are implemented.

[0217] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0218] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0219] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0220] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0221] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0222] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0223] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0224] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A speech recognition model training method, characterized in that: The method comprises: Acquire a mixed text sample, wherein the mixed text sample includes Chinese character text and pinyin text; Synthesize a speech sample based on the mixed text sample, add a switching label at the conversion point between the Chinese character text and the pinyin text, and obtain a labeled text sample; Extracting features from the speech sample to obtain speech features, and encoding the speech features to obtain an encoding matrix; Input the encoding matrix into a text recognition decoder in a speech recognition model to be trained to obtain a text recognition result, and calculate a first model loss based on the text recognition result and the mixed text sample; Input the encoding matrix into a conversion recognition decoder in a speech recognition model to be trained to obtain a text conversion recognition result, and calculate a second model loss based on the text conversion recognition result and the labeled text sample; Based on the first model loss and the second model loss, the speech recognition model to be trained is trained to obtain a trained speech recognition model.

2. The method according to claim 1, characterized in that The synthesizing a speech sample based on the mixed text sample comprises: Performing speech synthesis on the Chinese text in the mixed text sample to obtain Chinese speech; Querying a preset pinyin-speech relationship to obtain the pinyin speech corresponding to the pinyin text; Determine the arrangement order of the Chinese character text and the Pinyin text in the mixed text sample, and according to the arrangement order, splice the Chinese speech and the Pinyin speech to obtain a speech sample.

3. The method according to claim 1, characterized in that The text recognition decoder includes a connection time classification decoder and an attention decoder; the encoding matrix is ​​input into the text recognition decoder in the speech recognition model to be trained to obtain a text recognition result, including: Inputting the encoding matrix into the connection temporal classification decoder to obtain a first text recognition result; The encoding matrix is ​​input into the attention decoder to obtain a second text recognition result, where the text recognition result includes the first text recognition result and the second text recognition result.

4. The method according to claim 3, characterized in that: The speech sample includes a plurality of speech frames, the coding matrix includes coding vectors corresponding to the plurality of speech frames respectively; the coding matrix is ​​input into the connection timing classification decoder to obtain a first text recognition result, comprising: Extracting the encoding vectors corresponding to the respective speech frames from the encoding matrix; For each coding vector, the coding vector is decoded by the connection timing classification decoder to obtain the probability that the speech frame corresponding to the coding vector is a plurality of preset characters; and a target text whose probability meets a preset probability condition is determined from the plurality of preset characters; Deduplication processing is performed on the target texts corresponding to the multiple encoding vectors, and the deduplication result is used as the first text recognition result.

5. The method according to claim 3, characterized in that: The speech sample includes a plurality of speech frames, the encoding matrix includes encoding vectors corresponding to the plurality of speech frames respectively; and inputting the encoding matrix into the attention decoder to obtain a second text recognition result includes: Extracting the encoding vectors corresponding to the respective speech frames from the encoding matrix; Determine a first target character corresponding to each encoding vector from among a plurality of first characters included in the first dictionary; Linear mapping is performed on the plurality of first target characters to obtain a second text recognition result.

6. The method according to claim 3, characterized in that The calculating a first model loss based on the text recognition result and the mixed text sample includes: Obtaining a first loss according to the first text recognition result and the mixed text sample; Obtaining a second loss according to the second text recognition result and the mixed text sample; A weighted sum is performed on the first loss and the second loss to obtain a first model loss.

7. The method according to claim 1, characterized in that The speech sample includes a plurality of speech frames, the coding matrix includes coding vectors corresponding to the plurality of speech frames respectively; the coding matrix is ​​input into a conversion recognition decoder in a speech recognition model to be trained to obtain a text conversion recognition result, including: Extracting the encoding vectors corresponding to the respective speech frames from the encoding matrix; Determine, from among the plurality of second characters and the plurality of conversion labels contained in the second dictionary, a second target character or a target conversion label corresponding to each encoding vector; Linear mapping is performed on multiple second target characters and target conversion labels to obtain a text conversion recognition result.

8. The method according to claim 1, characterized in that The step of training the speech recognition model to be trained based on the first model loss and the second model loss to obtain a trained speech recognition model includes: Weighting the first model loss and the second model loss to obtain a total loss; The model parameters of the speech recognition model to be trained are updated according to the total loss to obtain a trained speech recognition model.

9. The method according to claim 1, characterized in that: The method further comprises: Acquire a plurality of mixed text data, and synthesize a plurality of mixed voice data based on the plurality of mixed text data, wherein the mixed voice data and the mixed text data correspond to each other one by one; Using the speech recognition model to recognize multiple mixed speech data respectively, and obtain recognition texts corresponding to each of the multiple mixed speech data; Cleaning multiple mixed text data according to multiple recognition texts, and using the cleaned results as target mixed text data; Determine target mixed voice data corresponding to the target mixed text data from a plurality of mixed voice data; Based on the target mixed speech data and the target mixed text data, the speech recognition model is optimized and trained to obtain an optimized speech recognition model.

10. A speech recognition method, characterized in that: The method comprises: Acquire a mixed speech to be recognized; extract features of the mixed speech to be recognized to obtain features of the speech to be recognized, encode the features of the speech to be recognized to obtain an encoding matrix to be recognized; Performing text recognition on the coding matrix to be recognized by using a text recognition decoder in the trained speech recognition model to obtain a text recognition result corresponding to the coding matrix to be recognized; Performing text conversion recognition on the coding matrix to be recognized by using the conversion recognition decoder in the trained speech recognition model to obtain a text conversion recognition result corresponding to the coding matrix to be recognized; Based on the text recognition result corresponding to the coding matrix to be recognized and the text conversion recognition result corresponding to the coding matrix to be recognized, a final text recognition result is determined.

11. A speech recognition model training device, characterized in that: The device comprises: A first acquisition module, used to acquire a mixed text sample, wherein the mixed text sample includes Chinese character text and pinyin text; A synthesis module, used for synthesizing a speech sample based on the mixed text sample, adding a switching label at the conversion point between the Chinese character text and the pinyin text, and obtaining a labeled text sample; A feature processing module is used to extract features from the speech sample to obtain speech features, and encode the speech features to obtain a coding matrix; A first decoding module, used for inputting the encoding matrix into a text recognition decoder in a speech recognition model to be trained to obtain a text recognition result, and calculating a first model loss based on the text recognition result and the mixed text sample; A second decoding module is used to input the encoding matrix into a conversion recognition decoder in a speech recognition model to be trained to obtain a text conversion recognition result, and calculate a second model loss based on the text conversion recognition result and the labeled text sample; A training module is used to train the speech recognition model to be trained based on the first model loss and the second model loss to obtain a trained speech recognition model.

12. A speech recognition device, characterized in that: The method comprises: The second acquisition module is used to acquire the mixed speech to be recognized; extract the features of the mixed speech to be recognized to obtain the features of the speech to be recognized, and encode the features of the speech to be recognized to obtain the encoding matrix to be recognized; A first recognition module is used to perform text recognition on the coding matrix to be recognized through a text recognition decoder in a trained speech recognition model to obtain a text recognition result corresponding to the coding matrix to be recognized; The second recognition module is used to perform text conversion recognition on the coding matrix to be recognized through the conversion recognition decoder in the trained speech recognition model to obtain the text conversion recognition result corresponding to the coding matrix to be recognized; The determination module is used to determine the final result of text recognition based on the text recognition result corresponding to the coding matrix to be recognized and the text conversion recognition result corresponding to the coding matrix to be recognized.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images