Speech recognition method and apparatus, server, and computer-readable storage medium

Abstract

Embodiments of the present application provide a speech recognition method and device, a server and a computer readable storage medium. The method obtains a speech feature sequence of a to-be-recognized speech, and then performs recognition on the speech feature sequence of the to-be-recognized speech to obtain a to-be-tested tonal phoneme feature sequence, a to-be-tested non-tonal syllable feature sequence and a to-be-tested tone feature sequence. Then, the to-be-tested tonal phoneme feature sequence, the to-be-tested non-tonal syllable feature sequence and the to-be-tested tone feature sequence are fused to obtain a fused to-be-tested tonal syllable feature sequence. Then, text recognition is performed on the to-be-tested tonal syllable feature sequence to obtain a target text corresponding to the to-be-recognized speech. The speech feature sequence is extracted from three angles of tonal phoneme, non-tonal syllable and tone. The three angle features obtained by extraction are fused to obtain a tonal syllable feature sequence with stronger expression capability. The target text obtained by performing text recognition on the tonal syllable feature sequence is more accurate.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a speech recognition and classification method, device, server, and computer-readable storage medium. Background Technology

[0002] With the development of technology, AI (Artificial Intelligence) technology is being applied more and more widely. For example, AI technology has been applied in various fields such as autonomous driving, image processing, and speech recognition. Neural network models play an important role in speech recognition based on artificial intelligence technology.

[0003] Traditional speech recognition methods rely solely on Chinese acoustic models with toned phonemes. However, due to the limited frequency of certain pronunciations and tone combinations in the training corpus, the training of individual phonemes is incomplete. Furthermore, Chinese exhibits tone sandhi rules, meaning some pronunciations differ from individual character pronunciations. Dictionaries often fail to cover all phoneme pronunciation methods. Consequently, the speech features extracted from the trained Chinese acoustic model with toned phonemes are not comprehensive enough, leading to low accuracy in the final speech recognition results. Summary of the Invention

[0004] This application provides a speech recognition method, apparatus, server, and computer-readable storage medium to improve the accuracy of speech recognition.

[0005] On the one hand, this application provides a speech recognition method, including:

[0006] Obtain the speech feature sequence of the speech to be recognized;

[0007] The speech feature sequence of the speech to be identified is identified to obtain the toned phoneme feature sequence, the untoned syllable feature sequence, and the tone feature sequence corresponding to the speech to be identified.

[0008] The to-be-tested tone-phoneme feature sequence, the to-be-tested toneless syllable feature sequence, and the to-be-tested tone feature sequence are fused to obtain the fused to-be-tested tone-syllable feature sequence.

[0009] Text recognition is performed on the to-be-tested tone syllable feature sequence to obtain the target text corresponding to the to-be-recognized speech.

[0010] In one possible implementation of this application, the step of recognizing the speech feature sequence of the speech to be recognized to obtain the corresponding tonal phoneme feature sequence, the tonal syllable feature sequence, and the tone feature sequence includes:

[0011] The speech feature sequence of the speech to be identified is identified by a trained tonal phoneme recognition model, a trained untone syllable recognition model, and a trained tone recognition model, respectively obtaining the tonal phoneme feature sequence, the untone syllable feature sequence, and the tone feature sequence corresponding to the speech to be identified.

[0012] In one possible implementation of this application, before the step of recognizing the speech feature sequence of the speech to be recognized using a trained tonal phoneme recognition model, a trained untoned syllable recognition model, and a trained tone recognition model to obtain the test tonal phoneme feature sequence, the test untoned syllable feature sequence, and the test tone feature sequence corresponding to the speech to be recognized, the following steps are included:

[0013] Obtain the sample speech feature sequence and the target tone phoneme feature sequence corresponding to the sample speech feature sequence;

[0014] Based on the sample speech feature sequence and the target tonal phoneme feature sequence, the tonal phoneme recognition model, the untone syllable recognition model, and the tone recognition model are trained respectively to obtain the trained tonal phoneme recognition model, the trained untone syllable recognition model, and the trained tone recognition model.

[0015] In one possible implementation of this application, the step of obtaining the sample speech feature sequence and the target tone-modulated phoneme feature sequence corresponding to the sample speech feature sequence includes:

[0016] The sample speech feature sequence is forcibly aligned on the time axis to obtain the target tone-modulated phoneme feature sequence.

[0017] In one possible implementation of this application, the target toned phoneme feature sequence includes index number information; the step of training the toned phoneme recognition model, the untoned syllable recognition model, and the tone recognition model based on the sample speech feature sequence and the target toned phoneme feature sequence to obtain the trained toned phoneme recognition model, the untoned syllable recognition model, and the tone recognition model respectively includes:

[0018] Based on the index number information of each target tone phoneme, the target toneless syllable feature sequence and the target tone feature sequence are determined according to the target tone phoneme feature sequence.

[0019] The sample speech feature sequences are respectively input into the toned phoneme recognition model, the untoned syllable recognition model, and the tone recognition model for processing to obtain the real toned phoneme feature sequences, the real untoned syllable feature sequences, and the real tone feature sequences.

[0020] Based on the preset first loss value, first loss function, the real tonal phoneme feature sequence and the target tonal phoneme feature sequence, the tonal phoneme recognition model is iteratively trained until the tonal phoneme recognition model converges, and the trained tonal phoneme recognition model is obtained.

[0021] Based on the preset second loss value, second loss function, the real untone syllable feature sequence, and the target untone syllable feature sequence, the untone syllable recognition model is iteratively trained until the untone syllable recognition model converges, thus obtaining the trained untone syllable recognition model.

[0022] The tone recognition model is iteratively trained based on the preset third loss value, third loss function, the real tone feature sequence, and the target tone feature sequence until the tone recognition model converges, thus obtaining the trained tone recognition model.

[0023] In one possible implementation of this application, the step of fusing the to-be-tested tone-phoneme feature sequence, the to-be-tested toneless syllable feature sequence, and the to-be-tested tone feature sequence to obtain a fused to-be-tested tone-phoneme feature sequence includes:

[0024] Call the trained fusion model;

[0025] The trained fusion model is used to fuse the tonal phoneme feature sequence, the untone syllable feature sequence, and the tone feature sequence to obtain the fused tonal syllable feature sequence.

[0026] In one possible implementation of this application, before the step of invoking the trained fusion model, the method further includes:

[0027] Based on the index number information of each target tone phoneme, the target tone syllable feature sequence is determined according to the target tone phoneme feature sequence;

[0028] The real tonal phoneme feature sequence, the real untone syllable feature sequence, and the real tone feature sequence are input into the fusion model for processing to obtain the real tonal syllable feature sequence.

[0029] The fusion model is iteratively trained based on the preset fourth loss value, fourth loss function, the real tonal syllable feature sequence, and the target tonal syllable feature sequence until the fusion model converges, thus obtaining the trained fusion model.

[0030] On the one hand, this application provides a voice recognition device, including:

[0031] The acquisition unit is used to acquire the speech feature sequence of the speech to be recognized;

[0032] The first recognition unit is used to recognize the speech feature sequence of the speech to be recognized, and obtain the tonal phoneme feature sequence, the untone syllable feature sequence, and the tone feature sequence corresponding to the speech to be recognized.

[0033] The fusion unit is used to fuse the to-be-tested tone-phoneme feature sequence, the to-be-tested toneless syllable feature sequence, and the to-be-tested tone feature sequence to obtain the fused to-be-tested tone-phoneme feature sequence.

[0034] The second recognition unit is used to perform text recognition on the to-be-tested tone syllable feature sequence to obtain the target text corresponding to the speech to be recognized.

[0035] The first identification unit is specifically:

[0036] This is used to identify the speech feature sequence of the speech to be identified by using a trained tonal phoneme recognition model, a trained untoned syllable recognition model, and a trained tone recognition model, respectively obtaining the tonal phoneme feature sequence, the untoned syllable feature sequence, and the tone feature sequence corresponding to the speech to be identified.

[0037] The fusion unit is specifically:

[0038] Used to invoke the trained fusion model;

[0039] The fusion model is used to fuse the toned phoneme feature sequence, the untone syllable feature sequence, and the tone feature sequence to be tested, thereby obtaining the fused toned syllable feature sequence.

[0040] On one hand, embodiments of this application provide a server, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the speech recognition method described above.

[0041] On the one hand, this application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps in the above-described speech recognition method.

[0042] This application provides a speech recognition method, device, server, and computer-readable storage medium. This application extracts speech feature sequences from three perspectives: toned phonemes, untoned syllables, and tones, respectively obtaining toned phoneme feature sequences, untoned syllable feature sequences, and tones. These toned phoneme feature sequences, untoned syllable feature sequences, and tones are then fused to obtain a more expressive fused toned syllable feature sequence. Finally, text recognition is performed on the more expressive toned syllable feature sequence to obtain more accurate target text. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of a scenario for the speech recognition system provided in an embodiment of this application.

[0045] Figure 2 This is a flowchart illustrating the speech recognition method provided in an embodiment of this application.

[0046] Figure 3a The first schematic diagram provided for the embodiment of this application is a feature sequence fusion process.

[0047] Figure 3b The following is a second schematic diagram of the feature sequence fusion process provided for the embodiments of this application.

[0048] Figure 4 This is a schematic diagram of the structure of the speech recognition device provided in the embodiments of this application.

[0049] Figure 5 This is a schematic diagram of the server structure provided in an embodiment of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] In the description of this application, "multiple" means two or more, unless otherwise expressly and specifically defined.

[0052] In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use this application. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be made without using these specific details. In other instances, well-known structures and processes are not described in detail to avoid obscuring the description of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0053] It should be noted that the method in this application embodiment is executed in a computer device, such as a server. The processing objects of each computer device exist in the form of data or information, such as time, which is essentially time information, so that electronic devices can process it. The specifics will not be elaborated here.

[0054] Please see Figure 1 , Figure 1 This is a schematic diagram of a scenario for the speech recognition system provided in an embodiment of this application; please refer to... Figure 1 Taking the integration of a speech recognition device into a server as an example, firstly, the server 10 acquires the speech feature sequence of the speech to be recognized and recognizes the speech feature sequence to obtain the corresponding tonal phoneme feature sequence, untone syllable feature sequence, and tone feature sequence. The server 10 then fuses these three sequences to obtain a fused tonal syllable feature sequence. Finally, text recognition is performed on the tonal syllable feature sequence to obtain the target text corresponding to the speech to be recognized. This application extracts speech feature sequences from three perspectives: tonal phonemes, untone syllables, and tones. The extracted features from these three perspectives are then fused to obtain a more expressive fused tonal syllable feature sequence. Text recognition of this tonal syllable feature sequence yields more accurate target text.

[0055] It should be noted that, Figure 1 The schematic diagram of the speech recognition system shown is merely an example. The server described in this application embodiment is for the purpose of more clearly illustrating the technical solutions of this application embodiment and does not constitute a limitation on the technical solutions provided in this application embodiment. As those skilled in the art will know, with the evolution of systems and the emergence of new business scenarios, the technical solutions provided in this application embodiment are also applicable to similar technical problems. Detailed descriptions are provided below. It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of embodiments.

[0056] Please see Figure 2 , Figure 2 This is a first flowchart illustrating a speech recognition method provided in an embodiment of this application, applied to the aforementioned server. The speech recognition method includes:

[0057] Step 201: Obtain the speech feature sequence of the speech to be recognized.

[0058] In this application, the speech feature sequence of the speech to be recognized is a feature sequence that can express the speech to be recognized. The purpose is to organize the complex and redundant speech data into a specific file format and process it into a binary feature vector form that can be recognized by a computer.

[0059] In one embodiment, the step of obtaining the speech feature sequence of the speech to be recognized includes: preprocessing the speech to be recognized to obtain processed speech to be recognized; and extracting features from the processed speech to obtain the speech feature sequence of the speech to be recognized.

[0060] Since the speech to be recognized is easily affected by the speaker's speaking speed and environmental noise, resulting in a low recognition rate, which in turn affects the subsequent speech recognition results, it is necessary to preprocess the speech to be recognized to remove noise and enhance the speech to improve the recognition rate.

[0061] The methods for preprocessing the speech to be recognized include one or more preprocessing techniques such as pre-emphasis, framing, windowing, and endpoint detection. The methods for extracting features from the processed speech to obtain a speech feature sequence include: extracting speech features based on Mel-Frequency Cepstral Coefficients (MFCC) from the processed speech to obtain a speech feature sequence.

[0062] Step 202: Recognize the speech feature sequence of the speech to be recognized, and obtain the tonal phoneme feature sequence, the untoned syllable feature sequence, and the tone feature sequence respectively.

[0063] Since the speech feature sequence of the speech to be recognized includes features in three directions, namely, toned phonemes, untoned syllables, and tones, it is necessary to recognize the speech feature sequence of the speech to be recognized from three angles to obtain the toned phoneme feature sequence, the untoned syllable feature sequence, and the tone feature sequence respectively.

[0064] Among them, the to-be-tested tone phoneme feature sequence, the to-be-tested toneless syllable feature sequence, and the to-be-tested tone feature sequence can be obtained manually, such as through human experience, or through adaptive methods, such as using a neural network model. The extraction method is not limited here and can be selected according to the actual situation.

[0065] In one embodiment, the step of recognizing the speech feature sequence of the speech to be recognized to obtain the toned phoneme feature sequence, the untoned syllable feature sequence, and the tone feature sequence corresponding to the speech to be recognized includes: recognizing the speech feature sequence of the speech to be recognized through a trained toned phoneme recognition model, a trained untoned syllable recognition model, and a trained tone recognition model to obtain the toned phoneme feature sequence, the untoned syllable feature sequence, and the tone feature sequence corresponding to the speech to be recognized.

[0066] Before calling the trained tonal phoneme recognition model, untone syllable recognition model, and tone recognition model, it is necessary to train the tonal phoneme recognition model, untone syllable recognition model, and tone recognition model.

[0067] In one embodiment, before the steps of recognizing the speech feature sequence of the speech to be recognized by the trained tonal phoneme recognition model, the trained untone syllable recognition model, and the trained tone recognition model to obtain the to-be-recognized tonal phoneme feature sequence, the to-be-recognized untone syllable feature sequence, and the to-be-recognized tone feature sequence, respectively, the method includes: obtaining the sample speech feature sequence and the target tonal phoneme feature sequence corresponding to the sample speech feature sequence; and training the tonal phoneme recognition model, the untone syllable recognition model, and the tone recognition model based on the sample speech feature sequence and the target tonal phoneme feature sequence to obtain the trained tonal phoneme recognition model, the untone syllable recognition model, and the tone recognition model, respectively.

[0068] The specific method for obtaining the sample speech feature sequence and the target tone-enhanced phoneme feature sequence corresponding to the sample speech feature sequence is as follows: the sample speech feature sequence is forcibly aligned on the time axis to obtain the target tone-enhanced phoneme feature sequence.

[0069] The sample speech feature sequence is input into the trained triphone (GMM) model and the time axis is aligned. The trained triphone (GMM) model is based on a phoneme-level model set that is independent of initial consonants and merges vowels and tones. The tones are labeled as 1 to 5. The purpose of aligning the time axis of the sample speech feature sequence is to ensure that each tone phoneme in the sample speech feature sequence has a corresponding time region.

[0070] For example, a sound that lasts 0.8 seconds is pronounced as "morning". First, the feature sequence corresponding to "morning" is input into the triphone GMM for time axis alignment. Assume that 0.8 seconds includes: 0-0.1 is silence (sil), 0.1-0.15 (sh), 0.15-0.4 (ang4), 0.4-0.43 (w), 0.43-0.7 (u3), and 0.7-0.8 is silence (sil). Assuming the index number is 0 and the frame is 10ms, the 0.8s speech has 80 frames. The alignment result only needs to be identified as the sequence of the index number. The toned phonemes are: sh(21)ang4(4)w(23)u3(19)01. The sequence dimension is 40, and the corresponding target toned phoneme feature sequence is: 0 0 0 0 0 00 0 0 21 21 21 21 21 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 2323 23 19 ... 1919 19 19 0 0 0 0 0 0 0 0 0 0, where, in the feature representation method of phonemes or syllables, the number in parentheses after the phoneme or syllable represents the index number. For example, for a set of toned phonemes of size 40, if the index number of sh in it is 21, then for a 40-dimensional vector of a certain frame, if it is labeled as sh, then the 21st dimension is 1, and the other dimensions are 0. For example, the phoneme sh can be represented as: 00000000000000000000100000000000000000000.

[0071] In the above 0.8s pronunciation, 0-0.1s (10 frames) is the silence sil (0), so 0 appears 10 times in a row. 0.1-0.15 is the initial sh, which has a length of 5 frames, so 21 appears 5 times in a row. And so on, to obtain the tone-based phoneme feature sequence.

[0072] The above steps yield the target tonal phoneme feature sequence. Next, the target tonal phoneme feature sequence needs to be used to train the tonal phoneme recognition model, the untone syllable recognition model, and the tone recognition model.

[0073] In one embodiment, the target tonal phoneme feature sequence includes index number information; the steps of training the tonal phoneme recognition model, the toneless syllable recognition model, and the tone recognition model based on the sample speech feature sequence and the target tonal phoneme feature sequence to obtain the trained tonal phoneme recognition model, toneless syllable recognition model, and tone recognition model respectively include: determining the target toneless syllable feature sequence and the target tone feature sequence based on the index number information of each target tonal phoneme and according to the target tonal phoneme feature sequence; inputting the sample speech feature sequence into the tonal phoneme recognition model, the toneless syllable recognition model, and the tone recognition model respectively for processing to obtain the real tonal phoneme feature sequence, the real toneless syllable feature sequence, and the real tone feature sequence. The training process involves iteratively training a tone-based phoneme recognition model using a preset first loss value, a first loss function, a true tone-based phoneme feature sequence, and a target tone-based phoneme feature sequence, until the model converges. Similarly, iteratively training a tone-free syllable recognition model using a preset second loss value, a second loss function, a true tone-free syllable feature sequence, and a target tone-free syllable feature sequence, until the model converges. Finally, iteratively training a tone recognition model using a preset third loss value, a third loss function, a true tone-based feature sequence, and a target tone-based feature sequence, until the tone recognition model converges.

[0074] The training of the toned phoneme recognition model, the untoned syllable recognition model, and the tone recognition model will be explained below.

[0075] The specific training process for the tone-controlled phoneme recognition model is as follows:

[0076] The sample speech feature sequence is input into the tone-controlled phoneme recognition model to obtain the real tone-controlled phoneme feature sequence. Then, based on the preset first loss value, first loss function, real tone-controlled phoneme feature sequence, and target tone-controlled phoneme feature sequence obtained in the above steps, the tone-controlled phoneme recognition model is iteratively trained until the tone-controlled phoneme recognition model converges, and the trained tone-controlled phoneme recognition model is obtained.

[0077] The training process for the toneless syllable recognition model is as follows:

[0078] Before training the untone syllable recognition model, it is necessary to determine the target untone syllable feature sequence based on the index number information of each target tone phoneme and the target tone phoneme feature sequence.

[0079] Herein, based on the toneless syllable feature sequence, the initial phoneme is directly mapped to the corresponding toneless syllable (Pinyin), the final phoneme is mapped to the toneless syllable (Pinyin) after removing the tone information, the non-pronunciation segment is set as SILENCE, and the final classification number is the number of toneless syllables plus one.

[0080] Taking the above content as an example, the toneless syllables corresponding to "shangwu (morning)" are: shang(768)wu(903), and the dimension of column 01 is 983, then the target toneless syllable feature sequence corresponding to "shangwu" is: 0 0 0 0 0 0 0 0 0 0 768 768 768 768 768 768 768 768 768 768 768 768 768 768 768 768 768 768 768 768 768 768 768 768 768 768 768 768 768 768 903 903 903 903 903 903 903 903 903 903 903 903 903 903 903 903 903 903 903 903 903 903 903 903 903 903 903 903 903 903 0 0 0 0 0 0 0 0 0 0. Wherein, the four toned phonemes of the toneless syllable shang (sh ang1 / sh ang2 / sh ang3 / sh ang4) all become the toneless syllable shang after the tone is removed, the index numbers are => ang1(1) ang2(2) ang3(3) ang4(4), then 211, 212, 213, 214 will all be mapped to shang 768.

[0081] Input the sample speech feature sequence into the toneless syllable recognition model to obtain the real toneless syllable feature sequence, then perform iterative training on the toneless syllable recognition model based on the preset second loss value, the second loss function, the real toneless syllable feature sequence, and the target toneless syllable feature sequence obtained in the above steps, until the toneless syllable recognition model converges, the training process of the trained toneless syllable recognition model and tone recognition model is completed

[0082] Before training the tone recognition model, it is necessary to determine the target tone feature sequence according to the target toned phoneme feature sequence based on the index information of each target toned phoneme.

[0083] Based on the tone feature sequence, on the premise of retaining its tones 1 to 4 and the neutral tone, a toneless label is added to identify the tones of non-speech segments and speech segments that do not contain tonality such as unvoiced sounds, and the final number of classifications is 6.

[0084] Taking the above content as an example, the tones are classified into 6 categories: 0, 1, 2, 3, 4, and 5. Initials have no tone, only finals have a tone (refer to phonetics for details). The index number for no tone is 0, the index number for neutral tone is 5, the index number for first tone is 1, the index number for second tone is 2, the index number for third tone is 3, and the index number for fourth tone is 4. Then, the characteristic sequence of the target tone corresponding to "mou" is: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 0 0 0 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 0 0 0 0 0 0 0 0 0 0.

[0085] The sample speech feature sequence is input into the tone recognition model to obtain the real tone feature sequence. Then, based on the preset third loss value, third loss function, real tone feature sequence, and target tone feature sequence obtained in the above steps, the tone recognition model is iteratively trained until the tone recognition model converges, and the trained tone recognition model is obtained.

[0086] Step 203: Fuse the tonal phoneme feature sequence to be tested, the untone syllable feature sequence to be tested, and the tone feature sequence to be tested to obtain the fused tonal syllable feature sequence to be tested.

[0087] Since the tonal phoneme feature sequence, the tonal syllable feature sequence, and the tone feature sequence are three sub-features from three perspectives, in practical applications, the three sub-feature sequences need to be fused to obtain a tonal syllable feature sequence, and then the tonal syllable feature sequence is used for downstream speech recognition tasks.

[0088] The tonal phoneme feature sequence, the untone syllable feature sequence, and the tone feature sequence can be fused using a fusion formula to obtain a fused tonal syllable feature sequence. Alternatively, a neural network model can be used to fuse these sequences. The specific fusion method is not limited here and can be selected based on the actual situation. Figure 3a The diagram shown is the first type of feature sequence fusion process.

[0089] In one embodiment, the step of fusing the to-be-tested tone-phoneme feature sequence, the to-be-tested toneless syllable feature sequence, and the to-be-tested tone feature sequence to obtain the fused to-be-tested tone-phoneme feature sequence includes: calling a trained fusion model; and fusing the to-be-tested tone-phoneme feature sequence, the to-be-tested toneless syllable feature sequence, and the to-be-tested tone feature sequence through the trained fusion model to obtain the fused to-be-tested tone-phoneme feature sequence.

[0090] Before calling the fusion model of the training number, the fusion model needs to be trained to obtain a trained fusion model.

[0091] In one embodiment, the method further includes the following steps before invoking the trained fusion model:

[0092] Based on the index information of each target toned phoneme, the target toned syllable feature sequence is determined according to the target toned phoneme feature sequence. The real toned phoneme feature sequence, the real untoned syllable feature sequence, and the real tone feature sequence are input into the fusion model for processing to obtain the real toned syllable feature sequence. The fusion model is iteratively trained according to the preset fourth loss value, fourth loss function, real toned syllable feature sequence, and target toned syllable feature sequence until the fusion model converges, resulting in a trained fusion model. Figure 3a The diagram shown is a second schematic of the feature sequence fusion process.

[0093] First, based on the index number information of each target tone phoneme, the target tone syllable feature sequence needs to be determined according to the target tone phoneme feature sequence.

[0094] Based on the feature sequence of toned syllables, the initials and finals can be directly mapped to the corresponding toned syllables (pinyin). In practical applications, a blank state is added to represent the case where there is no predicted value. The final number of classifications is the number of toned syllables plus one.

[0095] Taking the above content as an example, first, for the tonal syllable corresponding to "morning": shang4(2677)wu3(3012), the 01 sequence dimension is 3468, then the tonal syllable feature sequence corresponding to "morning" is: 0 0 0 0 0 0 0 0 0 0 2677 26772677 2677 2677 2677 2677 2677 2677 2677 2677 2677 2677 2677 2677 2677 26772677 2677 2677 2677 2677 2677 2677 2677 2677 2677 2677 2677 2677 3012 30123012 3012 3012 3012 3012 3012 3012 3012 3012 3012 3012 3012 3012 3012 30123012 3012 3012 3012 3012 3012 3012 3012 3012 3012 3012 3012 3012 0 0 0 0 0 00 0 0 0, where the index number of the tonal syllable shang4 is 2677, which is split into the sequence of the tonal phoneme shang4(214), and all frame positions of the substring of 214 are replaced with 2677, so as to obtain the target tonal syllable feature sequence.

[0096] In the second step, the real tonal phoneme feature sequence, the real toneless syllable feature sequence, and the real tone feature sequence obtained in the above steps are input into a fusion model for processing, so as to obtain a real tonal syllable feature sequence.

[0097] Finally, the fusion model is subjected to iterative training according to a preset fourth loss value, a fourth loss function, the real tonal syllable feature sequence and the target tonal syllable feature sequence, until the fusion model converges, and a trained fusion model is obtained.

[0098] In speech recognition tasks, the boundaries of many phonemes are difficult to distinguish, and it is difficult to align input data and output data, while CTC does not require alignment of input and output data. Therefore, the fusion model can be trained based on the CTC criterion. Different from the traditional frame-level alignment method that strictly relies on cross entropy, CTC does not require alignment of training data, and it will merge all alignments with the same output.

[0099] In the embodiments of the present application, the structures of the tonal phoneme recognition model, the toneless syllable recognition model, and the tone recognition model include an input layer, n hidden layers and an output layer, and the specific neural network modeling method may adopt, but is not limited to, structures such as TDNN and LSTM.

[0100] Step 204: Perform text recognition on the to-be-tested tone syllable feature sequence to obtain the target text corresponding to the speech to be recognized.

[0101] In one embodiment, based on a word segmentation text training model and a preset syllable dictionary, text recognition is performed on the to-be-tested tonal syllable feature sequence to obtain the target text corresponding to the speech to be recognized.

[0102] First, an n-gram model is trained based on the segmented text to obtain the grammatical information corresponding to the tonal syllable feature sequence to be tested. Then, in a pre-defined syllable dictionary, a word graph search is performed on the grammatical information to obtain multiple translated texts corresponding to the tonal syllable feature sequence and the probability of each translated text. The translated text with the highest probability is determined as the target text.

[0103] This application provides a speech recognition method. The method first acquires the speech feature sequence of the speech to be recognized, then recognizes the speech feature sequence to obtain a toned phoneme feature sequence, a toned syllable feature sequence, and a tone feature sequence. Next, the toned phoneme feature sequence, the toned syllable feature sequence, and the tone feature sequence are fused to obtain a fused toned syllable feature sequence. Finally, text recognition is performed on the toned syllable feature sequence to obtain the target text corresponding to the speech to be recognized. This application extracts speech feature sequences from three perspectives: toned phonemes, toned syllables, and tones. The fusion of these three features yields a more expressive fused toned syllable feature sequence, resulting in more accurate target text recognition.

[0104] Based on the methods described in the above embodiments, this embodiment will be further described from the perspective of a speech recognition device. Please refer to [link / reference]. Figure 4 , Figure 4 The present application provides a detailed structural diagram of a speech recognition device 400, which may include:

[0105] The acquisition unit 401 is used to acquire the speech feature sequence of the speech to be recognized.

[0106] The first recognition unit 402 is used to recognize the speech feature sequence of the speech to be recognized, and obtain the tonal phoneme feature sequence, the untone syllable feature sequence, and the tone feature sequence corresponding to the speech to be recognized.

[0107] In one embodiment, the first identification unit 402 includes:

[0108] The first recognition subunit is used to recognize the speech feature sequence of the speech to be recognized by using a trained tonal phoneme recognition model, a trained untoned syllable recognition model, and a trained tone recognition model, respectively obtaining the tonal phoneme feature sequence, the untoned syllable feature sequence, and the tone feature sequence corresponding to the speech to be recognized.

[0109] The fusion unit 403 is used to fuse the toned phoneme feature sequence to be tested, the untoned syllable feature sequence to be tested, and the tone feature sequence to be tested to obtain the fused toned syllable feature sequence to be tested.

[0110] In one embodiment, the fusion unit 403 includes:

[0111] Calling a sub-unit is used to call the trained fusion model;

[0112] The fusion subunit is used to fuse the toned phoneme feature sequence, the untone syllable feature sequence, and the tone feature sequence to be tested using a trained fusion model to obtain the fused toned syllable feature sequence.

[0113] The second recognition unit 404 is used to perform text recognition on the to-be-tested tone syllable feature sequence to obtain the target text corresponding to the speech to be recognized.

[0114] This application also provides a server, such as... Figure 5 The diagram shown is a structural schematic of the server provided in an embodiment of this application. Specifically:

[0115] The server includes components such as a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a power supply 503, and an input unit 504. Those skilled in the art will understand that... Figure 5 The server architecture shown does not constitute a limitation on the server and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. Wherein:

[0116] Processor 501 is the control center of the server, connecting various parts of the server through various interfaces and lines. It performs various server functions and processes data by running or executing software programs and / or modules stored in memory 502, and by calling data stored in memory 502, thereby providing overall monitoring of the server. Optionally, processor 501 may include one or more processing cores; preferably, processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into processor 501.

[0117] The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the server, etc. In addition, the memory 502 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

[0118] The server also includes a power supply 503 that supplies power to the various components. Preferably, the power supply 503 can be logically connected to the processor 501 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 503 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0119] The server may also include an input unit 504, which can be used to receive input numeric or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0120] Although not shown, the server may also include a display processor, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 501 in the server loads the executable files corresponding to the processes of one or more applications into the memory 502 according to the following instructions, and the processor 501 runs the applications stored in the memory 502 to realize various functions, as follows:

[0121] Obtain the speech feature sequence of the speech to be recognized;

[0122] The speech feature sequence of the speech to be recognized is identified to obtain the toned phoneme feature sequence, the untoned syllable feature sequence, and the tone feature sequence corresponding to the speech to be recognized.

[0123] The to-test tone-phone feature sequence, the to-test toneless syllable feature sequence, and the to-test tone feature sequence are fused to obtain the fused tone-phone syllable feature sequence.

[0124] Text recognition is performed on the tonal syllable feature sequence to obtain the target text corresponding to the speech to be recognized.

[0125] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the detailed description of the speech recognition method above, which will not be repeated here.

[0126] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0127] Therefore, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the speech recognition methods provided in embodiments of this application.

[0128] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0129] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0130] Since the instructions stored in the readable storage medium of the calculator can execute the steps of any of the speech recognition methods provided in the embodiments of this application, the beneficial effects that any of the speech recognition methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0131] Meanwhile, embodiments of this application provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described above.

[0132] The above provides a detailed description of a speech recognition method, apparatus, server, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A speech recognition method, characterized in that, include: Obtain the speech feature sequence of the speech to be recognized; Based on the sample speech feature sequence and the target toned phoneme feature sequence corresponding to the sample speech feature sequence, a trained toned phoneme recognition model, a trained untoned syllable recognition model, and a trained tone recognition model are trained; the target toned phoneme feature sequence includes index number information; based on the index number information of each target toned phoneme, the target untoned syllable feature sequence and the target tone feature sequence are determined according to the target toned phoneme feature sequence; The speech feature sequence of the speech to be identified is identified by the trained tonal phoneme recognition model, the trained untone syllable recognition model, and the trained tone recognition model, respectively obtaining the tonal phoneme feature sequence, the untone syllable feature sequence, and the tone feature sequence corresponding to the speech to be identified. The to-be-tested tone-phoneme feature sequence, the to-be-tested toneless syllable feature sequence, and the to-be-tested tone feature sequence are fused to obtain the fused to-be-tested tone-syllable feature sequence. Text recognition is performed on the to-be-tested tone syllable feature sequence to obtain the target text corresponding to the to-be-recognized speech.

2. The speech recognition method as described in claim 1, characterized in that, The steps of training a trained tonal phoneme recognition model, a trained untoned syllable recognition model, and a trained tone recognition model based on sample speech feature sequences and the target tonal phoneme feature sequences corresponding to the sample speech feature sequences include: Obtain the sample speech feature sequence and the target tone phoneme feature sequence corresponding to the sample speech feature sequence; Based on the sample speech feature sequence and the target tonal phoneme feature sequence, the tonal phoneme recognition model, the untone syllable recognition model, and the tone recognition model are trained respectively to obtain the trained tonal phoneme recognition model, the trained untone syllable recognition model, and the trained tone recognition model.

3. The speech recognition method as described in claim 2, characterized in that, The steps of obtaining the sample speech feature sequence and the target tone-modulated phoneme feature sequence corresponding to the sample speech feature sequence include: The sample speech feature sequence is forcibly aligned on the time axis to obtain the target tone-modulated phoneme feature sequence.

4. The speech recognition method as described in claim 2, characterized in that, The step of training the toned phoneme recognition model, the untoned syllable recognition model, and the tone recognition model based on the sample speech feature sequence and the target toned phoneme feature sequence to obtain the trained toned phoneme recognition model, the untoned syllable recognition model, and the tone recognition model, respectively, further includes: The sample speech feature sequences are respectively input into the toned phoneme recognition model, the untoned syllable recognition model, and the tone recognition model for processing to obtain the real toned phoneme feature sequences, the real untoned syllable feature sequences, and the real tone feature sequences. Based on the preset first loss value, first loss function, the real tonal phoneme feature sequence and the target tonal phoneme feature sequence, the tonal phoneme recognition model is iteratively trained until the tonal phoneme recognition model converges, and the trained tonal phoneme recognition model is obtained. Based on the preset second loss value, second loss function, the real untone syllable feature sequence, and the target untone syllable feature sequence, the untone syllable recognition model is iteratively trained until the untone syllable recognition model converges, thus obtaining the trained untone syllable recognition model. The tone recognition model is iteratively trained based on the preset third loss value, third loss function, the real tone feature sequence, and the target tone feature sequence until the tone recognition model converges, thus obtaining the trained tone recognition model.

5. The speech recognition method as described in claim 4, characterized in that, The step of fusing the to-be-tested tone-phoneme feature sequence, the to-be-tested toneless syllable feature sequence, and the to-be-tested tone feature sequence to obtain the fused to-be-tested tone-syllable feature sequence includes: Call the trained fusion model; The trained fusion model is used to fuse the tonal phoneme feature sequence, the untone syllable feature sequence, and the tone feature sequence to obtain the fused tonal syllable feature sequence.

6. The speech recognition method as described in claim 5, characterized in that, Before the step of calling the trained fusion model, the following is also included: Based on the index number information of each target tone phoneme, the target tone syllable feature sequence is determined according to the target tone phoneme feature sequence; The real tonal phoneme feature sequence, the real untone syllable feature sequence, and the real tone feature sequence are input into the fusion model for processing to obtain the real tonal syllable feature sequence. The fusion model is iteratively trained based on the preset fourth loss value, fourth loss function, the real tonal syllable feature sequence, and the target tonal syllable feature sequence until the fusion model converges, thus obtaining the trained fusion model.

7. A voice recognition device, characterized in that, The toned phoneme recognition model, the untoned syllable recognition model, and the tone recognition model in the speech recognition device are trained based on the sample speech feature sequence and the target toned phoneme feature sequence corresponding to the sample speech feature sequence; the target toned phoneme feature sequence contains index number information, and the target untoned syllable feature sequence and the target tone feature sequence are determined based on the index number information of each target toned phoneme and the target toned phoneme feature sequence. The speech recognition device includes: The acquisition unit is used to acquire the speech feature sequence of the speech to be recognized; The first recognition unit is used to recognize the speech feature sequence of the speech to be recognized by the trained tone phoneme recognition model, the trained toneless syllable recognition model and the trained tone recognition model, respectively obtaining the test tone phoneme feature sequence, the test toneless syllable feature sequence and the test tone feature sequence corresponding to the speech to be recognized. The fusion unit is used to fuse the to-be-tested tone-phoneme feature sequence, the to-be-tested toneless syllable feature sequence, and the to-be-tested tone feature sequence to obtain the fused to-be-tested tone-phoneme feature sequence. The second recognition unit is used to perform text recognition on the to-be-tested tone syllable feature sequence to obtain the target text corresponding to the speech to be recognized.

8. A server, characterized in that, The server includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the speech recognition method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the speech recognition method as described in any one of claims 1 to 6.

Images