Speech recognition method and device, equipment, storage medium and program product

Abstract

The invention discloses a voice recognition method and device, equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence, and the method comprises the steps: coding a target voice, obtaining a coding feature, enabling the coding feature to be used for determining a text corresponding to the target voice and a language to which the target voice belongs, and enabling the target voice to be a voice of any language, any language is mandarin or any dialect; and decoding the coding feature to at least obtain a text corresponding to the target voice. The coding features obtained by coding the target voice can be used for voice recognition and can also be used for language classification, that is, the language to which the target voice belongs is considered in the process of coding the target voice and the process of decoding the coding features, so that the accuracy of voice recognition is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a speech recognition method, apparatus, device, storage medium and program product. Background Art

[0002] Speech recognition is a technology that automatically converts speech into text. Its application areas are very broad, such as voice assistants, smart homes, and car voice interactions.

[0003] In speech recognition, the dialect problem is an important challenge faced by current speech recognition technology. Current speech recognition generally has the problem of low recognition accuracy when facing dialect recognition. Summary of the invention

[0004] In view of the above problems, the present application provides a speech recognition method, apparatus, device, storage medium and program product to improve the accuracy of speech recognition. The specific scheme is as follows:

[0005] The first aspect of the present application provides a speech recognition method, comprising:

[0006] Encode the target speech to obtain encoding features; the encoding features are used to determine the text corresponding to the target speech and the language to which the target speech belongs; the target speech is a speech of any language, and the any language is Mandarin or any dialect;

[0007] The encoded features are decoded to at least obtain text corresponding to the target speech.

[0008] In a possible implementation, the text corresponding to the target speech is: a Mandarin text; or a text in the language to which the target speech belongs;

[0009] The decoding of the coded feature comprises:

[0010] The encoded features are decoded to obtain a text sequence, wherein the text sequence includes: the text corresponding to the target speech and the language to which the target speech belongs.

[0011] In a possible implementation, the process of encoding the target speech and decoding the encoding feature includes:

[0012] Encoding the acoustic features of the target speech through a first speech recognition model to obtain the encoded features; decoding the encoded features to obtain the text sequence;

[0013] The first speech recognition model is trained using speech samples of various languages ​​as training samples and text sequences annotated with the corresponding speech samples as labels; the text sequence corresponding to each speech sample includes: the text corresponding to the speech sample and the language to which the speech sample belongs.

[0014] In a possible implementation, the text corresponding to the target speech is: a Mandarin text; or a text in the language to which the target speech belongs;

[0015] The decoding of the coded feature comprises:

[0016] Decoding the encoded features to obtain text corresponding to the target speech;

[0017] The method further includes: classifying the coding features to obtain the language to which the target speech belongs.

[0018] In a possible implementation, the coding feature is a coding feature of each speech frame of the target speech; and the classifying process of the coding feature includes:

[0019] The coding features of the respective speech frames are merged to obtain sentence-level coding features of the target speech;

[0020] The sentence-level coding features are classified by a classifier to obtain the language to which the target speech belongs.

[0021] In a possible implementation, the process of encoding the target speech and decoding and classifying the encoded features includes:

[0022] Encoding the acoustic features of the target speech through a second speech recognition model to obtain the encoded features; decoding the encoded features to obtain text corresponding to the target speech; and classifying the encoded features to obtain the language to which the target speech belongs;

[0023] The second speech recognition model is trained using speech samples of various languages ​​as training samples and using text sequences and languages ​​annotated with the corresponding speech samples as labels; the text sequence corresponding to each speech sample includes: the text corresponding to the speech sample.

[0024] In a possible implementation, the process of training the second speech recognition model includes:

[0025] Inputting the acoustic features of the speech sample into the second speech recognition model, obtaining a text corresponding to the speech sample obtained by decoding the coded features of the speech sample by the second speech recognition model, and a language to which the speech sample belongs obtained by classifying the coded features of the speech sample;

[0026] Based on a first difference between the text corresponding to the speech sample obtained by the second speech recognition model and the text sequence label, and a second difference between the language to which the speech sample belongs obtained by the second speech recognition model and the language label, the parameters of the second speech recognition model are updated.

[0027] In a possible implementation, the text corresponding to the target speech is: a Mandarin text; or a text in the language to which the target speech belongs;

[0028] The decoding of the coded feature comprises:

[0029] Decoding the encoded features to obtain a text sequence, wherein the text sequence includes: text corresponding to the target speech and a first language to which the target speech belongs;

[0030] The method further comprises:

[0031] Classifying the coding features to obtain the second language to which the target speech belongs;

[0032] A final language classification of the target speech is determined in the first language and the second language.

[0033] In a possible implementation, determining a final language classification of the target speech in the first language and the second language includes:

[0034] If the first language and the second language are different, scaling a first probability that the target speech belongs to the first language based on a preset scaling factor;

[0035] Compare the difference between a second probability that the target speech belongs to the second language and the scaled first probability with a threshold;

[0036] If the difference is smaller than the threshold, the first language is determined as the final language classification of the target speech; otherwise, the second language is determined as the final language classification of the target speech.

[0037] In a possible implementation, the process of encoding the target speech and decoding and classifying the encoded features includes:

[0038] Encoding the acoustic features of the target speech through a third speech recognition model to obtain the encoded features; decoding the encoded features to obtain the text sequence; and classifying the encoded features to obtain the second language to which the target speech belongs;

[0039] The third speech recognition model is trained using speech samples of various languages ​​as training samples, and using text sequences and languages ​​annotated with the corresponding speech samples as labels; the text sequence corresponding to each speech sample includes: the text corresponding to the speech sample and the language to which the speech sample belongs.

[0040] In a possible implementation, the process of training the third speech recognition model includes:

[0041] Inputting the acoustic features of the speech sample into the third speech recognition model, obtaining the text corresponding to the speech sample and the first language to which the speech sample belongs, obtained by decoding the encoding features of the speech sample by the third speech recognition model, and classifying the encoding features of the speech sample to obtain the second language to which the speech sample belongs;

[0042] The parameters of the third speech recognition model are updated based on the text corresponding to the speech sample obtained by the third speech recognition model, the first language to which the speech sample belongs and the first difference between the text sequence label, and the second language obtained by the third speech recognition model and the language label.

[0043] In a possible implementation, the text corresponding to the target speech is a Mandarin text; the text annotated with the corresponding speech sample is a Mandarin text;

[0044] In the case that the language to which the speech sample belongs is any dialect, in the Mandarin text annotated with the corresponding speech sample, at least part of the Mandarin text has a preset mark, and the preset mark indicates that the pronunciation of at least part of the Mandarin text is different from that of the corresponding dialect text.

[0045] A second aspect of the present application provides a speech recognition device, comprising:

[0046] An encoding module, used for encoding the target speech to obtain encoding features; the encoding features are used to determine the text corresponding to the target speech and the language to which the target speech belongs; the target speech is a speech of any language, and the any language is Mandarin or any dialect;

[0047] The recognition module is used to decode the coding feature to at least obtain the text corresponding to the target speech.

[0048] The third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the speech recognition method of the first aspect or any implementation of the first aspect.

[0049] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0050] The memory is used to store computer programs;

[0051] The processor is used to execute the computer program so that the electronic device can implement the speech recognition method of the above-mentioned first aspect or any implementation manner of the first aspect.

[0052] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the speech recognition method of the above-mentioned first aspect or any implementation of the first aspect.

[0053] By means of the above technical scheme, the speech recognition method, apparatus, device, storage medium and program product provided by the present application encode the target speech to obtain coding features, which are used to determine the text corresponding to the target speech and the language to which the target speech belongs, the target speech is a speech of any language, the any language being Mandarin or any dialect; the coding features are decoded to at least obtain the text corresponding to the target speech. The coding features obtained by encoding the target speech in the present application can be used for both speech recognition and language classification, that is, the process of encoding the target speech and the process of decoding the coding features in the present application both take into account the language to which the target speech belongs, thereby improving the accuracy of speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0055] Figure 1 A schematic diagram of the architecture of the multi-dialect speech recognition system provided in this application;

[0056] Figure 2 Another schematic diagram of the architecture of the multi-dialect speech recognition system provided in this application;

[0057] Figure 3 A flow chart of an implementation of the speech recognition method provided in this application;

[0058] Figure 4 A structural schematic diagram of a first speech recognition model provided in this application;

[0059] Figure 5A flowchart for implementing classification processing of coding features provided by this application;

[0060] Figure 6 A structural diagram of a second speech recognition model provided in this application;

[0061] Figure 7 A structural diagram of a third speech recognition model provided in this application;

[0062] Figure 8 A schematic diagram of the structure of the speech recognition device provided by this application;

[0063] Fig. 9 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0064] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0065] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0066] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0067] For dialect recognition, different dialects have significant differences in voice, intonation, phonology, etc. For example, some dialects may have unique pronunciation or tone changes, which increases the difficulty of voice recognition; even for the same dialect, different people's accents and expression habits may be different, further increasing the difficulty of recognition. In addition, the vocabulary and grammar of dialects are also quite different from the standard language (i.e. Mandarin), which requires the voice recognition system to have strong language understanding capabilities.

[0068] Early pronunciation-based speech recognition frameworks, because they were based on pronunciation modeling, needed to build a corresponding pronunciation dictionary for each dialect. However, due to the regional characteristics of dialect pronunciation and the lack of reference standards, dialect recognition has not made much progress.

[0069] With the development of end-to-end (Encoder-Decoder) technology, dialect recognition has gradually attracted attention in recent years. End-to-end models are usually based on character or sub-word modeling, allowing the model to independently learn pronunciation information, avoiding the problem of non-standard dialect pronunciation, simplifying the entire recognition process, and reducing the complexity of the system.

[0070] Currently, there are two main solutions for multi-dialect speech recognition technology based on an end-to-end framework:

[0071] One solution is based on front-end language classification + multiple single-dialect recognition integrated systems. This solution first classifies the input speech through the language classifier, and then, based on the classification results, calls the corresponding single-dialect recognition system on the back end (that is, a dialect recognition system is set up for each dialect) to perform speech recognition on the input speech. Figure 1 The figure shows a schematic diagram of the architecture of the multi-dialect speech recognition system provided by the present application.

[0072] Another solution is to perform output modeling for each dialect at the output end of the decoder (i.e., output modeling based on multiple languages), and finally determine the final speech recognition result by comparing the speech recognition results of multiple languages. Figure 2 As shown, it is another architecture diagram of the multi-dialect speech recognition system provided by this application.

[0073] Among them, the integrated system based on front-end language classification + multiple single dialect recognition has the following problems: First, errors will gradually accumulate. For example, if the language classification is wrong, it will lead to an error in the called dialect recognition branch, which will lead to poor recognition results; second, the system latency is high, because the system must first classify the language and then recognize it. Usually, language classification requires the input of relatively long audio to obtain a high-accuracy classification result, which means an increase in latency; third, the system overhead and maintenance cost are high. As the number of dialects increases, the system overhead also increases linearly, which will eventually make the system unbearable.

[0074] The solution based on modeling of multiple language outputs has the following problems: First, it increases the complexity of the system. First, establishing multi-language outputs increases the number of model parameters, which requires more training data to optimize the parameters. At the same time, due to the existence of multiple outputs, complex training algorithms and decoding algorithms need to be designed during the training and recognition stages to obtain accurate recognition results. Moreover, compared with standard languages, dialect data is relatively small and not standardized, so the trained model has low accuracy and poor generalization ability. Second, it increases model redundancy. Although there are differences between multiple dialects, there are also some common parts. These common parts are redundant in each dialect output.

[0075] In order to solve at least some of the problems existing in the above-mentioned speech recognition scheme, the present application scheme is proposed.

[0076] like Figure 3 As shown, a flow chart of an implementation of the speech recognition method provided in an embodiment of the present application may include:

[0077] Step S301: Encode the target speech to obtain encoding features. The encoding features are used to determine the text corresponding to the target speech and the language to which the target speech belongs; the target speech is a speech of any language, and the any language is Mandarin or any dialect.

[0078] The target voice is the voice to be recognized, which may be voice collected in real time by a voice collection device (eg, a microphone), or may be pre-stored voice read from a storage unit, or may be voice uploaded by a user.

[0079] The target speech may be a dialect speech (any dialect), that is, the content of the target speech is expressed in the dialect. Alternatively, the target speech may be a Mandarin speech, that is, the content of the target speech is expressed in Mandarin. In other words, the speech recognition method provided in the present application can be used for speech recognition of both dialect speech and Mandarin speech.

[0080] The coding features obtained by encoding the target speech in the present application can not only determine the text corresponding to the target speech, but also determine the language to which the target speech belongs (that is, the coding features can be used to determine the language category of the target speech). In other words, the coding features obtained by encoding the target speech are extracted from the target speech, and represent the features of the text corresponding to the target speech and the language to which the target speech belongs.

[0081] Step S302: Decode the encoded features to at least obtain the text corresponding to the target speech.

[0082] The present application decodes the coding features to obtain only the text corresponding to the target speech. In this case, the language to which the target speech belongs can be obtained by processing the coding features in other ways.

[0083] Alternatively, the coding features are decoded to obtain the text corresponding to the target speech and the language to which the target speech belongs. In this case, the language to which the decoded target speech belongs can be determined as the language category of the target speech; or the language category of the target speech can be finally determined by combining the language to which the target speech belongs determined by processing the coding features in other ways.

[0084] The speech recognition method provided in the embodiment of the present application encodes the target speech to obtain a coding feature, and the coding feature is used to determine the text corresponding to the target speech and the language to which the target speech belongs, the target speech is a speech of any language, and the any language is Mandarin or any dialect; the coding feature is decoded to at least obtain the text corresponding to the target speech. The coding feature obtained by encoding the target speech in the present application can be used for both speech recognition and language classification, that is, the process of encoding the target speech and the process of decoding the coding feature in the present application both take into account the language to which the target speech belongs, thereby improving the accuracy of speech recognition.

[0085] Moreover, the solution of the present application can recognize speech in multiple languages. Compared with single-language speech recognition, it can simultaneously complete language classification and multi-language speech recognition with only a small amount of additional parameters. Compared with the prior art of first classifying the language and then selecting the speech recognition system of the corresponding language for speech recognition, the present application changes the serial processing method into a parallel processing method, thereby reducing system latency.

[0086] In an optional embodiment, the text corresponding to the target speech is the text of the language to which the target speech belongs. For example, if the target speech is Henan dialect, the coding feature obtained by encoding the target speech is used to determine the Henan dialect text corresponding to the target speech and the language to which the target speech belongs (i.e., Henan dialect). In other words, the coding feature is used to determine the text of the language to which the target speech belongs and the language to which the target speech belongs; accordingly, the coding feature is decoded to at least obtain the text of the language to which the target speech belongs.

[0087] The research of this application discovers that if the speech is recognized as the text in the language to which the speech belongs, when the speech is a dialect, the recognized text is a dialect text. In some scenarios (such as customer service, Q&A), users cannot understand what it means. For example, "bó xiàng" in Shanghainese means playing or having fun. If one is not from Shanghai, they simply cannot understand what it means. If the dialect speech is recognized as a Mandarin text, this problem can be overcome. Based on this, in an optional embodiment, the text corresponding to the target speech is a Mandarin text. That is to say, the encoded features are used to determine the Mandarin text corresponding to the target speech and the language to which the target speech belongs; correspondingly, the encoded features are decoded to obtain at least the Mandarin text corresponding to the target speech.

[0088] In the case where the target speech is a dialect speech, this application directly recognizes the dialect speech as a Mandarin text, which can achieve barrier-free communication in different dialects.

[0089] That is to say, regardless of whether the target speech is a Mandarin speech or a dialect speech, when the encoded features of the target speech are decoded, the obtained text is a Mandarin text.

[0090] In an optional embodiment, one implementation manner of decoding the encoded features may be:

[0091] The encoded features are decoded to obtain a text sequence, which includes: the text corresponding to the target speech and the language to which the target speech belongs. In this implementation manner, the language to which the target speech belongs is recognized as the text.

[0092] As an example, the language to which the target speech belongs can be decoded first, and then based on the language to which the target speech belongs, the text corresponding to the target speech is decoded.

[0093] As an example, the text corresponding to the target speech can be decoded first, and then based on the text corresponding to the target speech, the language to which the target speech belongs is decoded.

[0094] In an optional embodiment, the acoustic features of the target speech can be encoded through a speech recognition model (for the convenience of description and distinction, denoted as the first speech recognition model) to obtain encoded features, and the encoded features are decoded to obtain a text sequence.

[0095] The acoustic features of the target speech may be the acoustic features of each speech frame of the target speech (i.e., the target speech is first divided into frames to obtain multiple speech frames, and the acoustic features of each speech frame are extracted). The acoustic features may include but are not limited to any of the following: Mel-Frequency Cepstral Coefficients (MFCC), Filter Bank (Fbank) features, Perceptual Linear Predictive (PLP) features, etc.

[0096] The encoding of the acoustic features of the target speech results in the encoding features of each speech frame.

[0097] like Figure 4 As shown, a structural schematic diagram of a first speech recognition model provided in an embodiment of the present application may include:

[0098] A first encoder 401 and a first decoder 402;

[0099] The first encoder 401 is used to encode the acoustic features of the target speech to obtain the encoding features of each speech frame. The first encoder 401 can adopt any of the following network structures: Recurrent Neural Network (RNN), Convolutional Neural Network (CNN), transformer, conformer, etc., which are not limited in this case.

[0100] The first decoder 402 is used to decode the encoding features of each speech frame output by the first encoder 401 to obtain a text sequence corresponding to the target speech (such as Figure 4 y1,y2,y3,…y i …y I ,lid), the text sequence includes: the text corresponding to the target speech (such as Figure 4 y1,y2,y3,…y i …y I ) and language tags (such as Figure 4 The language label lid represents the language to which the target speech belongs. i represents the i-th character recognized; I is the total number of characters in the text corresponding to the target speech.

[0101] Optionally, the first decoder 402 decodes the encoding features of each speech frame output by the first encoder 401 to obtain the probability distribution of each decoding moment, and the probability distribution of each decoding moment represents the probability of belonging to each character (including language label) at the decoding moment; wherein, the character decoded at each decoding moment is the character corresponding to the maximum probability in the probability distribution of the decoding moment.

[0102] The first speech recognition model is trained using speech samples of various languages ​​as training samples and text sequences annotated with the corresponding speech samples as labels (recorded as text sequence labels). The text sequence label of each speech sample includes: the text corresponding to the speech sample and the language to which the speech sample belongs (specifically, it can be a language label).

[0103] When decoding the encoded features, the first speech recognition model may use autoregressive decoding or other decoding methods. This application does not specifically limit the decoding method.

[0104] The speech sample set used in the present application for training the first speech recognition model consists of speech samples in multiple languages ​​(including Mandarin and multiple dialects).

[0105] Optionally, in the text sequence label of each speech sample, the text corresponding to the speech sample is the text of the language to which the speech sample belongs.

[0106] Optionally, in the text sequence label of each speech sample, the text corresponding to the speech sample is Mandarin text.

[0107] Optionally, in the text sequence label of each speech sample, the language to which the speech sample belongs is located before the text corresponding to the speech sample, that is, the language to which the speech sample belongs is the beginning part of the text sequence label.

[0108] Optionally, in the text sequence label of each speech sample, the language to which the speech sample belongs is located after the text corresponding to the speech sample, that is, the language to which the speech sample belongs is the end of the text sequence label. Compared with when the language to which the speech sample belongs is located before the text corresponding to the speech sample, when the language to which the speech sample belongs is located after the text corresponding to the speech sample, the accuracy of the first speech recognition model obtained by training is higher.

[0109] By adding the language information of the speech sample (for example, the language label) to the text label of the speech sample, the speech recognition model can make a prediction of the language information after seeing all the inputs. The predicted language information helps to correctly translate the dialect audio into Mandarin text.

[0110] In an optional embodiment, one implementation method of training the first speech recognition model may be:

[0111] Extract the acoustic features of the speech sample, encode the acoustic features of the speech sample through the first speech recognition model to obtain the encoded features, decode the encoded features to obtain a text sequence corresponding to the speech sample; the text sequence includes: the text corresponding to the speech sample and the language to which the speech sample belongs.

[0112] Extracting the acoustic features of the speech sample may include: dividing the speech sample into frames to obtain multiple speech frames, and extracting the acoustic features of each speech frame. The acoustic features may include but are not limited to any of the following: MFCC, Fbank, PLP, etc.

[0113] Based on the difference between the text sequence decoded by the first speech recognition model and the text sequence label of the speech sample, the parameters of the first speech recognition model are updated.

[0114] The difference between the text sequence decoded by the first speech recognition model and the text sequence label of the speech sample can be calculated using a preset first loss function, and the parameters of the first speech recognition model can be updated with the goal of reducing the difference.

[0115] In an optional embodiment, another implementation of the above decoding of the coding feature may be:

[0116] The coded features are decoded to obtain the text corresponding to the target speech. That is, only the text corresponding to the target speech is decoded. In this case, the language to which the target speech belongs can be obtained by classifying the coded features.

[0117] Unlike the previous embodiment, in which the text corresponding to the target speech and the language to which the target speech belongs are both obtained by decoding in one step, in the embodiment of the present application, the text corresponding to the target speech and the language to which the target speech belongs are obtained by performing different processing on the encoding features (i.e., decoding and language classification).

[0118] Optionally, an implementation of encoding the target speech may be:

[0119] The target speech is divided into frames to obtain multiple speech frames, and each speech frame is encoded to obtain the encoding features of each speech frame.

[0120] The present application provides a flowchart for implementing the classification processing of coding features. Figure 5 As shown, it may include:

[0121] Step S501: Fusing the coding features of each speech frame to obtain sentence-level coding features of the target speech.

[0122] Optionally, the coding features of each speech frame of the target speech may be averaged to obtain the sentence-level coding features of the target speech.

[0123] Optionally, the coding features of each speech frame of the target speech may be summed to obtain the sentence-level coding features of the target speech.

[0124] Optionally, the mean and standard deviation of the coding features of each speech frame of the target speech may be obtained, and the obtained mean and standard deviation may be concatenated to obtain the sentence-level coding features of the target speech.

[0125] Assume that the target speech is divided into J speech frames, and the encoding feature of the jth (j=1, 2, 3, ..., J) speech frame is an R-dimensional feature vector, denoted by e j , then the mean, sum and standard deviation of J speech frames are all R-dimensional vectors. When the mean and standard deviation of the coding features of each speech frame of the target speech are concatenated, the sentence-level coding feature of the target speech is a vector of dimension 2R.

[0126] Step S502: classify the sentence-level coding features through a classifier to obtain the language to which the target speech belongs.

[0127] In an optional embodiment, an implementation method of encoding the target speech and decoding and classifying the encoded features may be:

[0128] The acoustic features of the target speech are encoded through a speech recognition model (referred to as the second speech recognition model for ease of description and distinction) to obtain encoding features; the encoding features are decoded to obtain the text corresponding to the target speech; the encoding features are classified to obtain the language to which the target speech belongs.

[0129] The acoustic features of the target speech may be acoustic features of each speech frame of the target speech, and the acoustic features may include but are not limited to any of the following: MFCC, Fbank, PLP, etc.

[0130] The encoding of the acoustic features of the target speech results in the encoding features of each speech frame.

[0131] In the embodiment of the present application, the second speech recognition model does not obtain the language of the target speech by decoding the coded features, but obtains the language of the target speech by classifying the coded features.

[0132] like Figure 6 As shown, a structural diagram of a second speech recognition model provided in an embodiment of the present application may include:

[0133] A second encoder 601, a second decoder 602 and a first classifier 603;

[0134] The second encoder 601 is used to encode the acoustic features of the target speech to obtain the encoding features of each speech frame. The second encoder 601 can adopt any of the following network structures: RNN, CNN, transformer, conformer, etc., which are not limited in this case.

[0135] The second decoder 602 is used to decode the encoding features of each speech frame output by the second encoder 601 to obtain a text sequence corresponding to the target speech (such as Figure 6 y1,y2,y3,…y i …y I );y i represents the i-th character recognized; I is the total number of characters in the text corresponding to the target speech.

[0136] Optionally, the second decoder 602 decodes the coding features of each speech frame output by the second encoder 601 to obtain the probability distribution of each decoding moment, and the probability distribution of each decoding moment represents the probability that the decoding moment belongs to each character (excluding the language label); wherein the character at each decoding moment is the character corresponding to the maximum probability in the probability distribution of the decoding moment.

[0137] The first classifier 603 is used to classify the target speech based on the coding features of each speech frame output by the second encoder 601 (that is, taking the coding features of each speech frame as a whole and classifying the whole), and obtain the language to which the target speech belongs (such as Figure 6 The first classifier 603 can fuse the coding features of each speech frame to obtain the sentence-level coding features of the target speech, and classify the sentence-level coding features to obtain the language to which the target speech belongs.

[0138] Optionally, the first classifier 603 classifies the target speech based on the coding features of each speech frame output by the second encoder 601 to obtain a probability distribution of the target speech, which represents the probability that the target speech belongs to each language. The language to which the target speech belongs is the language corresponding to the maximum probability in the probability distribution.

[0139] Optionally, the first classifier 603 may be a linear classifier or a nonlinear classifier.

[0140] The second speech recognition model is trained using speech samples of various languages ​​as training samples, and using the text sequences and languages ​​annotated with the corresponding speech samples as labels (i.e., the speech samples have two labels, namely: text sequence label and language label); the text sequence label corresponding to each speech sample includes: the text corresponding to the speech sample.

[0141] When decoding the encoded features, the second speech recognition model may use autoregressive decoding or other decoding methods. This application does not specifically limit the decoding method.

[0142] The speech sample set used in the present application for training the second speech recognition model consists of speech samples in multiple languages ​​(including Mandarin and multiple dialects).

[0143] Optionally, the text corresponding to each voice sample is a text in the language to which the voice sample belongs.

[0144] Optionally, the text corresponding to each voice sample is Mandarin text.

[0145] In an optional embodiment, one implementation method of training the second speech recognition model may be:

[0146] The acoustic features of the speech sample are input into the second speech recognition model to obtain the text corresponding to the speech sample obtained by the second speech recognition model decoding the encoded features of the speech sample, and the language to which the speech sample belongs obtained by classifying the encoded features of the speech sample.

[0147] The acoustic features of the speech sample may be the acoustic features of each speech frame of the speech sample (the specific acquisition method may refer to the aforementioned embodiment, which will not be described in detail here). The second speech recognition model encodes the acoustic features of each speech frame of the speech sample to obtain the encoding features of each speech frame, decodes the acoustic features of each speech frame, and obtains the text corresponding to the speech sample; fuses the encoding features of each speech frame to obtain the sentence-level encoding features of the speech sample, and classifies the sentence-level encoding features to obtain the language to which the speech sample belongs.

[0148] Acoustic features may include but are not limited to any of the following: MFCC, Fbank, PLP, etc.

[0149] Based on a first difference between the text corresponding to the speech sample obtained by the second speech recognition model and the text sequence label, and a second difference between the language to which the speech sample obtained by the second speech recognition model belongs and the language label, the parameters of the second speech recognition model are updated.

[0150] The first difference between the text and the text sequence label corresponding to the speech sample obtained by the second speech recognition model can be calculated using a preset first loss function, and the second difference between the language to which the speech sample obtained by the second speech recognition model belongs and the language label can be calculated using a preset second loss function. The parameters of the second speech recognition model are updated with the goal of reducing both the first difference and the second difference.

[0151] Optionally, the first difference and the second difference may be weighted and summed to obtain a comprehensive difference (referred to as the first comprehensive difference), and the parameters of the second speech recognition model may be updated with the goal of reducing the first comprehensive difference. The weight of the first difference is greater than the weight of the second difference. As an example, the weight of the first difference is 1, and the weight of the second difference is between 0 and 0.5.

[0152] In an optional embodiment, an implementation method of decoding the coding feature may be:

[0153] The coded features are decoded to obtain a text sequence, which includes: the text corresponding to the target speech and the language to which the target speech belongs (for the convenience of description and distinction, recorded as the first language). In this case, the coded features can also be classified to obtain the language to which the target speech belongs (for the convenience of description and distinction, recorded as the second language), and the final language classification of the target speech is determined between the first language and the second language.

[0154] Different from the previous embodiment in which the coding features are processed in one way to determine the language to which the target speech belongs, in this embodiment, the coding features are first processed in two ways respectively to obtain the language to which the target speech belongs, and then one language is selected from the languages ​​to which the target speech belongs obtained based on the two ways as the final language category of the target speech.

[0155] Optionally, an implementation method of determining the final language category of the target speech in the first language and the second language may be:

[0156] If the first language and the second language are the same, the final language category of the target speech is the first language.

[0157] If the first language and the second language are different, the first probability that the target speech belongs to the first language is scaled based on a preset scaling factor to obtain a scaled first probability.

[0158] When decoding the coding features, the present application obtains the probability distribution of each decoded character in each language (that is, the probability of each decoded character belonging to each language, and the sum of the probabilities of the target language belonging to each language is 1). For each decoded character, the language corresponding to the maximum probability (for ease of description and distinction, recorded as the first probability) is determined as the first language to which the character belongs.

[0159] The first probability may be multiplied by a scaling factor to obtain a scaled first probability. The scaling factor may be between 0 and 1.

[0160] The difference between the second probability that the target speech belongs to the second language and the scaled first probability is compared with a threshold.

[0161] When classifying the coding features, the present application obtains the probability distribution of the target speech in each language (that is, the probability that the target speech belongs to each language, and the sum of the probabilities that the target speech belongs to each language is 1), and the language corresponding to the maximum probability (for the convenience of description and distinction, recorded as the second probability) is determined as the second language to which the target speech belongs.

[0162] If the difference is less than the threshold, the first language is determined as the final language category of the target speech; otherwise, the second language is determined as the final language category of the target speech.

[0163] The threshold value is between 0 and 1.

[0164] In an optional embodiment, an implementation method of encoding the target speech and decoding and classifying the encoded features may be:

[0165] The acoustic features of the target speech are encoded through a speech recognition model (referred to as the third speech recognition model for ease of description and distinction) to obtain encoded features; the encoded features are decoded to obtain a text sequence (including: the text corresponding to the target speech and the first language to which the target speech belongs); the encoded features are classified to obtain the second language to which the target speech belongs.

[0166] The acoustic features of the target speech may be acoustic features of each speech frame of the target speech, and the acoustic features may include but are not limited to any of the following: MFCC, Fbank, PLP, etc.

[0167] The encoding of the acoustic features of the target speech results in the encoding features of each speech frame.

[0168] In the embodiment of the present application, the third speech recognition model obtains the language to which the target speech belongs through two methods (ie, decoding and classification processing).

[0169] like Figure 7 As shown, a structural diagram of a third speech recognition model provided in an embodiment of the present application may include:

[0170] A third encoder 701, a third decoder 702 and a second classifier 703;

[0171] The third encoder 701 is used to encode the acoustic features of the target speech to obtain the encoding features of each speech frame. The third encoder 701 can adopt any of the following network structures: RNN, CNN, transformer, conformer, etc., which is not limited in this case.

[0172] The third decoder 702 is used to decode the encoding features of each speech frame output by the third encoder 701 to obtain a text sequence corresponding to the target speech (such as Figure 7 y1,y2,y3,…y i …y I ,lid1), the text sequence includes: the text corresponding to the target speech (such as Figure 7 y1,y2,y3,…y i …y I ) and language tags (such as Figure 7 lid1 in the text box indicates the first language); y i represents the i-th character recognized; I is the total number of characters in the text corresponding to the target speech.

[0173] Optionally, the third decoder 702 decodes the coding features of each speech frame output by the third encoder 701 to obtain the probability distribution of each decoding moment, and the probability distribution of each decoding moment represents the probability that the decoding moment belongs to each character (including language label); wherein the character at each decoding moment is the character corresponding to the maximum probability in the probability distribution of the decoding moment.

[0174] The second classifier 703 is used to classify the target speech based on the coding features of each speech frame output by the third encoder 701 (that is, taking the coding features of each speech frame as a whole and classifying the whole), and obtain the second language to which the target speech belongs (such as Figure 7 The second classifier 703 can fuse the coding features of each speech frame to obtain the sentence-level coding features of the target speech, and classify the sentence-level coding features to obtain the second language to which the target speech belongs.

[0175] Optionally, the second classifier 703 classifies the target speech based on the coding features of each speech frame output by the third encoder 701 to obtain a probability distribution of the target speech, which represents the probability that the target speech belongs to each language. The second language to which the target speech belongs is the language corresponding to the maximum probability in the probability distribution.

[0176] Optionally, the second classifier 703 may be a linear classifier or a nonlinear classifier.

[0177] The third speech recognition model is trained using speech samples of various languages ​​as training samples, and using the text sequences and languages ​​annotated with the corresponding speech samples as labels (i.e., the speech samples have two labels, namely: text sequence label and language label); the text sequence label of each speech sample includes: the text corresponding to the speech sample and the language to which the speech sample belongs.

[0178] When decoding the encoded features, the third speech recognition model may use autoregressive decoding or other decoding methods. This application does not specifically limit the decoding method.

[0179] The speech sample set used in the present application for training the third speech recognition model consists of speech samples in multiple languages ​​(including Mandarin and multiple dialects).

[0180] Optionally, in the text sequence label of each speech sample, the text corresponding to the speech sample is the text of the language to which the speech sample belongs.

[0181] Optionally, in the text sequence label of each speech sample, the text corresponding to the speech sample is Mandarin text.

[0182] Optionally, in the text sequence label of each speech sample, the language to which the speech sample belongs is located before the text corresponding to the speech sample, that is, the language to which the speech sample belongs is the beginning part of the text sequence label.

[0183] Optionally, in the text sequence label of each speech sample, the language to which the speech sample belongs is located after the text corresponding to the speech sample, that is, the language to which the speech sample belongs is the end of the text sequence label. Compared with when the language to which the speech sample belongs is located before the text corresponding to the speech sample, when the language to which the speech sample belongs is located after the text corresponding to the speech sample, the accuracy of the trained third speech recognition model is higher.

[0184] In an optional embodiment, one implementation method of training the third speech recognition model may be:

[0185] The acoustic features of the speech sample are input into the third speech recognition model, and the third speech recognition model decodes the encoded features of the speech sample to obtain the text corresponding to the speech sample and the first language to which the speech sample belongs, and classifies the encoded features of the speech sample to obtain the second language to which the speech sample belongs.

[0186] The acoustic features of the speech sample may be the acoustic features of each speech frame of the speech sample (the specific acquisition method may refer to the aforementioned embodiment, which will not be described in detail here). The third speech recognition model encodes the acoustic features of each speech frame of the speech sample to obtain the encoding features of each speech frame, decodes the encoding features of each speech frame, obtains the text corresponding to the speech sample and the first language to which the speech sample belongs; fuses the encoding features of each speech frame to obtain the sentence-level encoding features of the speech sample, classifies the sentence-level encoding features, and obtains the second language to which the speech sample belongs.

[0187] Acoustic features may include but are not limited to any of the following: MFCC, Fbank, PLP, etc.

[0188] Based on a first difference between a text sequence obtained by the third speech recognition model (i.e., the text corresponding to the speech sample and the first language to which the speech sample belongs) and the text sequence label, and a second difference between the second language obtained by the third speech recognition model and the language label, the parameters of the third speech recognition model are updated.

[0189] The third difference between the text corresponding to the speech sample obtained by the third speech recognition model and the first language to which the speech sample belongs and the text sequence label can be calculated using the preset first loss function. The fourth difference between the second language to which the speech sample obtained by the third speech recognition model belongs and the language label can be calculated using the preset second loss function. The parameters of the third speech recognition model are updated with the goal of reducing both the third difference and the fourth difference.

[0190] Optionally, the third difference and the fourth difference may be weighted and summed to obtain a comprehensive difference (recorded as a second comprehensive difference), and the parameters of the third speech recognition model may be updated with the goal of reducing the second comprehensive difference. The weight of the third difference is greater than the weight of the fourth difference. As an example, the weight of the third difference is 1, and the weight of the fourth difference is between 0 and 0.5.

[0191] In an optional embodiment, when the text corresponding to the target speech obtained by decoding the coding features of the target speech is a normal text, the text annotated with the corresponding speech sample is a Mandarin text. Based on this, if the language to which the speech sample belongs is any dialect, then, in the Mandarin text annotated with the corresponding speech sample, at least part of the Mandarin text has a preset mark, and the preset mark indicates that the pronunciation of the at least part of the Mandarin text is different from that of the corresponding dialect text.

[0192] For example, a certain speech sample (for the convenience of description and distinction, denoted as the first speech sample) is an audio in Henan dialect, and its content is "Bai rou le", and its corresponding Mandarin content is "Don't dawdle". Therefore, the Mandarin annotation for the first speech sample can be "Don't dawdle". If another speech sample (for the convenience of description and distinction, denoted as the second speech sample) is an audio in Mandarin, and its content is "Don't dawdle", then the Mandarin annotation for the second speech sample is also "Don't dawdle". Obviously, the Mandarin texts annotated for the first speech sample and the second speech sample are the same, but the pronunciation differences between the first speech sample and the second speech sample are very large, that is, the same Mandarin text corresponds to different speech samples with very different pronunciations (for example, in the above example, "别" corresponds to the two pronunciations "bie" and "bai", "磨" corresponds to the two pronunciations "mo" and "rou", and "蹭" corresponds to the two pronunciations "ceng" and "le"). In this way, the speech recognition model trained is prone to recognition crosstalk between different languages, resulting in recognition errors. To overcome the recognition crosstalk between different languages, in this application, when performing text annotation on dialect speech samples, if the pronunciation of the dialect is different from the pronunciation of the corresponding Mandarin, the Mandarin text corresponding to the dialect is marked to represent that the pronunciation of this Mandarin is different from the corresponding dialect.

[0193] For example, for the first speech sample, its corresponding text annotation can be "别_1磨_1蹭_1", and the mark "_1" after each character indicates that the pronunciation of this character is different from its corresponding dialect pronunciation.

[0194] In an optional embodiment, during the training process of each speech recognition model, in order for the speech recognition model to better recognize the start position and end position of the text, when performing text sequence annotation on the speech sample, a sentence start symbol can be set before the text sequence, and a sentence end symbol can be set after the text sequence. For example, for a certain speech sample (for the convenience of description and distinction, denoted as the third speech sample), the corresponding text is "Good morning", then the text sequence annotated for the third speech sample can be ( <s> Good morning, lid,< / s> ), a total of 6 characters; among them, <s> is the sentence starter,< / s> is the sentence end symbol, and lid is the language to which the third speech sample belongs. Or, the text sequence annotated for the third speech sample can be ( <s> ,Good morning,< / s> ), a total of 5 characters; in this example, the text sequence label of the third speech sample does not include the language to which the third speech sample belongs.

[0195] During the training process of each speech recognition model provided by the embodiments of this application, when the text annotated for the corresponding speech sample is a Mandarin text, it is equivalent to unified output modeling for all languages, sharing model parameters, reducing system redundancy, alleviating the problem of sparse single-dialect data and insufficient training. It can also achieve barrier-free communication between different dialects.

[0196] When the language classification task is introduced, the training of the speech recognition model is a joint training of multiple tasks (i.e., language classification task and speech recognition task). This joint training method not only utilizes acoustic information, but also language information, making the model input information richer. At the same time, with the help of the large-capacity modeling capability of the acoustic model (including encoder and decoder), the accuracy of speech recognition and language classification is improved, the confusion between languages ​​is reduced, the crosstalk problem of different language recognition is alleviated, and the accuracy of speech recognition and language classification is improved.

[0197] In the process of training the speech recognition model, by adding language labels to the text corresponding to the speech samples, the decoder can be guided to learn the pronunciation characteristics and pronunciation habits of different languages, improve the distinction between different languages, and further alleviate the crosstalk problem of recognition of different languages.

[0198] In the process of training the speech recognition model, when text annotation is performed on the dialect speech samples, if the pronunciation of the dialect is different from the corresponding Mandarin pronunciation, the Mandarin text corresponding to the dialect is marked to indicate the difference in pronunciation between the Mandarin and the corresponding dialect, further alleviating the crosstalk problem of recognition of different languages.

[0199] Corresponding to the method embodiment, the present application also provides a speech recognition device. A structural schematic diagram of the speech recognition device provided in the embodiment of the present application is as follows: Figure 8 As shown, it may include:

[0200] Encoding module 801 and identification module 802;

[0201] The encoding module 801 is used to encode the target speech to obtain encoding features; the encoding features are used to determine the text corresponding to the target speech and the language to which the target speech belongs; the target speech is a speech of any language, and the any language is Mandarin or any dialect;

[0202] The recognition module 802 is used to decode the coding feature to at least obtain the text corresponding to the target speech.

[0203] The speech recognition device provided in the embodiment of the present application encodes the target speech to obtain a coding feature, and the coding feature is used to determine the text corresponding to the target speech and the language to which the target speech belongs, the target speech is a speech of any language, and the any language is Mandarin or any dialect; the coding feature is decoded to at least obtain the text corresponding to the target speech. The coding feature obtained by encoding the target speech in the present application can be used for both speech recognition and language classification, that is, the process of encoding the target speech and the process of decoding the coding feature in the present application both take into account the language to which the target speech belongs, thereby improving the accuracy of speech recognition.

[0204] In an optional embodiment, the text corresponding to the target speech is: a Mandarin text; or a text in the language to which the target speech belongs;

[0205] When the recognition module 802 decodes the coding feature, it is used to:

[0206] The encoded features are decoded to obtain a text sequence, wherein the text sequence includes: the text corresponding to the target speech and the language to which the target speech belongs.

[0207] In an optional embodiment, when the encoding module 801 encodes the target speech and the recognition module 802 decodes the encoding feature, it is used to:

[0208] Encoding the acoustic features of the target speech through a first speech recognition model to obtain the encoded features; decoding the encoded features to obtain the text sequence;

[0209] The first speech recognition model is trained using speech samples of various languages ​​as training samples and text sequences annotated with the corresponding speech samples as labels; the text sequence corresponding to each speech sample includes: the text corresponding to the speech sample and the language to which the speech sample belongs.

[0210] In an optional embodiment, the text corresponding to the target speech is: a Mandarin text; or a text in the language to which the target speech belongs;

[0211] When the recognition module 802 decodes the coding feature, it is used to:

[0212] Decoding the encoded features to obtain text corresponding to the target speech;

[0213] The method further includes: classifying the coding features to obtain the language to which the target speech belongs.

[0214] In an optional embodiment, the coding feature is a coding feature of each speech frame of the target speech; when the recognition module 802 classifies the coding feature, it is used to:

[0215] The coding features of the respective speech frames are merged to obtain sentence-level coding features of the target speech;

[0216] The sentence-level coding features are classified by a classifier to obtain the language to which the target speech belongs.

[0217] In an optional embodiment, when the encoding module 801 encodes the target speech and the recognition module 802 decodes and classifies the encoded features, it is used to:

[0218] Encoding the acoustic features of the target speech through a second speech recognition model to obtain the encoded features; decoding the encoded features to obtain text corresponding to the target speech; and classifying the encoded features to obtain the language to which the target speech belongs;

[0219] The second speech recognition model is trained using speech samples of various languages ​​as training samples and using text sequences and languages ​​annotated with the corresponding speech samples as labels; the text sequence corresponding to each speech sample includes: the text corresponding to the speech sample.

[0220] In an optional embodiment, the speech recognition device further includes a training module for training the second speech recognition model, specifically for:

[0221] Inputting the acoustic features of the speech sample into the second speech recognition model, obtaining a text corresponding to the speech sample obtained by decoding the coded features of the speech sample by the second speech recognition model, and a language to which the speech sample belongs obtained by classifying the coded features of the speech sample;

[0222] Based on a first difference between the text corresponding to the speech sample obtained by the second speech recognition model and the text sequence label, and a second difference between the language to which the speech sample belongs obtained by the second speech recognition model and the language label, the parameters of the second speech recognition model are updated.

[0223] In an optional embodiment, the text corresponding to the target speech is: a Mandarin text; or a text in the language to which the target speech belongs;

[0224] When the recognition module 802 decodes the coding feature, it is used to:

[0225] Decoding the encoded features to obtain a text sequence, wherein the text sequence includes: text corresponding to the target speech and a first language to which the target speech belongs;

[0226] The method further comprises:

[0227] Classifying the coding features to obtain the second language to which the target speech belongs;

[0228] A final language classification of the target speech is determined in the first language and the second language.

[0229] In an optional embodiment, when determining the final language classification of the target speech in the first language and the second language, the recognition module 802 is configured to:

[0230] If the first language and the second language are different, scaling a first probability that the target speech belongs to the first language based on a preset scaling factor;

[0231] Compare the difference between a second probability that the target speech belongs to the second language and the scaled first probability with a threshold;

[0232] If the difference is smaller than the threshold, the first language is determined as the final language classification of the target speech; otherwise, the second language is determined as the final language classification of the target speech.

[0233] In an optional embodiment, when the encoding module 801 encodes the target speech and the recognition module 802 decodes and classifies the encoded features, it is used to:

[0234] Encoding the acoustic features of the target speech through a third speech recognition model to obtain the encoded features; decoding the encoded features to obtain the text sequence; and classifying the encoded features to obtain the second language to which the target speech belongs;

[0235] The third speech recognition model is trained using speech samples of various languages ​​as training samples, and using text sequences and languages ​​annotated with the corresponding speech samples as labels; the text sequence corresponding to each speech sample includes: the text corresponding to the speech sample and the language to which the speech sample belongs.

[0236] In an optional embodiment, the speech recognition device further includes a training module for training the third speech recognition model, specifically for:

[0237] Inputting the acoustic features of the speech sample into the third speech recognition model, obtaining the text corresponding to the speech sample and the first language to which the speech sample belongs, obtained by decoding the encoding features of the speech sample by the third speech recognition model, and classifying the encoding features of the speech sample to obtain the second language to which the speech sample belongs;

[0238] The parameters of the third speech recognition model are updated based on the text corresponding to the speech sample obtained by the third speech recognition model, the first language to which the speech sample belongs and the first difference between the text sequence label, and the second language obtained by the third speech recognition model and the language label.

[0239] In an optional embodiment, the text corresponding to the target speech is a Mandarin text; the text annotated with the corresponding speech sample is a Mandarin text;

[0240] In the case that the language to which the speech sample belongs is any dialect, in the Mandarin text annotated with the corresponding speech sample, at least part of the Mandarin text has a preset mark, and the preset mark indicates that the pronunciation of at least part of the Mandarin text is different from that of the corresponding dialect text.

[0241] The present application also provides an electronic device in an embodiment. Fig. 9 As shown, it shows a structural schematic diagram of an electronic device suitable for implementing the speech recognition method in the embodiment of the present application. The electronic device in the embodiment of the present application can be a terminal device (for example, a car machine, a large-screen device, a smart home, a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.), or a server (can be a single server, can be a server cluster, or can be a cloud server, etc.). Fig. 9 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0242] like Fig. 9 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 903. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0243] Typically, the following devices may be connected to the I / O interface 905: an input device 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 908 including, for example, a memory card, a hard disk, etc.; and a communication device 909. The communication device 909 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 9 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0244] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any of the speech recognition methods provided in the embodiments of the present application.

[0245] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any speech recognition method provided in the embodiment of the present application.

[0246] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0247] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0248] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. Professionals and technicians may use different methods to implement the described functions for each specific solution, but such implementation should not be considered beyond the scope of this application.

[0249] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0250] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0251] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech recognition method, characterized in that: include: Encode the target speech to obtain encoding features; The coding feature is used to determine the text corresponding to the target speech and the language to which the target speech belongs; The target speech is a speech of any language, and the any language is Mandarin or any dialect; The encoded features are decoded to at least obtain text corresponding to the target speech.

2. The method according to claim 1, characterized in that The text corresponding to the target speech is: Mandarin text; or text in the language to which the target speech belongs; The decoding of the coded feature comprises: The encoded features are decoded to obtain a text sequence, wherein the text sequence includes: the text corresponding to the target speech and the language to which the target speech belongs.

3. The method according to claim 2, characterized in that The process of encoding the target speech and decoding the encoding features includes: Encoding the acoustic features of the target speech through a first speech recognition model to obtain the encoded features; decoding the encoded features to obtain the text sequence; The first speech recognition model is trained using speech samples of various languages ​​as training samples and text sequences annotated with the corresponding speech samples as labels; the text sequence corresponding to each speech sample includes: the text corresponding to the speech sample and the language to which the speech sample belongs.

4. The method according to claim 1, characterized in that: The text corresponding to the target speech is: Mandarin text; or text in the language to which the target speech belongs; The decoding of the coded feature comprises: Decoding the encoded features to obtain text corresponding to the target speech; The method further includes: classifying the coding features to obtain the language to which the target speech belongs.

5. The method according to claim 4, characterized in that The coding features are coding features of each speech frame of the target speech; the classification processing of the coding features includes: The coding features of the respective speech frames are merged to obtain sentence-level coding features of the target speech; The sentence-level coding features are classified by a classifier to obtain the language to which the target speech belongs.

6. The method according to claim 4, characterized in that The process of encoding the target speech and decoding and classifying the encoded features includes: Encoding the acoustic features of the target speech through a second speech recognition model to obtain the encoded features; decoding the encoded features to obtain text corresponding to the target speech; and classifying the encoded features to obtain the language to which the target speech belongs; The second speech recognition model is trained using speech samples of various languages ​​as training samples and using text sequences and languages ​​annotated with the corresponding speech samples as labels; the text sequence corresponding to each speech sample includes: the text corresponding to the speech sample.

7. The method according to claim 6, characterized in that The process of training the second speech recognition model includes: Inputting the acoustic features of the speech sample into the second speech recognition model, obtaining a text corresponding to the speech sample obtained by decoding the coded features of the speech sample by the second speech recognition model, and a language to which the speech sample belongs obtained by classifying the coded features of the speech sample; Based on a first difference between the text corresponding to the speech sample obtained by the second speech recognition model and the text sequence label, and a second difference between the language to which the speech sample belongs obtained by the second speech recognition model and the language label, the parameters of the second speech recognition model are updated.

8. The method according to claim 1, characterized in that The text corresponding to the target speech is: Mandarin text; or text in the language to which the target speech belongs; The decoding of the coded feature comprises: Decoding the encoded features to obtain a text sequence, wherein the text sequence includes: text corresponding to the target speech and a first language to which the target speech belongs; The method further comprises: Classifying the coding features to obtain the second language to which the target speech belongs; A final language classification of the target speech is determined in the first language and the second language.

9. The method according to claim 8, characterized in that Determining a final language classification of the target speech in the first language and the second language includes: If the first language and the second language are different, scaling a first probability that the target speech belongs to the first language based on a preset scaling factor; Compare the difference between a second probability that the target speech belongs to the second language and the scaled first probability with a threshold; If the difference is smaller than the threshold, the first language is determined as the final language classification of the target speech; otherwise, the second language is determined as the final language classification of the target speech.

10. The method according to claim 8, characterized in that The process of encoding the target speech and decoding and classifying the encoded features includes: Encoding the acoustic features of the target speech through a third speech recognition model to obtain the encoded features; decoding the encoded features to obtain the text sequence; and classifying the encoded features to obtain the second language to which the target speech belongs; The third speech recognition model is trained using speech samples of various languages ​​as training samples, and using text sequences and languages ​​annotated with the corresponding speech samples as labels; the text sequence corresponding to each speech sample includes: the text corresponding to the speech sample and the language to which the speech sample belongs.

11. The method according to claim 10, characterized in that The process of training the third speech recognition model includes: Inputting the acoustic features of the speech sample into the third speech recognition model, obtaining the text corresponding to the speech sample and the first language to which the speech sample belongs, obtained by decoding the encoding features of the speech sample by the third speech recognition model, and classifying the encoding features of the speech sample to obtain the second language to which the speech sample belongs; The parameters of the third speech recognition model are updated based on the text corresponding to the speech sample obtained by the third speech recognition model, the first language to which the speech sample belongs and the first difference between the text sequence label, and the second language obtained by the third speech recognition model and the language label.

12. The method according to any one of claims 3, 6 and 10, characterized in that: The text corresponding to the target speech is a Mandarin text; the text annotated with the corresponding speech sample is a Mandarin text; In the case that the language to which the speech sample belongs is any dialect, in the Mandarin text annotated with the corresponding speech sample, at least part of the Mandarin text has a preset mark, and the preset mark indicates that the pronunciation of at least part of the Mandarin text is different from that of the corresponding dialect text.

13. An electronic device, characterized in that: The electronic device comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the speech recognition method as described in any one of claims 1 to 12.

Images