Speech recognition method, apparatus, device, and storage medium

Abstract

The application discloses a speech recognition method, device and equipment and a storage medium. A speech recognition model configured by the application predicts an initial predicted text based on speech coding features output by a speech encoder through a first speech classification layer. A text encoder encodes the initial predicted text, fuses the text coding features and the speech coding features, inputs the fused coding features into a shared encoder for secondary coding, and obtains a final predicted text based on the secondary coding features by a second speech classification layer. Since the speech recognition model can extract more rich fused coding features as a whole, the recognition accuracy can be further improved. In addition, since the speech recognition model includes a text encoder and a shared encoder, the text encoder and the shared encoder can be additionally trained using pure text data in the training process. Pure text data is easier to obtain in large quantities than annotated text of speech, greatly reducing the cost of manual annotation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, and more particularly to a speech recognition method and device, equipment and a storage medium. BACKGROUND

[0002] Most of the existing speech recognition systems based on deep learning scheme are trained by manually annotated speech segments, that is, a large number of speech is collected, professional personnel record the corresponding text of the speech, and then the speech-text sentence pair is used to optimize the speech recognition system.

[0003] The existing speech recognition system generally consists of a speech encoder and a speech classification module. The speech encoder extracts high-dimensional features of the input speech, and the speech classification module further performs speech recognition based on the high-dimensional features to output the recognized text. This speech recognition system can only learn speech information, so it needs a large number of speech-text sentence pairs for training. However, the annotation of speech-text sentence pairs requires human resources with professional knowledge, which has a very high annotation cost, resulting in insufficient number of speech-text sentence pairs, which leads to poor optimization effect of the speech recognition system and low speech recognition accuracy. SUMMARY

[0004] In view of the above problems, the present application is proposed to provide a speech recognition method, device, equipment and storage medium to solve the problem that the existing speech recognition system relies on a large number of speech-text sentence pairs for training, resulting in poor optimization effect of the speech recognition system and low speech recognition accuracy when the number of speech-text sentence pairs is insufficient. The specific scheme is as follows:

[0005] In a first aspect, a speech recognition method is provided, comprising:

[0006] obtaining a speech to be recognized;

[0007] inputting the speech to be recognized into a pre-configured speech recognition model to obtain a final predicted text output by a second speech classification layer of the model, wherein the speech recognition model is configured to extract speech coding features from the input speech using a speech encoder, obtain an initial predicted text based on the speech coding features using a first speech classification layer, perform text coding on the initial predicted text using a text encoder to obtain text coding features, combine the speech coding features and the text coding features to obtain fusion coding features, and perform secondary coding on the fusion coding features using a shared encoder, and obtain the final predicted text based on the secondary coding features using a second speech classification layer;

[0008] performing normalization processing on the final predicted text to obtain a recognized text corresponding to the speech to be recognized.

[0009] In a second aspect, a speech recognition apparatus is provided, comprising:

[0010] a speech obtaining unit configured to obtain a speech to be recognized;

[0011] a model processing unit configured to input the speech to be recognized into a preconfigured speech recognition model to obtain a final predicted text output by a second speech classification layer of the model, wherein the speech recognition model is configured to extract speech coding features from an input speech by using a speech encoder, obtain an initial predicted text based on the speech coding features by using a first speech classification layer, perform text coding on the initial predicted text by using a text encoder to obtain text coding features, combine the speech coding features and the text coding features to obtain fusion coding features, perform secondary coding on the fusion coding features by using a shared encoder, and obtain the final predicted text based on the secondary coding features by using a second speech classification layer;

[0012] a text normalization unit configured to perform normalization processing on the final predicted text to obtain a recognized text corresponding to the speech to be recognized.

[0013] In a third aspect, a speech recognition device is provided, comprising a memory and a processor.

[0014] The memory is configured to store a program.

[0015] The processor is configured to execute the program to implement each step of the speech recognition method.

[0016] In a fourth aspect, a storage medium having a computer program stored thereon is provided, wherein the computer program, when executed by a processor, implements each step of the speech recognition method.

[0017] According to the above technical solution, the speech recognition model is preconfigured, and the model comprises the first speech classification layer, the text encoder and the shared encoder on the basis of the speech encoder and the second speech classification layer. In the training process, the initial predicted text is predicted based on the speech coding features output by the speech encoder through the first speech classification layer, and a loss function can be constructed based on the initial predicted text to preliminarily correct the processing result of the intermediate layer of the model, thereby improving the overall recognition accuracy of the model.

[0018] Meanwhile, a text encoder is arranged to encode the initial predicted text, and the text encoded features and the speech encoded features are fused to obtain fused encoded features. Compared with the prior art of simply extracting speech features, the scheme of the present application can obtain fused encoded features with more information, and on this basis, the fused encoded features are input into the shared encoder, which is more conducive to the learning of the shared encoder. The shared encoder is used to perform secondary encoding on the fused encoded features, and the second speech classification layer obtains the final predicted text based on the secondary encoded features. Since the speech recognition model can extract more rich fused encoded features as a whole, the overall recognition accuracy can be further improved.

[0019] In addition, since the speech recognition model of the present application comprises a text encoder and a shared encoder, the text encoder and the shared encoder can be additionally trained using pure text data in the training process. Compared with the prior art of training only using speech-text sentence pairs, the training data type is more abundant, and pure text data is easier to obtain in large quantities than annotated text of speech, greatly reducing the cost of manual annotation. BRIEF DESCRIPTION OF DRAWINGS

[0020] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not intended to limit the scope of the present application. Moreover, the same reference numerals are used throughout the various drawings to designate identical parts. In the drawings:

[0021] Figure 1 A flowchart of a speech recognition method provided by an embodiment of the present application is shown;

[0022] Figure 2 An example of a speech recognition model structure schematic diagram is shown;

[0023] Figure 3 An example of a speech encoder structure schematic diagram is shown;

[0024] Figure 4 An example of a text encoder structure schematic diagram is shown;

[0025] Figure 5 An example of a shared encoder structure schematic diagram is shown;

[0026] Figure 6 An example of a pure text data expansion process schematic diagram is shown;

[0027] Figure 7 A speech recognition device structure schematic diagram provided by an embodiment of the present application is shown;

[0028] Figure 8 A speech recognition device structure schematic diagram provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the scope of protection of the present application.

[0030] The voice recognition scheme of the present application can be implemented based on a terminal with data processing capability, which can be a mobile phone, a computer, a server, a cloud, etc.

[0031] Next, the voice recognition method of the present application can include the following steps in combination with the above description. Figure 1 The voice recognition method of the present application can include the following steps in combination with the above description.

[0032] Step S100, obtaining a voice to be recognized.

[0033] Step S110, inputting the voice to be recognized into a pre-configured voice recognition model to obtain a final predicted text output by a second voice classification layer of the model.

[0034] The voice recognition model is configured to: extract speech coding features from the input voice by using a speech encoder, obtain an initial predicted text based on the speech coding features by using a first voice classification layer, perform text coding on the initial predicted text by using a text encoder to obtain text coding features, form fusion coding features composed of the speech coding features and the text coding features, and perform secondary coding on the fusion coding features by using a shared encoder, and obtain the final predicted text based on the secondary coding features by using a second voice classification layer.

[0035] Step S120, performing normalization processing on the final predicted text to obtain a recognized text corresponding to the voice to be recognized.

[0036] The pre-configured voice recognition model of the present embodiment further includes a first voice classification layer, a text encoder and a shared encoder on the basis of the speech encoder and the second voice classification layer. In the training process, the initial predicted text is predicted based on the speech coding features output by the speech encoder through the first voice classification layer, and a loss function can be constructed based on the initial predicted text to preliminarily correct the processing result of the intermediate layer of the model, thereby improving the overall recognition accuracy of the model.

[0037] Meanwhile, a text encoder is arranged to encode the initial predicted text, and the text encoded features and the speech encoded features are fused to obtain fused encoded features. Compared with the prior art of simply extracting speech features, the scheme of the present application can obtain fused encoded features with more information, and on this basis, the fused encoded features are input into the shared encoder, which is more conducive to the learning of the shared encoder. The shared encoder is used to perform secondary encoding on the fused encoded features, and the second speech classification layer obtains the final predicted text based on the secondary encoded features. Since the speech recognition model can extract more rich fused encoded features as a whole, the overall recognition accuracy can be further improved.

[0038] In addition, since the speech recognition model of the present application comprises a text encoder and a shared encoder, the text encoder and the shared encoder can be additionally trained using pure text data in the training process. Compared with the prior art of training only using speech-text sentence pairs, the training data type is more abundant, and pure text data is easier to obtain in large quantities than annotated text of speech, greatly reducing the cost of manual annotation.

[0039] In some embodiments of the present application, the structure of the speech recognition model is introduced.

[0040] In an optional case, the speech recognition model can be configured to consist of a speech encoder, a first speech classification layer, a text encoder, a shared encoder, a second speech classification layer and a text classification layer, wherein the connection relationship of each module refers to Figure 2 .

[0041] The speech encoder receives the input speech, specifically, the mel filter features of the input speech. That is, the mel filter features of the input speech are first extracted, and then the features are input into the speech encoder, and the speech encoder extracts high-level features useful for speech recognition to obtain speech encoded features.

[0042] The speech encoder in the present embodiment can adopt a conformer model structure, which can extract more rich acoustic information.

[0043] Further, the speech encoder in the present embodiment can also adopt an improved conformer model structure to enhance the feature extraction capability. Specifically, the improved conformer model structure can refer to Figure 3 .

[0044] It can include several layers of convolutional layers connected in series, such as Figure 3 4 layers of convolutional layers as shown in Figure 3 16 layers of conformer modules as shown in

[0045] The input speech is input to a 4-layer convolutional layer after the Mel filter feature is extracted to generate 512-dimensional shallow acoustic features. Further, the 16-layer conformer module is used to finally obtain the high-level speech features of [512, T], where T is the frame number of the input speech.

[0046] Further, the text encoder in the embodiment can adopt a transformer model structure. Through the embedding layer and the multi-layer stacked transformer module in the transformer model, deep text information can be extracted, such as Figure 4 As shown in the figure, the transformer module can be set to 8 layers.

[0047] Then, the input text sequence is converted into 512-dimensional high-dimensional features by the embedding layer, and then the 8-layer transformer module is used to finally obtain the high-level text features of [512, S], where S is the length of the input text sequence.

[0048] Further, the shared encoder in the embodiment can adopt a transformer model structure, as shown in the figure. Figure 5 The 4-layer transformer module is stacked to receive the features output by the speech encoder and the text encoder, and further extract secondary encoding features.

[0049] In the embodiment, in order to simultaneously use speech and text information, a speech classification layer is added behind the speech encoder and the shared encoder, that is, Figure 2 As shown in the figure, the first speech classification layer and the second speech classification layer are used to obtain the predicted text through the speech classification layer, so as to facilitate the use of the loss function to train the speech encoder and the shared encoder. At the same time, the first speech classification layer behind the speech encoder can generate a predicted text in the training process, and the predicted text is input into the text encoder to extract text features. Further, the extracted text encoding features and the speech encoding features are fused to obtain the fused encoding features. Compared with the prior art of simply extracting speech features, the scheme of the present application can obtain more information-rich fused encoding features.

[0050] The first and second speech classification layers can be composed of two linear transformation layers to convert the features of [512, T] into predicted probabilities of [k, T], where k is the size of the dictionary, representing the number of characters.

[0051] It should be noted that at the same time, the input of the speech recognition model can only be speech input or only text input.

[0052] In training the speech recognition model, training speech can be input into the speech recognition model, or training text can be input into the speech recognition model.

[0053] In inputting training speech into the speech recognition model:

[0054] The speech encoder is configured to encode the training speech to obtain speech encoding features, the first speech classification layer is configured to obtain initial predicted text based on the speech encoding features, the text encoder is configured to encode the initial predicted text to obtain text encoding features of the training speech, the shared encoder is configured to perform secondary encoding on the fusion encoding features obtained by fusing the speech encoding features and the text encoding features, to obtain secondary encoding features of the training speech, and the second speech classification layer is configured to obtain the final predicted text corresponding to the training speech based on the secondary encoding features of the training speech.

[0055] In inputting training text into the speech recognition model:

[0056] The text encoder is configured to encode the training text to obtain text encoding features, the shared encoder is configured to perform secondary encoding on the text encoding features, to obtain secondary encoding features of the training text, and the text classification layer is configured to obtain the final predicted text corresponding to the training text based on the secondary encoding features of the training text.

[0057] Optionally, in the speech recognition model training process, the first speech classification layer and the second speech classification layer can be trained by using a CTC (Connectionist temporal classification) loss function. Since the CTC algorithm can include a blank symbol and repeated characters in the final predicted text, the final predicted text obtained by processing the speech to be recognized by using the trained speech recognition model can include the blank symbol and the repeated characters. Therefore, the process of performing the normalization processing on the final predicted text in step S120 can include:

[0058] The repeated characters between adjacent blank symbols in the final predicted text are removed, and the blank symbols are deleted, to obtain the recognized text.

[0059] Optionally, in the speech recognition model training process, the text classification layer can be trained by using a cross-entropy loss function.

[0060] Based on the structure of the speech recognition model, a training process of the speech recognition model is introduced in the embodiment, and the training process can include the following steps:

[0061] S1, obtaining a training speech text pair and pure text data, the training speech text pair including training speech and corresponding labeled text.

[0062] Specifically, the application can obtain a small amount of training speech and corresponding labeled text. At the same time, a large amount of pure text data can be obtained, and the pure text data can be obtained from Internet resources in various automated ways, for example, by using an automated tool to automatically crawl Internet text.

[0063] The pure text data has a much lower acquisition cost than the training speech text pair because it does not require manual labeling. Compared with the prior art that requires a large amount of training speech text pair, the present case can greatly reduce the cost of manual labeling.

[0064] S2, training a speech recognition model using the training speech text pair and the pure text data until a set convergence condition is reached, obtaining a trained speech recognition model.

[0065] After obtaining the training speech text pair and the pure text data, the above-mentioned speech recognition model can be trained using these data as training data until a set convergence condition is reached, obtaining a trained speech recognition model.

[0066] In some embodiments of the application, an optional implementation of the above step S2 is provided, and the specific training process can include two training stages:

[0067] First training stage:

[0068] S21, using the network parameters of the speech encoder, the first speech classification layer, the text encoder, the shared encoder and the second speech classification layer of the speech recognition model to train the speech recognition model, and calculating a first loss based on the predicted text output by the first speech classification layer and the second speech classification layer respectively and the labeled text corresponding to the training speech, until the first loss meets a set condition.

[0069] Specifically, in the first training stage, the input of the speech recognition model is only the training speech, and the training text is not required. As introduced above, the training speech is input into the speech encoder to obtain speech encoding features, and then input into the first speech classification layer to obtain preliminary predicted text. The preliminary predicted text is input into the text encoder for text encoding to obtain text encoding features. The text encoding features and the speech encoding features are fused, and the fused encoding features are input into the shared encoder for secondary encoding. Based on the secondary encoding features, the second speech classification layer obtains the final predicted text corresponding to the training speech.

[0070] In the first training stage, the first speech classification layer and the second speech classification layer can be trained respectively using the CTC loss function, and the total first loss is composed of two CTC losses. The iterative training is performed until the first loss meets the set condition, for example, until the first loss no longer decreases or the decrease amplitude is less than a preset value. The network parameters of each module in the first training stage speech recognition model are obtained.

[0071] It can be understood that when the first speech classification layer is trained using the CTC loss function, the preliminary predicted text output by the first speech classification layer needs to be obtained. Therefore, a sampling algorithm is provided in this embodiment to obtain the preliminary predicted text output by the intermediate layer (the first speech classification layer). Specifically, the output of the first speech classification layer is sampled according to the following sampling algorithm:

[0072]

[0073] wherein y represents the sampled predicted text, p i represents the prediction probability of the first speech classification layer for the i-th character, g i represents the gumbel distribution, τ is a preset temperature coefficient, and K represents the number of nodes of the first speech classification layer.

[0074] S22, the pure text data is expanded to obtain an expanded text, and the expanded text has added a silence symbol.

[0075] Specifically, when training the speech recognition model in the first training stage, the sampling CTC loss function is used to train the first speech classification layer, so that the preliminary predicted text output by the first speech classification layer contains a silence symbol blank, and the number of silence symbols blank is related to the pronunciation duration corresponding to each character. It can be understood that compared with the pure text data, the preliminary predicted text contains blank, so its length is greater than that of the pure text data.

[0076] In order to facilitate sampling of pure text data to train the speech recognition model in the second training stage, and to ensure that the input text length of the text encoder remains matched, the pure text data is expanded in this embodiment, that is, the silence symbol blank is added to the pure text data to obtain the expanded text. The length of the expanded text after the expansion processing is more matched with the length of the preliminary predicted text output by the first speech classification layer, so as to ensure the consistency of the input length of the text encoder and improve the training effect.

[0077] It can be understood that the above step S22 can be executed at any position before step S23, for example, before, after or at the same time as step S21.

[0078] The second training stage:

[0079] S23, performing mask processing on the extended text, training network parameters of a text encoder, a shared encoder and a text classification layer of the speech recognition model using the extended text after mask processing, and calculating a second loss based on a mask prediction result output by the text classification layer until the second loss meets a set condition.

[0080] It should be noted that in the second training stage, the network parameters of each module in the speech recognition model obtained in the first training stage can be used for initialization. On this basis, the network parameters of the text encoder, the shared encoder and the text classification layer in the speech recognition model can be further adjusted using the extended text.

[0081] When training using the extended text, a mask training method can be used, that is, the extended text is subjected to mask processing, and the text classification layer predicts the original characters that are masked, so as to train the network parameters of the text encoder, the shared encoder and the text classification layer.

[0082] When masking the extended text, a% of the characters in the extended text can be randomly selected, and for the selected characters, b1% of the characters are replaced with <mask>The mark is replaced by a random other character according to a probability of b2%, and remains unchanged according to a probability of b3%. Wherein, the sum of b1, b2 and b3 is equal to 100, generally, b1 is greater than b2 and b3.

[0083] During the training, a cross-entropy loss function can be used for training until the second loss meets a set condition, and the network parameters of each module in the second training stage of the trained speech recognition model are obtained.

[0084] Optionally, in some embodiments of the present application, the process of expanding the pure text data to obtain the expanded text is introduced.

[0085] An optional implementation is provided in the embodiment, which specifically includes the following steps.

[0086] S221, the training speech text pair is forcedly aligned.

[0087] Specifically, a forced alignment algorithm can be used to force the alignment of the training speech and the annotated text, and determine the corresponding speech frame of each character in the annotated text in the training speech.

[0088] S222, the training speech text pair after forced alignment is used to train the text duration prediction model.

[0089] Specifically, the training speech text pair after forced alignment is used as training data to train a text duration prediction model. The trained text duration prediction model has the ability to predict the pronunciation duration of each character in the input pure text data.

[0090] S223, the pure text data is input into the trained text duration prediction model to obtain the pronunciation duration of each character in the pure text data output by the model.

[0091] S224, according to the pronunciation duration of each character, add a number of silence symbols matching the pronunciation duration after each character in the pure text data to obtain the expanded text.

[0092] Specifically, define the predicted pronunciation duration of each character as t, then add t-1 silence symbols blank marks after each character in the pure text data to obtain the expanded text.

[0093] As Figure 6 which illustrates a pure text data expansion process diagram.

[0094] The pure text data example is "I love my home".

[0095] After the text duration prediction model is predicted, the pronunciation duration t of each character is obtained as Figure 6 shown.

[0096] Further add t-1 silence symbols after each character ( For <blank>The abbreviation of the first training phase is TP1.

[0097] In some embodiments of the present application, another optional implementation of step S2 is provided, which is based on the first and second training phases introduced in the foregoing embodiments and further includes a third training phase.

[0098] The third training phase includes:

[0099] S24, the annotated text corresponding to the training speech is expanded to obtain an expanded annotated text, and a silence symbol is added in the expanded annotated text.

[0100] Specifically, step S24 can be performed synchronously with the process of expanding the pure text data in step S22, or can be performed at any position before step S25.

[0101] The way of expanding the annotated text corresponding to the training speech can refer to the introduction of the foregoing embodiments, that is, using a text duration prediction model to predict the duration of each character in the annotated text, and adding a silence symbol based on the duration to obtain the expanded annotated text.

[0102] S25, using the training speech and the expanded annotated text to alternately train the speech recognition model, and calculating a third loss based on the results output by the first speech classification layer, the second speech classification layer and the text classification layer respectively until the third loss meets a set condition.

[0103] In the third training phase, the network parameters of each module in the speech recognition model obtained in the second training phase can be used for initialization. On this basis, the training speech and the expanded annotated text are used to alternately train the speech recognition model, and by using the training speech and the expanded annotated text to alternately train the speech recognition model, the training effect of the speech recognition model can be further improved.

[0104] In the third training phase, the first and second speech classification layers can be trained using a CTC loss function. The text classification layer is trained using a cross-entropy loss function, and then the third loss is finally composed of two CTC losses and a cross-entropy loss until the third loss meets a set condition, and the network parameters of each module in the speech recognition model after the third training phase are obtained.

[0105] The speech recognition device provided by the embodiments of the present application is described below, and the speech recognition device described below can be mutually corresponding with the speech recognition method described above.

[0106] Referring to Figure 7 , Figure 7 is a structural schematic diagram of a speech recognition device disclosed by the embodiments of the present application.

[0107] As Figure 7 As shown, the device can include:

[0108] The voice acquisition unit 11 is configured to acquire a to-be-recognized voice.

[0109] The model processing unit 12 is configured to input the to-be-recognized voice into a pre-configured voice recognition model to obtain a final predicted text output by a second voice classification layer of the model, wherein the voice recognition model is configured to: extract voice coding features from the input voice by using a voice encoder, obtain an initial predicted text based on the voice coding features by using a first voice classification layer, perform text coding on the initial predicted text by using a text encoder to obtain text coding features, combine the voice coding features and the text coding features to obtain fusion coding features, and perform secondary coding on the fusion coding features by using a shared encoder, and obtain the final predicted text based on the secondary coding features by using a second voice classification layer.

[0110] The text normalization unit 13 is configured to perform normalization processing on the final predicted text to obtain a recognized text corresponding to the to-be-recognized voice.

[0111] Optionally, in the voice recognition model training process, the first voice classification layer and the second voice classification layer can be trained by using a CTC loss function. On this basis, the process of performing normalization processing on the final predicted text by the text normalization unit to obtain the recognized text corresponding to the to-be-recognized voice can include:

[0112] de-duplicating repeated characters between adjacent silence symbols in the final predicted text and deleting the silence symbols to obtain the recognized text.

[0113] Optionally, the voice recognition model is configured to include a voice encoder, a first voice classification layer, a text encoder, a shared encoder, a second voice classification layer, and a text classification layer.

[0114] When the voice recognition model inputs a training voice:

[0115] The voice encoder is configured to code the training voice to obtain voice coding features, the first voice classification layer is configured to obtain an initial predicted text based on the voice coding features, the text encoder is configured to code the initial predicted text to obtain text coding features of the training voice, the shared encoder is configured to perform secondary coding on fusion coding features obtained by fusing the voice coding features and the text coding features, to obtain secondary coding features of the training voice, and the second voice classification layer is configured to obtain a final predicted text corresponding to the training voice based on the secondary coding features of the training voice.

[0116] When the voice recognition model inputs a training text:

[0117] The text encoder is configured to encode the training text to obtain text encoding features, the text encoding features are encoded again by the shared encoder to obtain secondary encoding features of the training text, and the final predicted text corresponding to the training text is obtained based on the secondary encoding features of the training text by the text classification layer.

[0118] Optionally, the device of the present application can also include a speech recognition model training unit for training a speech recognition model, the process including:

[0119] Obtaining a training speech text pair and pure text data, the training speech text pair including training speech and corresponding labeled text;

[0120] Training the speech recognition model using the training speech text pair and the pure text data until a set convergence condition is reached to obtain a trained speech recognition model.

[0121] Optionally, the process of training the speech recognition model using the training speech text pair and the pure text data by the speech recognition model training unit can include:

[0122] First training stage: training the network parameters of the speech encoder, the first speech classification layer, the text encoder, the shared encoder and the second speech classification layer of the speech recognition model using the training speech, and calculating a first loss based on the predicted text output by the first speech classification layer and the second speech classification layer respectively and the labeled text corresponding to the training speech until the first loss meets a set condition;

[0123] Extending the pure text data to obtain extended text, the extended text having added silence symbols;

[0124] Second training stage: performing mask processing on the extended text, training the network parameters of the text encoder, the shared encoder and the text classification layer of the speech recognition model using the mask-processed extended text, and calculating a second loss based on the mask prediction result output by the text classification layer until the second loss meets a set condition.

[0125] Optionally, the process of training the speech recognition model using the training speech text pair and the pure text data by the speech recognition model training unit can also include:

[0126] Third training stage:

[0127] Extending the labeled text corresponding to the training speech to obtain an extended labeled text, the extended labeled text having added silence symbols;

[0128] The speech recognition model is alternately trained by using the training speech and the extended annotated text, and a third loss is calculated based on results output by the first speech classification layer, the second speech classification layer and the text classification layer until the third loss meets a set condition.

[0129] Optionally, the process of extending the pure text data to obtain the extended text by the speech recognition model training unit can include:

[0130] Forcing alignment on the training speech text pair;

[0131] Training the text duration prediction model by using the forcedly aligned training speech text pair;

[0132] Inputting the pure text data into the trained text duration prediction model to obtain pronunciation duration of each character in the pure text data output by the model;

[0133] According to the pronunciation duration of each character, adding a number of silence symbols matching the pronunciation duration after each character in the pure text data to obtain the extended text.

[0134] Optionally, the process of obtaining the predicted text output by the first speech classification layer in the first training stage by the speech recognition model training unit can include:

[0135] In the first training stage, the output of the first speech classification layer is sampled according to the following sampling algorithm:

[0136]

[0137] wherein y represents the sampled predicted text, p i represents the prediction probability of the first speech classification layer for the i-th character, g i represents the gumbel distribution, τ is a preset temperature coefficient, and K represents the number of nodes of the first speech classification layer.

[0138] The speech recognition device provided by the embodiments of the present application can be applied to a speech recognition device such as a mobile phone, a computer, a server, a cloud, etc. Optionally, Figure 8 A hardware structure block diagram of the speech recognition device is shown, referring to Figure 8 The hardware structure of the speech recognition device can include at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4.

[0139] In the embodiments of the present application, the number of processors 1, communication interfaces 2, memories 3 and communication buses 4 is at least one, and the processor 1, the communication interface 2 and the memory 3 complete communication with each other through the communication bus 4.

[0140] The processor 1 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present application, etc.

[0141] The memory 3 can include a high-speed RAM memory, and can also include a non-volatile memory, such as at least one disk memory, etc.

[0142] The memory stores a program, and the processor can invoke the program stored in the memory, and the program is used for:

[0143] Obtaining a to-be-recognized speech;

[0144] Inputting the to-be-recognized speech into a pre-configured speech recognition model to obtain a final predicted text output by a second speech classification layer of the model, wherein the speech recognition model is configured to: extract speech coding features from the input speech by using a speech encoder, obtain an initial predicted text based on the speech coding features by using a first speech classification layer, perform text coding on the initial predicted text by using a text encoder to obtain text coding features, form fusion coding features by combining the speech coding features and the text coding features, and perform secondary coding on the fusion coding features by using a shared encoder, and obtain the final predicted text based on the secondary coding features by using a second speech classification layer;

[0145] Performing a normalization process on the final predicted text to obtain a recognized text corresponding to the to-be-recognized speech.

[0146] Optionally, the refinement function and the extension function of the program can refer to the description above.

[0147] The embodiments of the present application also provide a storage medium which can store a program suitable for a processor to execute, and the program is used for:

[0148] Obtaining a to-be-recognized speech;

[0149] Inputting the to-be-recognized speech into a pre-configured speech recognition model to obtain a final predicted text output by a second speech classification layer of the model, wherein the speech recognition model is configured to: extract speech coding features from the input speech by using a speech encoder, obtain an initial predicted text based on the speech coding features by using a first speech classification layer, perform text coding on the initial predicted text by using a text encoder to obtain text coding features, form fusion coding features by combining the speech coding features and the text coding features, and perform secondary coding on the fusion coding features by using a shared encoder, and obtain the final predicted text based on the secondary coding features by using a second speech classification layer;

[0150] The final predicted text is normalized to obtain the recognized text corresponding to the speech to be recognized.

[0151] Optionally, the refinement function and the extension function of the program can refer to the above description.

[0152] Finally, it should be noted that in this document, the terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply these entities or operations have any such actual relationship or order. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, so that a process, method, article, or apparatus that includes a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by an indefinite article "a" does not exclude the existence, of additional identical elements in the process, method, article, or apparatus that includes the element.

[0153] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The various embodiments can be combined as needed, and the same or similar parts refer to each other.

[0154] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.< / blank> < / mask>

Claims

1. A voice recognition method, characterized by, The method comprises: acquiring a to-be-recognized voice; inputting the to-be-recognized voice into a pre-configured voice recognition model to obtain a final predicted text output by a second voice classification layer of the model, wherein the voice recognition model is configured to: extract voice coding features from the input voice by using a voice encoder, obtain an initial predicted text based on the voice coding features by using a first voice classification layer, perform text coding on the initial predicted text by using a text encoder to obtain text coding features, combine the voice coding features and the text coding features to obtain fusion coding features, and perform secondary coding on the fusion coding features by using a shared encoder, and obtain the final predicted text based on the secondary coding features by using a second voice classification layer; performing normalization processing on the final predicted text to obtain a recognized text corresponding to the to-be-recognized voice; wherein when a training text is input into the voice recognition model: the text encoder is configured to encode the training text to obtain text coding features, perform secondary coding on the text coding features by using the shared encoder to obtain secondary coding features of the training text, and obtain a final predicted text corresponding to the training text based on the secondary coding features of the training text by using a text classification layer; the training process of the voice recognition model comprises: acquiring a training voice text pair and pure text data, wherein the training voice text pair comprises a training voice and corresponding labeled text; training the voice recognition model by using the training voice text pair and the pure text data until a set convergence condition is reached to obtain a trained voice recognition model.

2. The method of claim 1, wherein, In the training process of the voice recognition model, the first voice classification layer and the second voice classification layer are trained by using a CTC loss function; the normalization processing on the final predicted text to obtain the recognized text corresponding to the to-be-recognized voice comprises: de-duplicating repeated characters between adjacent silence symbols in the final predicted text and deleting the silence symbols to obtain the recognized text.

3. The method of claim 1, wherein, The voice recognition model is configured to comprise a voice encoder, a first voice classification layer, a text encoder, a shared encoder, a second voice classification layer, and a text classification layer; when a training voice is input into the voice recognition model: the voice encoder is configured to encode the training voice to obtain voice coding features, the first voice classification layer is configured to obtain an initial predicted text based on the voice coding features, the text encoder is configured to encode the initial predicted text to obtain text coding features of the training voice, the shared encoder is configured to perform secondary coding on fusion coding features obtained by fusing the voice coding features and the text coding features, the second voice classification layer is configured to obtain a final predicted text corresponding to the training voice based on the secondary coding features of the training voice.

4. The method of claim 1, wherein, the training of the voice recognition model by using the training voice text pair and the pure text data comprises: The first training stage: network parameters of a speech encoder, a first speech classification layer, a text encoder, a shared encoder and a second speech classification layer of the speech recognition model are trained by using the training speech, and a first loss is calculated based on predicted texts output by the first speech classification layer and the second speech classification layer respectively and annotated texts corresponding to the training speech, until the first loss meets a set condition; The pure text data is expanded to obtain expanded text, and a silence symbol is added to the expanded text; The second training stage: network parameters of the text encoder, the shared encoder and a text classification layer of the speech recognition model are trained by using the expanded text after mask processing, and a second loss is calculated based on mask prediction results output by the text classification layer, until the second loss meets a set condition.

5. The method of claim 4, wherein, The process of training the speech recognition model by using the training speech text pair and the pure text data further includes: The third training stage: The annotated text corresponding to the training speech is expanded to obtain expanded annotated text, and a silence symbol is added to the expanded annotated text; The speech recognition model is alternately trained by using the training speech and the expanded annotated text, and a third loss is calculated based on results output by the first speech classification layer, the second speech classification layer and the text classification layer respectively, until the third loss meets a set condition.

6. The method of claim 4, wherein, The process of expanding the pure text data to obtain expanded text includes: The training speech text pair is forcedly aligned; The text duration prediction model is trained by using the forcedly aligned training speech text pair; The pure text data is input into the trained text duration prediction model to obtain pronunciation duration of each character in the pure text data output by the model; According to the pronunciation duration of each character, a silence symbol matching the pronunciation duration is added after each character in the pure text data to obtain the expanded text.

7. The method of claim 4, wherein, The process of obtaining the predicted text output by the first speech classification layer in the first training stage includes: In the first training stage, the output of the first speech classification layer is sampled according to the following sampling algorithm: where y represents the predicted text after sampling, p i represents the prediction probability of the first speech classification layer for the i-th character, g i represents the gumbel distribution, τ is a preset temperature coefficient, and K represents the number of nodes of the first speech classification layer.

8. The method according to any one of claims 1 to 7, characterized in that, The speech encoder adopts a conformer model structure, and the text encoder and the shared encoder both adopt a transformer model structure.

9. A speech recognition apparatus, characterized by comprising: It includes: A speech acquisition unit is configured to acquire a to-be-recognized speech; A model processing unit is configured to input the to-be-recognized speech into a preconfigured speech recognition model to obtain a final predicted text output by a second speech classification layer of the model, wherein the speech recognition model is configured to extract speech encoding features from the input speech by using a speech encoder, obtain an initial predicted text based on the speech encoding features by using a first speech classification layer, perform text encoding on the initial predicted text by using a text encoder to obtain text encoding features, combine the speech encoding features and the text encoding features to form fusion encoding features, and perform secondary encoding on the fusion encoding features by using a shared encoder, and obtain the final predicted text based on the secondary encoding features by using a second speech classification layer; A text normalization unit is configured to perform normalization processing on the final predicted text to obtain a recognized text corresponding to the to-be-recognized speech. In the input of the speech recognition model, the training text is: The text encoder is used to encode the training text to obtain text encoding features, and the shared encoder is used to encode the text encoding features to obtain secondary encoding features of the training text, and the final predicted text corresponding to the training text is obtained based on the secondary encoding features of the training text through the text classification layer. The speech recognition device further comprises a speech recognition model training unit for training a speech recognition model, and the process comprises: Obtain training speech text pairs and pure text data, wherein the training speech text pairs comprise training speech and corresponding labeled text; Train the speech recognition model using the training speech text pairs and the pure text data until a set convergence condition is reached to obtain a trained speech recognition model.

10. A speech recognition device, characterized by Comprise: Memory and processor; The memory is used to store programs; The processor is used to execute the program to realize the steps of the speech recognition method in any one of claims 1-8.

11. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to realize the steps of the speech recognition method in any one of claims 1-8.

Images