Speech recognition model training and speech recognition method, device and storage medium

Abstract

The application provides a speech recognition model training and speech recognition method, device and storage medium, the method comprising: obtaining a plurality of audio frames corresponding to a speech sample and a reference text sequence. A plurality of first feature vectors corresponding to the plurality of audio frames are determined according to an encoding network, and a plurality of second feature vectors corresponding to a plurality of texts are determined according to a prediction network. The corresponding relationship between the plurality of audio frames and the plurality of texts is determined according to the plurality of first feature vectors and a boundary perception model. First to-be-fused feature vectors corresponding to the plurality of audio frames and second to-be-fused feature vectors corresponding to the plurality of texts are determined. The first to-be-fused feature vectors and the second to-be-fused feature vectors are input into a fusion network to determine a target loss according to the prediction output information of the fusion network and the reference text sequence, and the parameters of the speech recognition model and the boundary perception model are updated based on the target loss. The application can reduce the training overhead of the speech recognition model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a speech recognition model training method, a speech recognition method, a device and a storage medium. BACKGROUND

[0002] Speech recognition technology refers to recognizing speech data and converting the content corresponding to the speech data into computer-readable input. For example, through speech recognition technology, the content contained in the speech data can be converted into corresponding text, facilitating subsequent processing of the content contained in the speech data.

[0003] At present, with the continuous development of deep learning, the existing technology generally uses RNN-T (Recurrent Neural Network Transducer) to perform speech recognition on speech data, and the RNN-T is an end-to-end speech recognition model. However, the RNN-T model has a large memory overhead and a slow training speed in actual training. SUMMARY

[0004] The embodiments of the present application provide a speech recognition model training method, a speech recognition method, a device and a storage medium, to reduce the training overhead of the speech recognition model and improve the training speed.

[0005] In a first aspect, the embodiments of the present application provide a speech recognition model training method, wherein the speech recognition model comprises an encoding network, a prediction network and a fusion network, and the method comprises:

[0006] obtaining a plurality of audio frames corresponding to a speech sample and a reference text sequence, wherein the reference text sequence comprises a plurality of texts corresponding to the speech sample;

[0007] determining a plurality of first feature vectors corresponding to the plurality of audio frames according to the encoding network, and determining a plurality of second feature vectors corresponding to the plurality of texts according to the prediction network;

[0008] determining a correspondence between the plurality of audio frames and the plurality of texts according to the plurality of first feature vectors and a boundary awareness model, wherein the boundary awareness model is used to determine the speech boundaries of the plurality of texts, each of the audio frames corresponds to a continuous set number of texts, and the set number is less than the number of the plurality of texts;

[0009] determining a first to-be-fused feature vector corresponding to the plurality of audio frames and a second to-be-fused feature vector corresponding to the plurality of texts according to the correspondence, the plurality of first feature vectors and the plurality of second feature vectors, wherein the dimensions of the first to-be-fused feature vector and the second to-be-fused feature vector are determined according to the number of the plurality of first feature vectors and the set number.

[0010] input the first to-be-fused feature vector and the second to-be-fused feature vector into the fusion network to determine a target loss according to predicted output information of the fusion network and the reference character sequence, and update parameters of the speech recognition model and the boundary awareness model based on the target loss.

[0011] In a second aspect, an embodiment of the present application provides a speech recognition model training device, and the device comprises:

[0012] The acquisition module is configured to acquire a plurality of audio frames corresponding to a speech sample and a reference character sequence, and the reference character sequence comprises a plurality of characters corresponding to the speech sample.

[0013] The first determination module is configured to determine a plurality of first feature vectors corresponding to the plurality of audio frames according to an encoding network, and determine a plurality of second feature vectors corresponding to the plurality of characters according to a prediction network.

[0014] The second determination module is configured to determine a correspondence between the plurality of audio frames and the plurality of characters according to the plurality of first feature vectors and a boundary awareness model, wherein the boundary awareness model is configured to determine speech boundaries of the plurality of characters, and each audio frame corresponds to a continuous set number of characters, and the set number is less than a number of the plurality of characters.

[0015] The third determination module is configured to determine a first to-be-fused feature vector corresponding to the plurality of audio frames and a second to-be-fused feature vector corresponding to the plurality of characters according to the correspondence, the plurality of first feature vectors, and the plurality of second feature vectors, wherein a dimension of the first to-be-fused feature vector and the second to-be-fused feature vector is determined according to a number of the plurality of first feature vectors and the set number.

[0016] The parameter update module is configured to input the first to-be-fused feature vector and the second to-be-fused feature vector into the fusion network to determine a target loss according to predicted output information of the fusion network and the reference character sequence, and update parameters of the speech recognition model and the boundary awareness model based on the target loss.

[0017] In a third aspect, an embodiment of the present application provides a speech recognition method, and the method comprises:

[0018] acquiring a speech to be recognized;

[0019] inputting the speech to be recognized into a speech recognition model to obtain a target text corresponding to the speech to be recognized, wherein the speech recognition model is obtained by training the speech recognition model training method.

[0020] In a fourth aspect, an embodiment of the present application provides a speech recognition device, and the device comprises:

[0021] The voice acquisition module is configured to acquire the voice to be recognized.

[0022] The text acquisition module is configured to input the voice to be recognized into a voice recognition model to obtain target text corresponding to the voice to be recognized.

[0023] In a fifth aspect, an embodiment of the present application provides a voice recognition model training method, and the method comprises the following steps:

[0024] The receiving terminal device triggers a request by calling a model training service, and the request comprises type indication information of a voice recognition model to be trained.

[0025] According to the computing resources corresponding to the model training service, the following steps are performed:

[0026] A plurality of audio frames corresponding to a voice sample and a reference text sequence are acquired, the reference text sequence comprises a plurality of texts corresponding to the voice sample, and the voice recognition model comprises an encoding network, a prediction network and a fusion network.

[0027] A plurality of first feature vectors corresponding to the plurality of audio frames are determined according to the encoding network, and a plurality of second feature vectors corresponding to the plurality of texts are determined according to the prediction network.

[0028] According to the plurality of first feature vectors and a boundary awareness model, a corresponding relationship between the plurality of audio frames and the plurality of texts is determined, wherein the boundary awareness model is used to determine voice boundaries of the plurality of texts, each of the audio frames corresponds to a continuous set number of texts, and the set number is less than the number of the plurality of texts.

[0029] According to the corresponding relationship, the plurality of first feature vectors and the plurality of second feature vectors, a first to-be-fused feature vector corresponding to the plurality of audio frames and a second to-be-fused feature vector corresponding to the plurality of texts are determined, and the dimensions of the first to-be-fused feature vector and the second to-be-fused feature vector are determined according to the number of the plurality of first feature vectors and the set number.

[0030] The first to-be-fused feature vector and the second to-be-fused feature vector are input into the fusion network to determine a target loss according to prediction output information of the fusion network and the reference text sequence, and parameters of the voice recognition model and the boundary awareness model are updated based on the target loss.

[0031] The trained voice recognition model is sent to the terminal device.

[0032] In a sixth aspect, an electronic device is provided, and the electronic device includes a memory, a processor, and a communication interface. The memory has stored thereon executable code that, when executed by the processor, causes the processor to perform the speech recognition model training method of the first aspect or the speech recognition method of the third aspect.

[0033] In a seventh aspect, a non-transitory machine-readable storage medium is provided, and the non-transitory machine-readable storage medium has stored thereon executable code that, when executed by a processor of an electronic device, causes the processor to at least implement the speech recognition model training method of the first aspect or the speech recognition method of the third aspect.

[0034] In the speech recognition model training method provided by the embodiments of the present application, a boundary perception model is introduced to assist in training the speech recognition model, and the speech recognition model includes an encoding network, a prediction network, and a fusion network. Specifically, a plurality of audio frames corresponding to a speech sample and a reference text sequence corresponding to the speech sample are first obtained, and then a plurality of first feature vectors corresponding to the plurality of audio frames are determined according to the encoding network, and a plurality of second feature vectors corresponding to a plurality of texts are determined according to the prediction network. According to the plurality of first feature vectors and the boundary perception model, the corresponding relationship between the plurality of audio frames and the plurality of texts can be determined. The boundary perception model is used to determine the speech boundaries of the plurality of texts, and finally a continuous set number of texts corresponding to each audio frame in the reference text sequence is determined, and the set number is less than the number of the plurality of texts. In fact, the boundary perception model may determine that a certain audio frame corresponds to a certain word in the reference text sequence, but considering that the recognition result of the boundary perception model is not necessarily very accurate, some redundancy is added, and after determining that the other several words continuous to the position of the word as the label of the audio frame, the corresponding relationship between each audio frame and the set number of texts, the plurality of first feature vectors, and the plurality of second feature vectors are used to determine a plurality of first to-be-fused feature vectors corresponding to the plurality of audio frames and a plurality of second to-be-fused feature vectors corresponding to the plurality of texts input into the fusion network, wherein the dimensions of the first to-be-fused feature vectors and the second to-be-fused feature vectors are determined according to the number of the plurality of first feature vectors and the set number, so that the dimension of the feature vector input into the fusion network is reduced, not only the storage space occupation is reduced, but also the calculation amount of the fusion network is reduced, thereby reducing the computing power consumption of the training process and improving the training speed. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings described below are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0036] Figure 1 A training schematic diagram of an RNN-T model provided for the prior art;

[0037] Figure 2 A training schematic diagram of a speech recognition model provided for the embodiments of the present application;

[0038] Figure 3 A flowchart of a speech recognition model training method provided for the embodiments of the present application;

[0039] Figure 4 A method flowchart for determining the correspondence between multiple audio frames and multiple characters provided for the embodiments of the present application;

[0040] Figure 5 A speech boundary generation schematic diagram provided for the embodiments of the present application;

[0041] Figure 6 A flowchart of a speech recognition method provided for the embodiments of the present application;

[0042] Figure 7 An interaction schematic diagram between a terminal device and the cloud provided for the embodiments of the present application;

[0043] Figure 8 A structural schematic diagram of a speech recognition model training device provided for the embodiments of the present application;

[0044] Figure 9 A structural schematic diagram of a speech recognition device provided for the embodiments of the present application;

[0045] Figure 10 A structural schematic diagram of an electronic device provided for the embodiments of the present application. DETAILED DESCRIPTION

[0046] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will combine the drawings in the embodiments of the present application to make a clear and complete description of the technical solutions in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application. In addition, the step timing in each method embodiment below is only an example, rather than a strict limitation.

[0047] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0048] To facilitate understanding of the technical solutions provided by this invention, the following will first introduce and explain the terms and related background knowledge involved in this invention:

[0049] 1. CIF (Continuous Integrate-and-Fire): A speech recognition algorithm that can generate real-time alignment between acoustic features and text. Simply put, it can locate the speech boundary of each word in a speech.

[0050] 2. RNN-T: Short for Recurrent Neural Network Transducer, it's an end-to-end speech recognition model, typically comprising an encoder network, a prediction network, a fusion network, and a prediction output layer (Softmax layer). The prediction output layer can be included within the fusion network or exist independently. The encoder network provides the acoustic model and can employ any acoustic model network structure, such as a Long Short-Term Memory (LSTM) network or a Transformer model. The prediction network provides the language model and can be implemented as a Recurrent Neural Network (RNN). The fusion network combines the states of the language model and the acoustic model in some way (e.g., concatenation or addition), and can be implemented as a Feedforward Neural Network (FNN).

[0051] 3. Emission latency: refers to the time difference between the speech recognition model generating the recognition result and the user stopping pronunciation. In other words, how long after the user stops pronouncing a word will the corresponding text recognition result be output.

[0052] The following is combined Figure 1 A brief introduction to the traditional training process of RNN-T models:

[0053] Acquire speech samples and extract acoustic features from them. Specifically, a speech sample of a certain duration can be divided into multiple audio frames. For example, if each 10ms segment is divided into one audio frame, then a 1s speech sample will yield 100 audio frames. Acoustic feature extraction is performed on each audio frame to obtain the acoustic features of those 100 audio frames. These 100 acoustic features are then input into the encoding network to obtain T feature vectors output by the encoding network. The number T can be less than or equal to 100; for example, if the encoding network performs downsampling, the number T will be less than 100.

[0054] During model training, for each speech sample, the supervision information is a reference text sequence consisting of the corresponding characters of that speech sample. In practical applications, a special character (referred to as the null character in this paper) is set and inserted at the beginning of the reference text sequence. Assuming the total length of the reference text sequence is U, when this reference text sequence is input into the prediction network, the prediction network outputs feature vectors corresponding to the U characters.

[0055] Suppose the above T feature vectors are represented as: h = {h1, h2, ..., h...} T The U eigenvectors are represented as: g = {g1, g2, ..., g} U When inputting into the fusion network, two feature vectors of dimension T*U are needed, one of which is the sum of {h1,h2,…,h...}. T The first step is to copy U copies to obtain the feature vector H, and the second step is to copy {g1,g2,…,g}. U The feature vector G is obtained by copying T copies. Then, the feature vectors H and G are input into the fusion network for fusion processing. The fusion network finally outputs the text sequence predicted for the current speech sample.

[0056] Assuming the RNN-T model is trained using batch training, with a batch size of N, a number of feature vectors output by the encoding network of T, a reference text sequence length of U corresponding to the current speech sample, and a vocabulary size of V (typically several thousand characters), then the fusion network will ultimately output a probability distribution matrix of dimension N*T*U*V. In other words, the RNN-T model actually performs loss calculations on a four-dimensional lattice of shape (N, T, U, V). Each point in the lattice represents the probability of outputting each character in the vocabulary for a given speech sample within a specific batch, given the feature vectors output by both the encoding and prediction networks.

[0057] In order to facilitate understanding of the replication and fusion process of the feature vectors in the conventional training scheme, the replication and fusion process of the feature vectors in the conventional training scheme will be described in combination with Figure 1 by way of example.

[0058] In Figure 1 , assuming that T=4 and U=3, h={h1, h2, h3, h4} and g={g1, g2, g3}, then replicating {h1, h2, h3, h4} three times will become a 4x3 matrix as follows: H={{h1, h1, h1}, {h2, h2, h2}, {h3, h3, h3}, {h4, h4, h4}}, and replicating {g1, g2, g3} four times will become a 4x3 matrix as follows: G={{g1, g2, g3}, {g1, g2, g3}, {g1, g2, g3}, {g1, g2, g3}}, it can be seen that the results of replicating h and g are both 4x3, only the replication direction is different, and they can be directly added in the fusion network.

[0059] As can be seen from the above example, the dimensions of the feature vectors H and G that need to be input into the fusion network are both T*U, and with the change of the number U of characters contained in different speech samples, for long speech, on the one hand, more storage space will be occupied, and on the other hand, the calculation amount of the fusion network is also very large. Moreover, as can be seen from the replication and fusion process described above, the feature vector g of one character i may correspond to the feature vector h of any audio frame i This unrestricted correspondence will result in a large transmission delay. For example, if there is a correspondence between h4 and g1, it means that the user cannot recognize the first character until after the fourth frame of speech is emitted, while in fact the first character may have been pronounced when the second audio frame is emitted.

[0060] In view of this, an embodiment of the present application provides a speech recognition model training system, a schematic diagram of which can be seen from Figure 2 , for example, Figure 2As shown, the application adds a boundary awareness model to the traditional RNN-T model, which is used to determine the speech boundaries of multiple characters, and then determines the correspondence between multiple audio frames and multiple characters in the reference character sequence based on a given number, wherein each audio frame corresponds to a continuous number of characters, and the number is less than the number of multiple characters, which is equivalent to tagging each audio frame, and the content of the label is the continuous number of characters determined based on the positioning result of the boundary awareness model, and the number is much smaller than the number of multiple characters in the reference character sequence. Therefore, based on the tagging result of each audio frame by the boundary awareness model, the dimensions of the feature vectors H and G input into the fusion network can be reduced, achieving the effect of pruning to reduce the consumption of storage space and computing power and improve the model training speed. Moreover, the above tagging result is equivalent to limiting the range of each audio frame corresponding to each character, or the range of the characters that each audio frame may correspond to, and based on the limitation, the transmission delay can be reduced.

[0061] Figure 3 A flowchart of a speech recognition model training method is provided for an embodiment of the application, wherein the speech recognition model is, for example, an RNN-T model, and specifically includes an encoding network, a prediction network, and a fusion network. As shown, Figure 3 The method includes:

[0062] 301. Obtain multiple audio frames corresponding to a speech sample and a reference character sequence, wherein the reference character sequence includes multiple characters corresponding to the speech sample.

[0063] 302. Determine multiple first feature vectors corresponding to the multiple audio frames according to the encoding network, and determine multiple second feature vectors corresponding to the multiple characters according to the prediction network.

[0064] 303. Determine the correspondence between the multiple audio frames and the multiple characters according to the multiple first feature vectors and a boundary awareness model, wherein the boundary awareness model is used to determine the speech boundaries of the multiple characters, and each audio frame corresponds to a continuous number of characters, and the number is less than the number of multiple characters.

[0065] 304. Determine first to-be-fused feature vectors corresponding to the multiple audio frames and second to-be-fused feature vectors corresponding to the multiple characters according to the correspondence, the multiple first feature vectors, and the multiple second feature vectors, wherein the dimensions of the first to-be-fused feature vectors and the second to-be-fused feature vectors are determined according to the number of the multiple first feature vectors and the number.

[0066] 305. Input the first to-be-fused feature vectors and the second to-be-fused feature vectors into the fusion network to determine a target loss according to the prediction output information of the fusion network and the reference character sequence, and update the parameters of the speech recognition model and the boundary awareness model based on the target loss.

[0067] In the actual training process, first, a plurality of audio frames corresponding to a speech sample and a reference text sequence are obtained, the plurality of audio frames are input into the encoding network to obtain a plurality of first feature vectors output by the encoding network; and a plurality of texts corresponding to the speech sample in the reference text sequence are input into the prediction network to obtain a plurality of second feature vectors output by the prediction network. Wherein, the plurality of first feature vectors can be represented as h={h1,h2,…,h T}, and similarly, the plurality of second feature vectors can also be represented as g={g1,g2,…,g U}. Wherein, it is assumed that the number of the plurality of first feature vectors is T, and the number of the plurality of second feature vectors is U (i.e. the total number of texts in the reference text sequence).

[0068] It should be noted that the number of first feature vectors is not necessarily equal to the number of audio frames. For example, it is assumed that 100 audio frames are obtained in total, if the encoding network does not perform down-sampling on the acoustic features of the 100 audio frames input thereto, then the number of first feature vectors is 100. If the encoding network performs down-sampling (for example, 2 times down-sampling) on the acoustic features of the 100 audio frames input thereto, then the number of first feature vectors is 50. Compared with the number of first feature vectors, the number of second feature vectors is relatively regular, which generally corresponds to the number of texts in the reference text sequence. For example, it is assumed that the reference text sequence includes 5 texts and one empty character set at the start position, then U=6, and 6 second feature vectors are output by the prediction network.

[0069] After obtaining the plurality of first feature vectors, the plurality of first feature vectors are input into the boundary awareness model to determine the correspondence between the plurality of audio frames and the plurality of texts. In simple terms, it is determined which audio frames correspond to a word. It is assumed that there are 10 audio frames, wherein the first 3 audio frames correspond to the first word, the fourth 6 audio frames correspond to the second word, and the seventh 10 audio frames correspond to the third word. Or, 1 audio frame data can be divided into two, so that the first 3 audio frames correspond to the first word, the third 6 audio frames correspond to the second word, and the sixth 10 audio frames correspond to the third word. For the specific determination process of the correspondence between the plurality of audio frames and the plurality of texts, please refer to the subsequent embodiments, which will not be introduced here. In the embodiments of the present application, the boundary awareness model can include a first network for determining the speech boundary, such as a CIF model, and can also include a second network for text prediction (which will be introduced below).

[0070] It should be noted that the process of determining the correspondence between multiple audio frames and multiple texts is specifically summarized into the following two stages: First, input multiple first feature vectors into the boundary perception model to determine the texts initially corresponding to each of the multiple audio frames, where one text corresponds to one audio frame; Second, according to the preset set number, for any audio frame, based on the text initially corresponding to it, determine the set number of texts (including the text initially corresponding to it) whose positions are consecutive with the position of the text initially corresponding to it in the reference text sequence, and establish the correspondence between the any audio frame and the set number of texts.

[0071] For example, assume that the boundary perception model determines that the text initially corresponding to the first audio frame is the first word. If the set number S = 1, then the final correspondence determination result at this time is: the first audio frame corresponds to the first word; if the set number S = 2, then the final correspondence determination result at this time can be: the first audio frame corresponds to the first word and the second word. Assume that the length of the reference text sequence is U as described above, then S < U. After determining the correspondence between multiple audio frames and multiple texts, that is, determining the correspondence between h and g, the first fusion feature vectors corresponding to multiple audio frames and the second fusion feature vectors corresponding to multiple texts can be determined according to the correspondence, multiple first feature vectors, and multiple second feature vectors. For example, as Figure 2 shown, assume that the first feature vector h = {h1, h2, h3, h4}, the second feature vector g = {g1, g2, g3}, the set number S is 1, and the determined correspondence is h1: g1, h2: g1, h3: g2, h4: g3 (the colon indicates that there is a correspondence between the two before and after it). At this time, copy {h1, h2, h3, h4} S times, that is, copy 1 time, and it will become the following 4-row 1-column matrix (i.e., the first fusion feature vector): H * ={{h1, h2, h3, h4}}, based on the correspondence, the following 4-row 1-column matrix (i.e., the second fusion feature vector) is obtained: G * ={{g1}, {g1}, {g2}, {g3}}, it can be seen that H * and G * are both 4-row 1-column matrices and can be directly added in the fusion network. Another example, if the set number S is 2 and the determined correspondence is h1: (g1, g2), h2: (g1, g2), h3: (g2, g3), h4: (g2, g3), at this time, copy {h1, h2, h3, h4} S times, that is, copy 2 times, and it will become the following 4-row 2-column matrix (i.e., the first fusion feature vector): H * ={{h1, h1}, {h2, h2}, {h3, h3}, {h4, h4}}, based on this correspondence, the following 4-row 2-column matrix (i.e., the second fusion feature vector) is obtained: G* {{g1, g2}, {g1, g2}, {g2, g3}, {g3, g4}}, it can be seen that, H * And G are both 4 rows and 2 columns matrix, can be added directly in the fusion network.

[0072] According to the above examples, the final purpose of the application is to cut the original h and g corresponding T*U size matrix into T*S size matrix, and the actual cutting process can be summarized as follows: according to the set number, a plurality of first feature vectors are copied to generate a plurality of first to-be-fused feature vectors corresponding to the audio frames; according to the set number of second feature vectors corresponding to each of the plurality of first feature vectors, a plurality of second to-be-fused feature vectors corresponding to the plurality of texts are generated. In specific implementation, the first feature vector h or the transpose of h is copied S times (determine whether h is a row vector or a column vector, if it is a row vector, it is transposed, and if it is a column vector, it is original h) to obtain H * And for the second feature vector g, actually no copy operation is needed, but according to the plurality of first feature vectors and the boundary perception model, the corresponding relationship between the plurality of audio frames and the plurality of texts is determined, and based on the corresponding relationship, the second to-be-fused feature vector G * .

[0073] After the first to-be-fused feature vector H * And the second to-be-fused feature vector G * are input into the fusion network for fusion processing, the text sequence predicted by the fusion network can be obtained, based on the prediction result and the reference text sequence, the forward-backward algorithm can be used to calculate the target loss, and the parameters of the speech recognition model and the boundary perception model are updated based on the target loss.

[0074] Compared with the training process of the traditional RNN-T model, it can be found that in the embodiment of the application, the dimensions of the two input feature vectors of the fusion network are cut to T*S, so the loss of the RNN-T model is actually calculated on a grid with the shape of [N*T*S*V], and the calculation amount is reduced.

[0075] Moreover, based on the positioning result of the speech boundary of each word by the boundary perception model, the corresponding text is labeled for each audio frame, so that the fusion network of the speech recognition model considers only the feature vectors of the audio frames within the limited field of view (set number of audio frames) for the prediction of each word, without considering the feature vectors of all audio frames, thereby reducing the transmission delay.

[0076] Based on the above, the speech recognition model training method provided by the embodiment of the present application introduces a boundary perception model to assist in training the speech recognition model, which includes an encoding network, a prediction network and a fusion network. Specifically, first, a plurality of audio frames corresponding to a speech sample and a reference text sequence corresponding to the speech sample are obtained, then a plurality of first feature vectors corresponding to the plurality of audio frames are determined according to the encoding network, and a plurality of second feature vectors corresponding to the plurality of texts are determined according to the prediction network. According to the plurality of first feature vectors and the boundary perception model, the correspondence between the plurality of audio frames and the plurality of texts can be determined. The boundary perception model is used to determine the speech boundary of the plurality of texts, and finally a continuous set number of texts corresponding to each audio frame in the reference text sequence is determined, which is less than the number of the plurality of texts. In fact, the boundary perception model may determine that a certain audio frame corresponds to a certain word in the reference text sequence, but considering that the recognition result of the boundary perception model is not necessarily very accurate, some redundancy is added, and after determining that the other several words continuous with the position of this word as the label of the audio frame, according to the correspondence between each audio frame and the set number of texts, the plurality of first feature vectors and the plurality of second feature vectors, the first to-be-fused feature vectors corresponding to the plurality of audio frames and the second to-be-fused feature vectors corresponding to the plurality of texts input into the fusion network are determined, wherein the dimensions of the first to-be-fused feature vectors and the second to-be-fused feature vectors are determined according to the number of the plurality of first feature vectors and the set number, so that the dimension of the feature vector input into the fusion network is reduced, not only the storage space occupation is reduced, but also the calculation amount of the fusion network is reduced, thereby reducing the computing power consumption of the training process and improving the training speed.

[0077] Figure 4 For the embodiment of the present application, a method flow chart for determining the correspondence between the plurality of audio frames and the plurality of texts is shown in Figure 4 The specific steps of the method include:

[0078] 401, input the plurality of first feature vectors into the boundary perception model to determine the respective weights of the plurality of first feature vectors.

[0079] 402, according to the order of the plurality of first feature vectors, traverse the respective weights of the first feature vectors, to determine the target text corresponding to each first feature vector according to the weight cumulative sum and the set threshold according to the order of the plurality of texts, wherein each first feature vector corresponds to a target text.

[0080] 403, determine the set number of second feature vectors corresponding to each first feature vector, wherein the set number of second feature vectors corresponding to any first feature vector includes the second feature vectors corresponding to the set number of texts adjacent to the position of the target text corresponding to the first feature vector.

[0081] For the convenience of understanding, the present embodiment is described below in combination with Figure 5 the voice boundary generation diagram shown in the following figure:

[0082] In Figure 5 , the "circle" in the first row refers to the prediction output information of the boundary awareness model, the "box" in the second row refers to the plurality of first feature vectors output by the encoding network, the data in the third row represents the respective weights corresponding to the plurality of first feature vectors, the data in the fourth row represents the weight cumulative sum generated in the order of the plurality of first feature vectors, the data in the fifth row represents the comparison strategy of the weight cumulative sum and the set threshold, wherein the set threshold is assumed to be 1, and the data in the sixth row represents the serial number of the word, i.e., the first word, the second word, and the third word.

[0083] As described above, the boundary awareness model includes a first network (such as a CIF model) for determining the voice boundary and a second network (such as a linear layer and a softmax layer connected to the output end of the first network) for performing word prediction. At this time, the plurality of first feature vectors output by the encoding network is input into the first network in the boundary awareness model to obtain the respective weights corresponding to the plurality of first feature vectors output by the first network, which is assumed to be as shown in Figure 5 .

[0084] Subsequently, the respective weights corresponding to the first feature vectors are traversed in the order of the plurality of first feature vectors to determine the target word corresponding to the plurality of first feature vectors respectively according to the size comparison result of the weight cumulative sum and the set threshold. Assuming that the reference word sequence is {I love to eat roast duck}, based on the respective weights corresponding to the eight first feature vectors shown in Figure 5 , it is found that the sum of the weights corresponding to the first four first feature vectors satisfies the condition of being greater than or equal to 1 (the set threshold is set to 1), which indicates that the target words corresponding to the first three first feature vectors are the first word in the reference word sequence: I. Therefore, the labels given to the first three first feature vectors are the first word. The fourth first feature vector is partially corresponding to the first word, wherein the weight 0.3 corresponding to the fourth first feature vector is divided into two parts: the weight cumulative sum 1.2-threshold 1=0.2, and the 0.2 part is divided to the next word, and the remaining weight 0.3-0.2=0.1 of the fourth first feature vector is divided to the first word. From the perspective of finally determining the word (label) corresponding to the fourth first feature vector, as shown in Figure 5 , because it belongs to the next word with a greater weight proportion, it is determined that the target word (i.e., the label) corresponding to the fourth first feature vector is the second word: love.

[0085] Subsequently, as shown in Figure 5As shown in the table, starting from the weight value 0.2 corresponding to the fourth first feature vector, the weights of the subsequent first feature vectors are continuously added. Since the weight corresponding to the fifth first feature vector is 0.6, the addition result 0.8 does not reach the set threshold 1, and the weight corresponding to the sixth first feature vector is continuously added. The addition result 1.3 is greater than the set threshold 1, so it is determined that the fourth to sixth first feature vectors correspond to the second word in the reference text sequence: love. In this way, the first network determines the phonetic boundary of each word. Figure 5 Only part of the first feature vectors and part of the words in the reference text sequence are shown in the table.

[0086] The above is the process of the first network positioning the phonetic boundary of each word. As can be seen, in fact, determining the target word corresponding to each audio frame is to determine the position index of the target word corresponding to each first feature vector in the reference text sequence.

[0087] Then, according to the pre-set number S of labels that each audio frame should be labeled, the S labels (i.e., S words) corresponding to each first feature vector are finally determined. Assuming that S = 1, the target word corresponding to each first feature vector determined above is the final determination result. Assuming that S = 2, for each first feature vector, the positionally adjacent words are determined to also correspond to the corresponding target word based on the initial determination. Since each word is represented by its corresponding second feature vector, the correspondence between each first feature vector and the S second feature vectors can be finally determined. Moreover, it can be understood that the S second feature vectors corresponding to the previous first feature vector and the S second feature vectors corresponding to the next first feature vector are positionally continuous, and the situation that the previous first feature vector corresponds to the second and third word second feature vectors, but the next first feature vector corresponds to the first and third word second feature vectors does not occur.

[0088] It should be understood that in the traditional RNN-T model, since there is no any restriction, any first feature vector can correspond to any second feature vector, so the situation that the user has said the word corresponding to g1 at time t1 corresponding to the first audio frame, but the RNN-T model outputs the word corresponding to g1 at time t4 corresponding to the fourth audio frame, with a large delay, can occur. In the embodiment of the present application, through the above labeling method, the "streaming" recognition effect of the voice can be achieved, and the delay is low. For example, the RNN-T model trained based on the above labeling result can quickly output the text recognition result as the voice data of one audio frame is input, or can quickly output the text recognition result as the voice data of the first three audio frames is input, with strong real-time performance.

[0089] In addition, it should be noted that the boundary awareness model includes a first network for determining the speech boundary and a second network for performing text prediction, and the weight corresponding to each of the plurality of first feature vectors is output by the first network. The first network can be a CIF model, and the second network can include a linear layer and a Softmax. Based on this, the method provided by the embodiment of the present application further includes: inputting the weight corresponding to each of the plurality of first feature vectors and the plurality of first feature vectors into the second network to obtain the prediction output information of the second network. The prediction output information includes two types of data: the first type is the total number of predicted texts, that is, the number of texts contained in the plurality of input audio frames is predicted; and the second type is a probability distribution matrix representing what each text is. In fact, the second network outputs the predicted text sequence information. For example Figure 6 As shown in the first row, based on the weight corresponding to each first feature vector that is determined again after the speech boundary is located, the prediction output information of the second network in the boundary awareness model indicates that there are a total of 3 texts, and what each text is.

[0090] On the basis of the prediction output information of the second network, the prediction output information of the fusion network and the reference text sequence are combined to determine the target loss. The target loss includes three types: the first type is the RNN-T loss determined by using the forward and backward algorithms according to the predicted text sequence output by the fusion network and the reference text sequence, the second type is the text quantity loss calculated according to the total number of predicted texts and the total number of texts in the reference text sequence, and the third type is the CE loss calculated according to the predicted text sequence output by the second network in the boundary awareness model and the reference text sequence. After the target loss is determined, the parameters of the speech recognition model and the boundary awareness model can be updated based on the target loss to complete the training of the speech recognition model.

[0091] From the training process, the second network of the boundary awareness model actually outputs a probability distribution matrix of [N*U*V], which is used to calculate the CE loss. Each point in the matrix represents the probability that the u-th word in the current speech sample n in the current batch of speech samples belongs to the v-th word in the vocabulary, where n is the n-th in the current batch of speech samples with a size of N, u is the u-th in the U texts in the current reference text sequence, and v is the v-th word in the vocabulary V.

[0092] The application process of the trained speech recognition model is described below:

[0093] Figure 6 The flowchart of the speech recognition method provided by the embodiment of the present application is shown in Figure 7 The method includes the following steps:

[0094] 601, obtaining the speech to be recognized.

[0095] 602. Input the speech to be recognized into the speech recognition model to obtain the target text corresponding to the speech to be recognized; wherein, the speech recognition model is trained by the speech recognition model training method described above.

[0096] In practical applications, the first step is to acquire the speech to be recognized. This speech can be uttered by the user or by an electronic device with a speech function. For example, the speech to be recognized could be "I'm going to eat." The speech "I'm going to eat" is then input into a trained speech recognition model to obtain the corresponding text.

[0097] It should be understood that since the speech recognition model is trained based on the speech recognition model training method provided in the above embodiments, the speech recognition model obtained by the above training method has the performance of low transmission delay. Therefore, using the speech recognition model to perform the recognition processing of the speech to be recognized will achieve the effect of low transmission delay.

[0098] It should be noted that the speech recognition method provided in this embodiment of the invention can be applied to both offline speech recognition scenarios and streaming speech recognition scenarios. In practical applications, optionally, the input method of the speech to be recognized to the speech recognition model can be similar in both offline and streaming speech recognition scenarios.

[0099] Optionally, the speech to be recognized is input into the speech recognition model, including:

[0100] Determine multiple consecutive audio frames corresponding to the speech to be recognized, and input these multiple audio frames into the encoding network of the speech recognition model to obtain multiple feature vectors corresponding to the multiple audio frames.

[0101] The target input character is input into the prediction network of the speech recognition model to obtain the feature vector corresponding to the target input character. The target input character is updated with the output character of the fusion network of the speech recognition model. Initially, the target input character is a set empty character, and the output character of the fusion network is an empty character or the predicted text.

[0102] Based on the order of multiple audio frames, multiple feature vectors corresponding to multiple audio frames are traversed. The feature vector corresponding to the currently traversed audio frame and the feature vector corresponding to the target input character are input into the fusion network to determine the fusion network's prediction result for the text corresponding to the currently traversed audio frame.

[0103] In practice, the speech to be recognized is not input into the encoding network frame by frame, but in blocks. For example, if a block contains 10 audio frames, the acoustic features of the 10 consecutively generated audio frames are input into the encoding network together.

[0104] Different from the training process, in the model using stage, in the initial state, the empty character is input into the prediction network, and then, whenever the fusion network outputs a character, the character is input into the prediction network in a self-recurrent manner. In other words, for the prediction network, the training process and the application process are different. In the training process, the reference character sequence is known data, while in the actual application, the target input character input into the prediction network is unknown and needs to be obtained from the output of the fusion network.

[0105] The speech recognition process can be regarded as a process of recognizing the corresponding character for each audio frame in the current input. In the process of recognizing each audio frame, some audio frames may not recognize any results, and then the output of the fusion network for the audio frames without any recognition results is an empty character. Because in actual application, not every audio frame corresponds to a character, and often multiple audio frames correspond to a character, only after all the audio frames corresponding to the same character are recognized, the corresponding character is output.

[0106] For ease of understanding, for example, it is assumed that after a "block" of acoustic feature is input into the encoding network, the feature vector h1-h 10 , that is, a "block" of acoustic features contains acoustic features of 10 audio frames. In the initial state, an empty character is input into the prediction network to obtain the feature vector g0output by the prediction network, and then the prediction is performed on each audio frame: the feature vector h1+g0corresponding to the first audio frame is input into the fusion network, and the result is assumed to be an empty character. Then, the feature vector h2+g0corresponding to the next audio frame is input into the fusion network, and the result is still assumed to be an empty character. At this time, the feature vector h3+g0corresponding to the next audio frame is input into the fusion network, and it is assumed that the first character output by the fusion network is obtained. The first character is input into the prediction network to obtain the feature vector g1output by the prediction network, and then the feature vector h4+g1corresponding to the next audio frame is input into the fusion network, and the result is assumed to be an empty character. The feature vector h5+g1corresponding to the next audio frame is input into the fusion network, and the result is an empty character. The feature vector h6+g1corresponding to the next audio frame is input into the fusion network, and it is assumed that the second character output by the fusion network is obtained. The second character is input into the prediction network to obtain the feature vector g2output by the prediction network, and then the feature vector h7+g2corresponding to the next audio frame is input into the fusion network, and so on, until h 10 The recognition ends. At this time, the above operation can be repeated for the next "block" of acoustic features.

[0107] It can be seen that by using the above method to input the to-be-recognized voice into the voice recognition model, a "streaming" voice recognition effect can be realized, the delay is low, and the real-time performance of voice recognition is good.

[0108] In addition, the voice recognition model training method provided by the embodiment of the present application can be executed in the cloud. A plurality of computing nodes (cloud servers) can be deployed in the cloud, and each computing node has computing, storage, and other processing resources. In the cloud, a plurality of computing nodes can be organized to provide a certain service. Of course, one computing node can also provide one or more services. The cloud can provide a service interface for the service (model training service), and the user can call the service interface through the terminal device to trigger a model training service request to the cloud. The request includes type indication information of the voice recognition model to be trained (such as training an RNN-T model). According to the computing resources corresponding to the model training service, the following steps are performed:

[0109] According to the scheme provided by the embodiment of the present application, the cloud can provide a service interface for a set service (model training service), and the user can call the service interface through the terminal device to trigger a model training service request to the cloud. The request includes type indication information of the voice recognition model to be trained (such as training an RNN-T model). According to the computing resources corresponding to the model training service, the following steps are performed:

[0110] Obtain a plurality of audio frames corresponding to a voice sample and a reference text sequence, the reference text sequence including a plurality of texts corresponding to the voice sample; the voice recognition model includes an encoding network, a prediction network, and a fusion network;

[0111] According to the encoding network, a plurality of first feature vectors corresponding to the plurality of audio frames are determined, and according to the prediction network, a plurality of second feature vectors corresponding to the plurality of texts are determined;

[0112] According to the plurality of first feature vectors and the boundary perception model, a corresponding relationship between the plurality of audio frames and the plurality of texts is determined, wherein the boundary perception model is used to determine the voice boundaries of the plurality of texts, each audio frame corresponds to a continuous set number of texts, and the set number is less than the number of the plurality of texts;

[0113] According to the corresponding relationship, the plurality of first feature vectors, and the plurality of second feature vectors, a first to-be-fused feature vector corresponding to the plurality of audio frames and a second to-be-fused feature vector corresponding to the plurality of texts are determined, and the dimensions of the first to-be-fused feature vector and the second to-be-fused feature vector are determined according to the number of the plurality of first feature vectors and the set number;

[0114] The first to-be-fused feature vector and the second to-be-fused feature vector are input into the fusion network to determine a target loss according to the prediction output information of the fusion network and the reference text sequence, and the parameters of the voice recognition model and the boundary perception model are updated based on the target loss;

[0115] The trained voice recognition model is sent to the terminal device.

[0116] The above execution process can refer to the relevant description in the foregoing other embodiments, and details are not described herein.

[0117] For the sake of understanding, exemplary description is made in conjunction with Figure 7 The user can invoke the model training service through the terminal device E1 as shown in Figure 7 The service interface for the user to invoke the service includes a software development kit (SDK), an application programming interface (API), and the like. Figure 8 The API interface is shown in the foregoing embodiment. In the cloud, as shown in the figure, it is assumed that the model training service is provided by a service cluster E2, and the service cluster E2 includes at least one computing node. After the service cluster E2 receives the request, the steps in the foregoing embodiment are executed to obtain the trained voice recognition model, and the trained voice recognition model is fed back to the terminal device E1.

[0118] The communication device of one or more embodiments of the application will be described in detail below. Those skilled in the art can understand that these devices can be configured by using commercially available hardware components through the steps taught by the present solution.

[0119] Figure 8 A structural schematic diagram of a voice recognition model training device provided by an embodiment of the application is shown in the figure. The voice recognition model includes an encoding network, a prediction network, and a fusion network. The device includes an acquisition module 11, a first determination module 12, a second determination module 13, a third determination module 14, and a parameter update module 15. Figure 9

[0120] The acquisition module 11 is configured to acquire a plurality of audio frames corresponding to a voice sample and a reference text sequence, and the reference text sequence includes a plurality of texts corresponding to the voice sample.

[0121] The first determination module 12 is configured to determine a plurality of first feature vectors corresponding to the plurality of audio frames according to the encoding network, and determine a plurality of second feature vectors corresponding to the plurality of texts according to the prediction network.

[0122] The second determination module 13 is configured to determine a correspondence between the plurality of audio frames and the plurality of texts according to the plurality of first feature vectors and a boundary awareness model, wherein the boundary awareness model is configured to determine voice boundaries of the plurality of texts, and each audio frame corresponds to a continuous set number of texts, and the set number is less than the number of the plurality of texts.

[0123] ​The third determining module 14 is configured to determine first to-be-fused feature vectors corresponding to the plurality of audio frames and second to-be-fused feature vectors corresponding to the plurality of texts according to the correspondence, the plurality of first feature vectors and the plurality of second feature vectors, wherein the dimensions of the first to-be-fused feature vectors and the second to-be-fused feature vectors are determined according to the number of the plurality of first feature vectors and a set number.

[0124] The parameter updating module 15 is configured to input the first to-be-fused feature vectors and the second to-be-fused feature vectors into the fusion network to determine a target loss according to predicted output information of the fusion network and a reference text sequence, and update parameters of the speech recognition model and the boundary awareness model based on the target loss.

[0125] Optionally, the second determining module 13 is specifically configured to: input the plurality of first feature vectors into the boundary awareness model to determine weights corresponding to the plurality of first feature vectors respectively; traverse the weights corresponding to the plurality of first feature vectors respectively according to the order of the plurality of first feature vectors to determine target texts corresponding to the plurality of first feature vectors respectively according to the weights, the set threshold and the order of the plurality of texts, wherein each first feature vector corresponds to a target text; and determine a set number of second feature vectors corresponding to each first feature vector, wherein the set number of second feature vectors corresponding to any first feature vector include second feature vectors corresponding to a set number of texts adjacent to a position of the target text corresponding to the first feature vector.

[0126] Optionally, the third determining module 14 is specifically configured to: replicate the plurality of first feature vectors according to the set number to generate the first to-be-fused feature vectors corresponding to the plurality of audio frames; and generate the second to-be-fused feature vectors corresponding to the plurality of texts according to the set number of second feature vectors corresponding to the plurality of first feature vectors respectively.

[0127] Optionally, the boundary awareness model includes a first network for determining a speech boundary and a second network for text prediction; the weights corresponding to the plurality of first feature vectors are output by the first network; and the device further includes:

[0128] The information obtaining module is configured to input the weights corresponding to the plurality of first feature vectors and the plurality of first feature vectors into the second network to obtain predicted output information of the second network; and the parameter updating module 15 is specifically configured to determine the target loss according to the predicted output information of the fusion network, the predicted output information of the second network and the reference text sequence.

[0129] Optionally, the speech recognition model includes an RNN-T model, and the boundary awareness model includes a CIF model.

[0130] Figure 9 A structural schematic diagram of a speech recognition device provided by an embodiment of the present application is shown in FIG. 1. Figure 8As shown, the apparatus comprises a voice acquisition module 21 and a text acquisition module 22.

[0131] The voice acquisition module 21 is configured to acquire the voice to be recognized.

[0132] The text acquisition module 22 is configured to input the voice to be recognized into a speech recognition model to obtain target text corresponding to the voice to be recognized, wherein the speech recognition model is obtained by training a speech recognition model training method.

[0133] Optionally, the text acquisition module 22 is specifically configured to: determine a plurality of continuous audio frames corresponding to the voice to be recognized; input the plurality of audio frames into an encoding network of the speech recognition model to obtain a plurality of feature vectors corresponding to the plurality of audio frames; input a target input character into a prediction network of the speech recognition model to obtain a feature vector corresponding to the target input character, wherein the target input character is updated according to an output character of a fusion network of the speech recognition model, and the target input character is an empty character at the beginning, and the output character of the fusion network is an empty character or a predicted word; according to the order of the plurality of audio frames, traverse the plurality of feature vectors corresponding to the plurality of audio frames to input a feature vector corresponding to a currently traversed audio frame and a feature vector corresponding to the target input character into the fusion network to determine a word prediction result of the currently traversed audio frame by the fusion network.

[0134] Figure 9 and Figure 10 The apparatus shown can perform the steps in the foregoing embodiments, and the detailed execution process and technical effects are described in the foregoing embodiments, which will not be described here.

[0135] The embodiment of the present application also provides an electronic device, such as ​ As shown, the electronic device can comprise a processor 31, a memory 32, and a communication interface 33. The memory 32 stores executable code, and when the executable code is executed by the processor 31, the processor 31 implements the speech recognition model training method or the speech recognition method as described in the foregoing embodiments.

[0136] In addition, the embodiment of the present application provides a non-transitory machine readable storage medium, and the non-transitory machine readable storage medium stores executable code, and when the executable code is executed by the processor of the electronic device, the processor can at least implement the speech recognition model training method or the speech recognition method as described in the foregoing embodiments.

[0137] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate. Part or all of the modules can be selected according to actual needs to achieve the purposes of the embodiments. Those skilled in the art can understand and implement without creative labor.

[0138] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of the necessary general hardware platform, and of course can also be realized by means of the combination of hardware and software. Based on such understanding, the above technical solutions can be embodied in the form of computer products, and the present application can be embodied in the form of computer program products implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0139] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training a speech recognition model, characterized in that, The speech recognition model includes an encoding network, a prediction network, and a fusion network; the method includes: Obtain multiple audio frames and a reference text sequence corresponding to the speech sample, wherein the reference text sequence includes multiple texts corresponding to the speech sample; The encoding network determines multiple first feature vectors corresponding to the multiple audio frames, and the prediction network determines multiple second feature vectors corresponding to the multiple characters. Based on the plurality of first feature vectors and the boundary awareness model, the correspondence between the plurality of audio frames and the plurality of characters is determined, wherein the boundary awareness model is used to determine the speech boundary of the plurality of characters, and each audio frame corresponds to a set number of consecutive characters, wherein the set number is less than the number of the plurality of characters; Based on the correspondence, the plurality of first feature vectors, and the plurality of second feature vectors, a first feature vector to be fused and a second feature vector to be fused corresponding to the plurality of audio frames are determined, and the dimensions of the first feature vector to be fused and the second feature vector to be fused are determined according to the number of the plurality of first feature vectors and the set number. The first feature vector to be fused and the second feature vector to be fused are input into the fusion network to determine the target loss based on the prediction output information of the fusion network and the reference text sequence, and the parameters of the speech recognition model and the boundary awareness model are updated based on the target loss.

2. The method according to claim 1, characterized in that, The step of determining the correspondence between the multiple audio frames and the multiple texts based on the multiple first feature vectors and the boundary awareness model includes: The plurality of first feature vectors are input into the boundary-aware model to determine the weights corresponding to each of the plurality of first feature vectors. According to the order of the plurality of first feature vectors, the weights corresponding to each of the first feature vectors are traversed, and the target text corresponding to the plurality of first feature vectors is determined according to the weight sum and a set threshold, and according to the order of the plurality of texts, wherein each first feature vector corresponds to one target text. Determine the predetermined number of second feature vectors corresponding to each of the plurality of first feature vectors, wherein the predetermined number of second feature vectors corresponding to any first feature vector includes the second feature vectors corresponding to a predetermined number of characters adjacent to the target text position corresponding to any first feature vector.

3. The method according to claim 2, characterized in that, The step of determining the first feature vector to be fused corresponding to the plurality of audio frames and the second feature vector to be fused corresponding to the plurality of texts based on the correspondence, the plurality of first feature vectors, and the plurality of second feature vectors includes: The plurality of first feature vectors are copied according to the set number to generate the first feature vectors to be fused corresponding to the plurality of audio frames; Based on the predetermined number of second feature vectors corresponding to each of the plurality of first feature vectors, a second feature vector to be fused corresponding to the plurality of characters is generated.

4. The method according to claim 2, characterized in that, The boundary-aware model includes a first network for determining speech boundaries and a second network for text prediction; the weights corresponding to the plurality of first feature vectors are output by the first network; the method further includes: The weights corresponding to each of the plurality of first feature vectors and the plurality of first feature vectors are input into the second network to obtain the prediction output information of the second network; The step of determining the target loss based on the predicted output information of the fusion network and the reference text sequence includes: The target loss is determined based on the prediction output information of the fusion network, the prediction output information of the second network, and the reference text sequence.

5. The method according to any one of claims 1 to 4, characterized in that, The speech recognition model includes an RNN-T model, and the boundary awareness model includes a CIF model.

6. A speech recognition method, characterized in that, include: Acquire the speech to be recognized; The speech to be recognized is input into a speech recognition model to obtain the target text corresponding to the speech to be recognized; wherein the speech recognition model is trained by any one of the speech recognition model training methods of claims 1 to 5.

7. The method according to claim 6, characterized in that, The step of inputting the speech to be recognized into the speech recognition model includes: Determine multiple consecutive audio frames corresponding to the speech to be recognized; The multiple audio frames are input into the encoding network of the speech recognition model to obtain multiple feature vectors corresponding to the multiple audio frames; The target input character is input into the prediction network of the speech recognition model to obtain the feature vector corresponding to the target input character. The target input character is updated with the output character of the fusion network of the speech recognition model. Initially, the target input character is a set empty character, and the output character of the fusion network is the empty character or the predicted text. Based on the order of the multiple audio frames, the multiple feature vectors corresponding to the multiple audio frames are traversed, and the feature vector corresponding to the currently traversed audio frame and the feature vector corresponding to the target input character are input into the fusion network to determine the text prediction result of the fusion network for the currently traversed audio frame.

8. A method for training a speech recognition model, characterized in that, The method includes: The receiving terminal device triggers a request by calling the model training service, the request including type indication information of the speech recognition model to be trained; Based on the computing resources corresponding to the model training service, the following steps are performed: The speech recognition model acquires multiple audio frames corresponding to the speech samples and a reference text sequence, wherein the reference text sequence includes multiple characters corresponding to the speech samples; the speech recognition model includes an encoding network, a prediction network, and a fusion network. The encoding network determines multiple first feature vectors corresponding to the multiple audio frames, and the prediction network determines multiple second feature vectors corresponding to the multiple characters. Based on the plurality of first feature vectors and the boundary awareness model, the correspondence between the plurality of audio frames and the plurality of characters is determined, wherein the boundary awareness model is used to determine the speech boundary of the plurality of characters, and each audio frame corresponds to a set number of consecutive characters, wherein the set number is less than the number of the plurality of characters; Based on the correspondence, the plurality of first feature vectors, and the plurality of second feature vectors, a first feature vector to be fused and a second feature vector to be fused corresponding to the plurality of audio frames are determined, and the dimensions of the first feature vector to be fused and the second feature vector to be fused are determined according to the number of the plurality of first feature vectors and the set number. The first feature vector to be fused and the second feature vector to be fused are input into the fusion network to determine the target loss based on the prediction output information of the fusion network and the reference text sequence, and the parameters of the speech recognition model and the boundary awareness model are updated based on the target loss. The trained speech recognition model is sent to the terminal device.

9. An electronic device, characterized in that, include: The system includes a memory, a processor, and a communication interface; wherein the memory stores executable code, which, when executed by the processor, causes the processor to perform the speech recognition model training method as described in any one of claims 1 to 5, or to perform the speech recognition method as described in claim 6 or 7.

10. A non-transitory machine-readable storage medium, characterized in that, The non-transitory machine-readable storage medium stores executable code that, when executed by a processor of an electronic device, causes the processor to perform the speech recognition model training method as described in any one of claims 1 to 5, or to perform the speech recognition method as described in claim 6 or 7.

Images