Acoustic model training method and device, equipment and storage medium

Abstract

The present disclosure provides an acoustic model training method and device, equipment and a storage medium, relating to the technical field of computers, in particular to the technical field of artificial intelligence such as audio processing, speech recognition and model training. The specific implementation scheme is: adding layer normalization processed absolute position encoding in the first feature data corresponding to the sample audio through the acoustic model to obtain the second feature data; adding layer normalization processed rotation position encoding after processing the second feature data through the acoustic model to obtain the third feature data; and training the acoustic model according to the third feature data. In the embodiment of the present disclosure, the generalization ability of the acoustic model can be improved by layer normalization processing of the position encoding, and the accuracy of audio processing can be improved by using the trained acoustic model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the field of artificial intelligence technology, such as audio processing, speech recognition, and model training. Background Technology

[0002] Acoustic models are a core module in current speech recognition services. In end-to-end speech recognition solutions, the results of the acoustic model can be used as the final result of the entire speech recognition service. Therefore, the results of the acoustic model essentially determine the final performance of the entire speech recognition service system. Current acoustic models are mainly trained using self-attention-based transformer neural network models, which offer significant performance improvements compared to other neural networks. However, transformer-based neural networks cannot model the positional information of speech sequences. Summary of the Invention

[0003] This disclosure provides an acoustic model training method, apparatus, device, and storage medium.

[0004] According to one aspect of this disclosure, an acoustic model training method is provided, comprising:

[0005] By adding the absolute position code after layer normalization to the first feature data corresponding to the sample audio through an acoustic model, the second feature data is obtained.

[0006] After processing the second feature data using the acoustic model, a rotational position encoding after layer normalization is added to obtain the third feature data;

[0007] The acoustic model is trained based on this third feature data.

[0008] According to another aspect of this disclosure, an audio processing method is provided, comprising:

[0009] By adding the absolute position code after layer normalization to the first feature data corresponding to the audio to be processed through the acoustic model, the second feature data is obtained.

[0010] The third feature data is obtained by processing the second feature data using the acoustic model and then adding a layer of normalized rotational position encoding.

[0011] The third feature data is processed using this acoustic model to obtain the audio processing result.

[0012] According to another aspect of this disclosure, an acoustic model training apparatus is provided, comprising:

[0013] The absolute position encoding module is used to add the layer-normalized absolute position encoding to the first feature data corresponding to the sample audio through the acoustic model to obtain the second feature data.

[0014] The rotation position encoding module is used to process the second feature data through the acoustic model and then add the rotation position encoding after layer normalization to obtain the third feature data;

[0015] The training module is used to train the acoustic model based on the third feature data.

[0016] According to another aspect of this disclosure, an audio processing apparatus is provided, comprising:

[0017] The absolute position encoding module is used to add the layer-normalized absolute position encoding to the first feature data corresponding to the audio to be processed through the acoustic model to obtain the second feature data.

[0018] The rotation position encoding module is used to process the second feature data through the acoustic model and then add the rotation position encoding after layer normalization to obtain the third feature data;

[0019] The processing module is used to process the third feature data through the acoustic model to obtain the audio processing result.

[0020] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0021] At least one processor; and

[0022] The memory is communicatively connected to the at least one processor; wherein,

[0023] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods of any embodiment of the present disclosure.

[0024] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform a method according to any embodiment of this disclosure.

[0025] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a method according to any embodiment of this disclosure.

[0026] In this embodiment of the disclosure, layer normalization of the position encoding can improve the generalization ability of the acoustic model, and the accuracy of audio processing can be improved by using the trained acoustic model.

[0027] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0028] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0029] Figure 1 This is a schematic flowchart of an acoustic model training method according to an embodiment of the present disclosure;

[0030] Figure 2 This is a flowchart illustrating an acoustic model training method according to another embodiment of the present disclosure;

[0031] Figure 3 This is a flowchart illustrating an acoustic model training method according to another embodiment of the present disclosure;

[0032] Figure 4 This is a flowchart illustrating an acoustic model training method according to another embodiment of the present disclosure;

[0033] Figure 5 This is a flowchart illustrating an acoustic model training method according to another embodiment of the present disclosure;

[0034] Figure 6 This is a schematic flowchart of an audio processing method according to an embodiment of the present disclosure;

[0035] Figure 7 This is a schematic flowchart of an audio processing method according to another embodiment of the present disclosure;

[0036] Figure 8 This is a schematic diagram of the overall network structure of the conformer acoustic model based on this disclosure;

[0037] Figure 9 This is a schematic diagram of self-attention using rotational position encoding in the MHSA layer according to this disclosure;

[0038] Figure 10 This is a flowchart illustrating the training and processing procedures according to this disclosure;

[0039] Figure 11 This is a schematic diagram of the structure of an acoustic model training device according to an embodiment of the present disclosure;

[0040] Figure 12 This is a schematic diagram of the structure of an acoustic model training device according to another embodiment of the present disclosure;

[0041] Figure 13 This is a schematic diagram of the structure of an audio processing apparatus according to an embodiment of the present disclosure;

[0042] Figure 14 This is a schematic diagram of the structure of an audio processing apparatus according to another embodiment of the present disclosure.

[0043] Figure 15 This is a block diagram of an electronic device used to implement embodiments of the present disclosure. Detailed Implementation

[0044] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0045] One current approach to adding positional encoding is to add absolute positional encoding information to the input speech sequence data. For example, in the original self-attention scheme, absolute positional encoding is used in the input sequence. The dimension of the positional encoding information is the same as the dimension of the input sequence. The positional encoding is added to the input sequence information, and then fed into the self-attention algorithm for computation. The positional encoding information in the self-attention algorithm is the same as that used in the original sequence.

[0046] Another approach is to use relative positional encoding information, primarily added during the attention calculation. No additional relative positional encoding information is added to the original sequence positional encoding information. Relative positional encoding information generally performs better than absolute positional encoding information.

[0047] The acoustic model training method of this disclosure is a position encoding scheme that combines absolute position encoding and rotational position encoding. It not only adds absolute position encoding information to the original speech sequence, but also provides relative position information when calculating attention parameters.

[0048] Figure 1 This is a flowchart illustrating an acoustic model training method according to an embodiment of the present disclosure, which may include:

[0049] S101. By adding the absolute position code after layer normalization to the first feature data corresponding to the sample audio through the acoustic model, the second feature data is obtained.

[0050] S102. After processing the second feature data using the acoustic model, add the rotation position encoding after layer normalization to obtain the third feature data.

[0051] S103. Train the acoustic model based on the third feature data.

[0052] In this embodiment, an acoustic model can be used to add various positional codes to sample audio. The acoustic model can first perform layer normalization (Layer Normalization) on the absolute positional codes, and then add the layer-normalized absolute positional codes to the first feature data corresponding to the sample audio to obtain the second feature data. Layer normalization can normalize all neurons in an intermediate layer of the acoustic model. If the acoustic model is a deep neural network, let z be the net input of the l-th layer neuron. (l) Its mean μ (l) and variance σ (l)2 As shown below, where M l This represents the number of neurons in layer l.

[0053]

[0054]

[0055] An example of a layer normalization formula is shown below, where γ and β represent the parameter vectors for scaling and translation, respectively, and z (l) They have the same dimension.

[0056]

[0057] Here, ∈ can be a small value to avoid the denominator being 0, and ⊙ represents multiplication.

[0058] In this embodiment, if the net input is a preset first absolute position code and the normalized result is a second absolute position code, then adding the first feature data to the second absolute position code can yield the second feature data.

[0059] In this embodiment, if the net input is a preset first rotation position code and the normalized result is a second rotation position code, then the data after processing the second feature data through the acoustic model can be added to the second rotation position code to obtain the third feature data.

[0060] Then, the acoustic model can be used to process the third feature data until the output data is obtained. The output data can be used to train the acoustic model; training can be stopped after convergence or after a certain number of iterations. Layer normalization of the positional encoding can improve the generalization ability of the acoustic model, and a well-trained acoustic model can improve the accuracy of audio processing such as speech recognition.

[0061] Figure 2 This is a flowchart illustrating an acoustic model training method according to another embodiment of the present disclosure. The method may include one or more features of the acoustic model training method described in the above embodiment. In one implementation, S102 involves adding a layer-normalized rotational position code to the second feature data using the acoustic model to obtain third feature data, including:

[0062] S201. The second feature data is linearly transformed using the acoustic model to obtain query information, key information, and value information.

[0063] S202. After adding the rotation position code after layer normalization to the query information and the key information respectively, the attention parameters are obtained.

[0064] S203. Based on the attention parameter and the value information, the third feature data is obtained.

[0065] For example, an acoustic model may include a first network that performs preliminary processing on audio, such as sample audio; a second network that performs layer normalization on the first absolute positional encoding; and a third network that processes the data after adding one positional encoding and adds another encoding. The second feature data, obtained by adding the first feature information output by the first network and the second absolute positional encoding output by the second network, can be input into the third network. The third network performs a linear transformation on the second feature data to obtain query information, key information, and value information. Then, a layer-normalized rotational positional encoding is added to the query information and key information. Next, attention parameters, such as the attention score, are calculated, and combined with the value information, the third feature data is calculated. The third network can then process the third feature data until output data is obtained. This output data can then be used to train at least one of the first, second, and third networks. By adding a layer-normalized rotational positional encoding to the feature data of the audio that already has absolute positional encoding, the generalization ability of the acoustic model can be improved. Furthermore, by leveraging the advantages of both positional encodings, the accuracy of the trained acoustic model in audio processing, such as speech recognition, can be improved.

[0066] In one implementation, the acoustic model includes a convolutionally enhanced transformer network. After adding layer-normalized rotational position codes to the query information and the key information, attention parameters are obtained, including:

[0067] By adding layer-normalized rotational position encoding to the query information and the key information respectively using the multi-head self-attention MHSA of the Conformer network, the coded query information and the coded key information are obtained.

[0068] The attention parameters are obtained by performing matrix multiplication on the encoded query information and the encoded key information.

[0069] In one implementation, obtaining third feature data based on the attention parameter and the value information includes: performing matrix multiplication of the attention parameter and the value information to obtain the third feature data.

[0070] For example, the third network in an acoustic model can be a Conformer network. The Conformer network can process the second feature data sequentially using a first layer of normalization, a feed-forward layer, and an add layer. This processed data is then fed into a multi-headed self-attention (MHSA) layer after passing through a second normalization layer. The MHSA performs layer normalization on the rotation position encoding, and then adds the query information (represented by Q) and key information (represented by K) obtained from the feature data transformation. The encoded Q and encoded K are then multiplied by a matrix to calculate the attention score. Finally, the attention score and the value information (represented by V) are multiplied by a matrix to calculate the third feature data (i.e., the encoded V).

[0071] In this embodiment of the disclosure, the Conformer network can be used to add rotation position encoding after layer normalization to the feature data of the audio, which can not only improve the generalization ability of the acoustic model, but also improve the accuracy of the trained acoustic model.

[0072] Figure 3 This is a flowchart illustrating an acoustic model training method according to another embodiment of the present disclosure. The method may include one or more features of the acoustic model training method described in the above embodiments. In one implementation, S103 trains the acoustic model based on the third feature data, including:

[0073] S301. The third feature data is processed using the acoustic model to obtain the output data;

[0074] S302. Input the output data into the loss function to calculate the loss value of the acoustic model;

[0075] S303. Update the parameters of the acoustic model based on the loss value of the acoustic model.

[0076] For example, a Connectionist Temporal Classifier (CTC) can be used as the loss function to calculate the loss value of the acoustic model based on its output data. Therefore, the loss value can be calculated relatively flexibly.

[0077] In one implementation, S303 updates the parameters of the acoustic model based on the loss value of the acoustic model, including updating at least one of the layer normalization parameter, batch normalization parameter, and linear transformation in the acoustic model based on the loss value of the acoustic model.

[0078] For example, the first network may include parameters for layer normalization of absolute position encoding. The second network includes parameters for audio framing, feature extraction, downsampling, and batch normalization. The third network includes parameters for layer normalization of rotational position encoding, as well as parameters for linear transformation of audio feature data. During model training, some or all parameters in these networks can be updated based on the loss value corresponding to each output data. After updating, training can continue using sample audio until the loss value corresponding to a certain output data satisfies the convergence condition. By updating various parameters in the acoustic model, the acoustic model can be trained quickly to obtain an acoustic model that meets the requirements.

[0079] Figure 4 This is a flowchart illustrating an acoustic model training method according to another embodiment of the present disclosure. The method may include one or more features of the acoustic model training method described in the above embodiments. In one implementation, the method further includes:

[0080] S401. Perform frame segmentation on the sample audio to obtain audio segments;

[0081] S402. Extract audio features from the audio segment;

[0082] S403. Calculate the global cepstral mean and variance normalization (CMVN) data of the training set, which includes audio features corresponding to multiple sample audios.

[0083] S404. Perform CMVN processing on the audio features of the audio segment in the training set.

[0084] In this embodiment, the training set may include multiple sample audio files. Each sample audio file in the training set can be segmented into frames to obtain multiple audio segments. Each audio segment may include audio features of a certain dimension, such as filter bank (FBank) features. FBank is one of the speech feature parameter extraction methods, a cepstral-based extraction method that is more in line with human auditory principles. If the training set includes N sample audio files, CMVN data can be calculated based on the FBank features of these N sample audio files. Different sample audio files in the training set will result in different CMVN data. If the sample audio files in the training set are fixed, the CMVN data can remain unchanged. CMVN data can be used to perform CMVN processing on the audio features of each sample audio segment. Using CMVN data can improve the robustness of the model.

[0085] In one embodiment, S404 performs CMVN processing on the audio features of the audio segment in the training set, including: subtracting the mean from the CMVN data from the audio features of the audio segment in the training set, and then dividing by the standard deviation from the CMVN data.

[0086] For example, the CMVN data in the training set can include the mean and standard deviation. For instance, a sample audio file might consist of M audio segments. Each audio segment includes S-dimensional FBank features. The mean from the CMVN data can be subtracted from each S-dimensional FBank feature, and then divided by the standard deviation from the CMVN data. After CMVN processing, a sample audio file can include M×S-dimensional feature data. Using the mean and standard deviation of the CMVN data, acoustic features can be transformed from one space to another, making the feature parameters more consistent with a certain probability distribution, compressing the dynamic range of the feature parameter value domain, reducing the mismatch between training and testing environments, and improving the robustness of the model.

[0087] Figure 5 This is a flowchart illustrating an acoustic model training method according to another embodiment of the present disclosure. The method may include one or more features of the acoustic model training method described in the above embodiments. In one implementation, the method further includes:

[0088] S501. Downsample the feature data after CMVN processing;

[0089] S502. Perform batch normalization on the downsampled feature data;

[0090] S503. Perform a linear transformation on the batch normalized feature data to obtain the first feature data.

[0091] For example, the M×S dimensional feature data of a sample audio can be downsampled to obtain feature data with fewer than M×S dimensions.

[0092] An exemplary batch normalization calculation process is as follows:

[0093] z (l) The expectation and variance are usually approximated using the mean and variance of the current mini-batch. Given a mini-batch containing K samples, the net input z of the l-th layer neuron... (1，l) ,…,z (K，l) The mean μ B and variance for:

[0094]

[0095]

[0096] For net input z (l) Standard normalization causes the values ​​to concentrate around 0. When using a sigmoid activation function, this range is close to the linear transformation range, weakening the non-linear nature of the neural network. Therefore, to ensure that normalization does not negatively impact the network's representational power, an additional scaling and translation transformation can be used to change the range of values.

[0097]

[0098] Where ∈ can be a small value to avoid a denominator of 0; the value of ∈ can differ between batch normalization and layer normalization. ⊙ represents multiplication. γ and β represent the parameter vectors for scaling and translation, respectively. From a most conservative perspective, the normalized variable can be restored to its original value through the inverse transformation of standard normalization. For example, when hour, Batch normalization can be viewed as a special neural layer, added before the nonlinear activation function of each layer.

[0099] Downsampling and batch normalization can reduce computation, improve model convergence speed, and enhance model generalization ability.

[0100] In one implementation, S502 downsamples the feature data processed by CMVN, including:

[0101] A CNN network is used to perform temporal dimensionality reduction on the feature data processed by CMVN.

[0102] For example, using a two-layer Convolutional Neural Network (CNN) to perform temporal dimensionality reduction on sample audio can reduce the dimensionality to one-quarter of the original, for example, to M×m / 4 dimensions. The downsampling factor can be adjusted by changing the parameters of the CNN network.

[0103] Figure 6 This is a flowchart illustrating an audio processing method according to an embodiment of the present disclosure, which may include:

[0104] S601. By adding the absolute position code after layer normalization to the first feature data corresponding to the audio to be processed through the acoustic model, the second feature data is obtained.

[0105] S602. By adding the rotation position code after layer normalization to the second feature data through the acoustic model, the third feature data is obtained.

[0106] S603. The third feature data is processed using the acoustic model to obtain the audio processing result.

[0107] In the embodiments of this disclosure, the acoustic model obtained by performing layer normalization on the position encoding has strong generalization ability, and the accuracy of audio processing can be improved by using the trained acoustic model.

[0108] In one implementation, the acoustic model is used to add a layer-normalized rotational position code to the second feature data to obtain the third feature data, which includes:

[0109] The second feature data is linearly transformed using the acoustic model to obtain query information, key information, and value information.

[0110] After adding the rotation position code after layer normalization to the query information and the key information respectively, the attention parameters are obtained;

[0111] The third feature data is obtained based on the attention parameter and the value information. For example, the third feature data is obtained by performing matrix multiplication on the attention parameter and the value information.

[0112] In this embodiment of the disclosure, by adding a layer-normalized rotational position code to the feature data of the audio that has already been encoded with absolute position code using an acoustic model, the advantages of both position codes can be utilized to improve the accuracy of audio processing.

[0113] In one implementation, the acoustic model includes a convolution-enhanced transformer network. After adding layer-normalized rotation position codes to the query information and the key information respectively, attention parameters are obtained, including:

[0114] By using the multi-head self-attention MHSA of the conformer network, the normalized rotation position encoding is added to the query information and the key information respectively to obtain the encoded query information and the encoded key information;

[0115] The attention parameter is obtained by performing a matrix multiplication of the encoded query information and the encoded key information.

[0116] In this embodiment of the disclosure, the Conformer network can be used to add rotation position encoding after layer normalization to the feature data of the audio, which can not only improve the generalization ability of the acoustic model, but also improve the accuracy of audio processing by using the trained acoustic model.

[0117] In one implementation, such as Figure 7 As shown, the method also includes:

[0118] S701. Perform frame segmentation on the audio to be processed to obtain audio segments;

[0119] S702. Extract audio features from the audio segment;

[0120] S703, Obtain global CMVN data;

[0121] S704. Perform CMVN processing on the audio features of the audio segment to be processed.

[0122] In audio processing, the global CMVN data can be the global CMVN data of the training set obtained during training. Then, CMVN processing is applied to the FBank features of each audio segment. The mean of the CMVN data is subtracted from the S-dimensional FBank features of the audio segment, and then divided by the standard deviation of the CMVN data. Using the mean and standard deviation of the CMVN data makes the acoustic features of the audio to be processed more consistent with a certain probability distribution, improving the robustness of the model and thus obtaining accurate audio processing results.

[0123] In one embodiment, the audio features of the audio segment to be processed are subjected to CMVN processing, which includes: subtracting the mean from the CMVN data from the audio features of the audio segment to be processed, and then dividing by the standard deviation from the CMVN data. CMVN processing can yield more accurate audio processing results.

[0124] In one implementation, the method further includes:

[0125] S705. Downsample the feature data after CMVN processing;

[0126] S706. Perform batch normalization on the downsampled feature data;

[0127] S707. Perform a linear transformation on the batch normalized feature data to obtain the first feature data.

[0128] In audio processing, batch normalization can be performed on the downsampled audio features obtained during training using batch processing parameters. Downsampling and batch normalization reduce computational load and improve the model's audio processing speed and accuracy.

[0129] In one implementation, downsampling the CMVN-processed feature data includes: using a CNN network to perform temporal dimensionality reduction on the CMVN-processed feature data. The downsampling factor can be adjusted by changing the parameters of the CNN network.

[0130] The principle of audio processing in this embodiment can be found in the relevant description of acoustic model training methods. The sample audio in the training method can be replaced with the audio to be processed, which will not be repeated here.

[0131] In one specific example, the model structure used in this disclosure embodiment may include a conformer acoustic model, such as... Figure 8 As shown, an example of the overall network structure of the Conformer acoustic model is as follows:

[0132] Absolute positional encoding is used on each speech frame data, and the encoding schemes for odd and even dimensions are shown below:

[0133]

[0134]

[0135] Where pos represents the position of the speech sequence, and i represents the index of the dimension information in each sequence, with a value range of [0, d_model / 2]. Even-numbered indices use the sin function, and odd-numbered indices use the cos function.

[0136] A layer normalization layer can be used to first normalize the absolute positional encoding. Then, the layer-normalized absolute positional encoding can be added to the audio feature data.

[0137] In the self-attention of the MHSA layer, the query (Q) and key (K) information use a rotational position encoding scheme, as shown in the overall block diagram below. Figure 9 As shown:

[0138] For example, in a rotation position encoding scheme, the encoding matrix for the sequence at the m-th position is shown below:

[0139]

[0140] For example, this rotation matrix can be layer-normalized, added to query and key, and then multiplied by matrix multiplication, where q and k are column vectors:

[0141] R m q m ,R n k n

[0142] When performing attention calculations, we can obtain:

[0143]

[0144] Thus, the attention score is calculated based on the relative position (nm) of the query position m and the key position n. The angle θ... i Here are some examples:

[0145] θ i =10000 -2(i-1) / d i∈[1,2,…,d / 2]

[0146] like Figure 10 As shown, based on the above network architecture, an example of a training and processing flow is as follows:

[0147] S1: Process the audio, for example, dividing it into frames every 25ms. After extracting one frame, move the frame by, for example, 10ms to extract the next frame. Use, for example, 80-dimensional FBANK features for speech features. During model training, the audio can be sample audio. During audio processing, the audio can be the audio to be processed.

[0148] S2: Calculate the mean and variance of all features in the training set to obtain the global CMVN data, which is the global mean and standard deviation of the training set.

[0149] S3: Start training. For each audio file, normalize the extracted audio FBank by subtracting the mean of CMVN and then dividing by the standard deviation of CMVN.

[0150] S4: Downsample the normalized features. For example, use a two-layer CNN to perform temporal dimensionality reduction on the normalized features, reducing the time-series data to 1 / 4 of its original size. For example, the two-layer neural network has a kernel size of 3 and a stride of 2.

[0151] S5: Perform batch normalization on the features of all audio data in each batch. A batch consists of one or more downsampled data sets.

[0152] S6: Perform a linear transformation on the batch-normalized audio data to obtain the transformed first feature data.

[0153] S7: Perform layer normalization transformation on the absolute position code to obtain the layer normalized absolute position code.

[0154] S8: Add an absolute position encoding after normalization to the first feature data to obtain the second feature data.

[0155] S9: Perform linear transformations on the second feature data after adding absolute position information to obtain the corresponding Q (query), K (key), and V (value).

[0156] S10: Perform layer normalization transformation on the rotation position code to obtain the transformed rotation position code.

[0157] S11: Add rotation position encoding to Q(query) and K(key) after layer normalization, and then multiply the two matrices to calculate the attention score. Then multiply the attention score with the V(value) matrix to calculate the third feature data after attention, which is the new V(value).

[0158] S12: Use a Conformer network for training. For example, use a Conformer network to further process the third feature data to obtain the output data.

[0159] S13: Calculate the loss value using CTC. For example, input the output data of the Conformer network into CTC, calculate the loss value, and use the loss value to update the network parameters in the Conformer network, such as the transformation matrix and normalization parameters. It can also update the relevant parameters used for layer normalization of positional encoding.

[0160] S14: After the acoustic model is trained, it can be used to perform speech recognition and other processing on the features of each audio file.

[0161] Figure 11 This is a schematic diagram of an acoustic model training device according to an embodiment of the present disclosure. The device may include:

[0162] The absolute position encoding module 1101 is used to add the layer-normalized absolute position encoding to the first feature data corresponding to the sample audio through an acoustic model to obtain the second feature data.

[0163] The rotation position encoding module 1102 is used to process the second feature data through the acoustic model and then add the rotation position encoding after layer normalization to obtain the third feature data;

[0164] Training module 1103 is used to train the acoustic model based on the third feature data.

[0165] Figure 12 This is a schematic diagram of an acoustic model training apparatus according to another embodiment of the present disclosure. The apparatus may include one or more features of the acoustic model training apparatus described in the above embodiments. In one embodiment, the rotation position encoding module 1102 includes:

[0166] The linear transformation submodule 1201 is used to perform a linear transformation on the second feature data through the acoustic model to obtain query information, key information and value information.

[0167] The rotation encoding submodule 1202 is used to add the rotation position encoding after layer normalization to the query information and the key information respectively to obtain the attention parameters;

[0168] The value information processing submodule 1203 is used to obtain the third feature data based on the attention parameter and the value information.

[0169] In one implementation, the acoustic model includes a convolution-enhanced transformer network, and the rotation encoding submodule 1202 is further configured to:

[0170] By using the multi-head self-attention MHSA of the conformer network, the normalized rotation position encoding is added to the query information and the key information respectively to obtain the encoded query information and the encoded key information;

[0171] The attention parameter is obtained by performing a matrix multiplication of the encoded query information and the encoded key information.

[0172] In one implementation, such as Figure 12 As shown, the training module 1103 includes:

[0173] The output submodule 1204 is used to process the third feature data through the acoustic model to obtain output data;

[0174] The loss submodule 1205 is used to input the output data into the loss function to calculate the loss value of the acoustic model;

[0175] The update submodule 1206 is used to update the parameters of the acoustic model based on the loss value of the acoustic model.

[0176] In one implementation, the update submodule 1206 is further configured to:

[0177] Based on the loss value of the acoustic model, at least one of the layer normalization parameters, batch normalization parameters, and linear transformation parameters in the acoustic model is updated.

[0178] In one implementation, such as Figure 12 As shown, the device also includes:

[0179] The framing module 1104 is used to perform framing processing on the sample audio to obtain audio segments;

[0180] Feature extraction module 1105 is used to extract audio features from the audio segment;

[0181] CMVN data module 1106 is used to calculate the global cepstral mean and variance-normalized CMVN data of the training set, which includes audio features corresponding to multiple sample audios.

[0182] CMVN processing module 1107 is used to perform CMVN processing on the audio features of the audio segment in the training set.

[0183] In one implementation, such as Figure 12 As shown, the CMVN processing module 1107 is used to subtract the mean of the CMVN data from the audio features of the audio segment in the training set, and then divide by the standard deviation of the CMVN data.

[0184] In one implementation, such as Figure 12 As shown, the device also includes:

[0185] The downsampling module 1108 is used to downsample the feature data after CMVN processing;

[0186] The batch normalization module 1109 is used to perform batch normalization processing on the downsampled feature data;

[0187] The linear transformation module 1110 performs a linear transformation on the batch normalized feature data to obtain the first feature data.

[0188] In one implementation, the downsampling module 1108 is further configured to perform temporal dimensionality reduction on the CMVN-processed feature data using a CNN network.

[0189] Figure 13 This is a schematic diagram of an audio processing apparatus according to an embodiment of the present disclosure, the apparatus may include:

[0190] The absolute position encoding module 1301 is used to add the layer-normalized absolute position encoding to the first feature data corresponding to the audio to be processed through an acoustic model to obtain the second feature data.

[0191] The rotation position encoding module 1302 is used to add the rotation position encoding after layer normalization to the second feature data after processing by the acoustic model to obtain the third feature data;

[0192] The processing module 1303 is used to process the third feature data through the acoustic model to obtain the audio processing result.

[0193] Figure 14 This is a schematic diagram of an audio processing apparatus according to another embodiment of the present disclosure. The apparatus may include one or more features of the audio processing apparatus of the above embodiments. In one embodiment, the rotation position encoding module 1302 includes:

[0194] The linear transformation submodule 1401 is used to perform a linear transformation on the second feature data through the acoustic model to obtain query information, key information and value information.

[0195] The rotation encoding submodule 1402 is used to add the rotation position encoding after layer normalization to the query information and the key information respectively, and perform matrix multiplication to obtain the attention parameters;

[0196] The value information processing submodule 1403 is used to perform matrix multiplication of the attention parameter and the value information to obtain the third feature data.

[0197] In one embodiment, the acoustic model includes a convolution-enhanced converter network, and the rotation encoding submodule 1402 is further configured to:

[0198] By using the multi-head self-attention MHSA of the conformer network, the normalized rotation position encoding is added to the query information and the key information respectively to obtain the encoded query information and the encoded key information;

[0199] The attention parameter is obtained by performing a matrix multiplication of the encoded query information and the encoded key information.

[0200] In one implementation, such as Figure 14 As shown, the device also includes:

[0201] The frame segmentation module 1304 is used to perform frame segmentation on the audio to be processed to obtain audio segments.

[0202] Feature extraction module 1305 is used to extract audio features from the audio segment;

[0203] CMVN data module 1306 is used to obtain global CMVN data;

[0204] CMVN processing module 1307 is used to perform CMVN processing on the audio features of the audio segment to be processed.

[0205] In one embodiment, the CMVN processing module 1307 is further configured to subtract the mean value in the CMVN data from the audio features of the audio segment to be processed, and then divide by the standard deviation in the CMVN data.

[0206] In one implementation, such as Figure 14 As shown, the device also includes:

[0207] The downsampling module 1308 is used to downsample the feature data after CMVN processing;

[0208] The batch normalization module 1309 is used to perform batch normalization processing on the downsampled feature data;

[0209] The linear transformation module 1310 is used to perform a linear transformation on the feature data after batch normalization to obtain the first feature data.

[0210] In one implementation, the downsampling module 1308 is also used to perform temporal dimensionality reduction on the CMVN-processed feature data using a CNN network.

[0211] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0212] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0213] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0214] Figure 15A schematic block diagram of an example electronic device 1500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0215] like Figure 15 As shown, device 1500 includes a computing unit 1501, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1502 or a computer program loaded from storage unit 1508 into random access memory (RAM) 1503. The RAM 1503 may also store various programs and data required for the operation of device 1500. The computing unit 1501, ROM 1502, and RAM 1503 are interconnected via bus 1504. Input / output (I / O) interface 1505 is also connected to bus 1504.

[0216] Multiple components in device 1500 are connected to I / O interface 1505, including: input unit 1506, such as keyboard, mouse, etc.; output unit 1507, such as various types of monitors, speakers, etc.; storage unit 1508, such as disk, optical disk, etc.; and communication unit 1509, such as network card, modem, wireless transceiver, etc. Communication unit 1509 allows device 1500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0217] The computing unit 1501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1501 performs the various methods and processes described above, such as acoustic model training methods and / or audio processing methods. For example, in some embodiments, the acoustic model training methods and / or audio processing methods may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1500 via ROM 1502 and / or communication unit 1509. When the computer program is loaded into RAM 1503 and executed by the computing unit 1501, one or more steps of the acoustic model training methods and / or audio processing methods described above may be performed. Alternatively, in other embodiments, the computing unit 1501 may be configured by any other suitable means (e.g., by means of firmware) to perform acoustic model training methods and / or audio processing methods.

[0218] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0219] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0220] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0221] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0222] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0223] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0224] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0225] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An acoustic model training method, comprising: The acoustic model adds a normalized absolute position code to the first feature data corresponding to the sample audio to obtain the second feature data; wherein, the first network of the acoustic model is used to process the audio segment of the audio to obtain the first feature data, the second network of the acoustic model is used to normalize the preset first absolute position code layer to obtain the second absolute position code, and the second feature data is obtained by adding the first feature data and the second absolute position code. After processing the second feature data using the acoustic model, a rotational position code after layer normalization is added to obtain the third feature data. The third network of the acoustic model is a conformer network, which performs a linear transformation on the second feature data to obtain query information, key information, and value information. After adding a rotational position code after layer normalization to the query information and the key information, attention parameters are obtained. The third feature data is then obtained based on the attention parameters and the value information. The acoustic model is trained based on the third feature data.

2. The method according to claim 1, wherein, After adding layer-normalized rotation position codes to the query information and the key information respectively, attention parameters are obtained, including: By adding layer-normalized rotational position encoding to the query information and the key information respectively using the multi-head self-attention MHSA of the Conformer network, the coded query information and the coded key information are obtained. The attention parameters are obtained by performing matrix multiplication on the encoded query information and the encoded key information.

3. The method according to claim 1 or 2, wherein, Training the acoustic model based on the third feature data includes: The third feature data is processed using the acoustic model to obtain output data; The output data is input into the loss function to calculate the loss value of the acoustic model; The parameters of the acoustic model are updated based on the loss value of the acoustic model.

4. The method according to claim 3, wherein, The parameters of the acoustic model are updated based on the loss value of the acoustic model, including: Based on the loss value of the acoustic model, at least one of the layer normalization parameters, batch normalization parameters, and linear transformation parameters in the acoustic model is updated.

5. The method according to claim 1 or 2, further comprising: The sample audio is segmented into frames to obtain audio segments; Extract audio features from the audio segment; Calculate the global cepstral mean and variance-normalized CMVN data of the training set, wherein the training set includes audio features corresponding to multiple sample audios; The audio features of the audio segments in the training set are processed using CMVN.

6. The method according to claim 5, wherein, The audio features of the audio segments in the training set are processed using CMVN, including: The audio features of the audio segments in the training set are subtracted from the mean in the CMVN data and then divided by the standard deviation in the CMVN data.

7. The method according to claim 5, further comprising: Downsample the feature data after CMVN processing; The downsampled feature data is then batch normalized. The first feature data is obtained by performing a linear transformation on the batch normalized feature data.

8. The method according to claim 7, wherein, Downsampling of the CMVN-processed feature data includes: A CNN network is used to perform temporal dimensionality reduction on the feature data processed by CMVN.

9. An audio processing method, comprising: The acoustic model adds a normalized absolute position code to the first feature data corresponding to the audio to be processed to obtain the second feature data; wherein, the first network of the acoustic model is used to process the audio segment of the audio to obtain the first feature data, the second network of the acoustic model is used to normalize the preset first absolute position code layer to obtain the second absolute position code, and the second feature data is obtained by adding the first feature data and the second absolute position code. After processing the second feature data using the acoustic model, a rotational position code after layer normalization is added to obtain the third feature data. The third network of the acoustic model is a conformer network, which performs a linear transformation on the second feature data to obtain query information, key information, and value information. After adding a rotational position code after layer normalization to the query information and the key information, attention parameters are obtained. The third feature data is then obtained based on the attention parameters and the value information. The third feature data is processed using the acoustic model to obtain the audio processing result.

10. The method according to claim 9, wherein, The acoustic model includes a convolution-enhanced transformer network. After adding layer-normalized rotation position codes to the query information and key information respectively, attention parameters are obtained, including: The multi-head self-attention (MHSA) of the conformer network is used to add rotation position encoding after layer normalization to the query information and key information respectively to obtain the encoded query information and the encoded key information; The attention parameters are obtained by performing matrix multiplication on the encoded query information and the encoded key information.

11. The method according to claim 9 or 10, further comprising: The audio to be processed is divided into frames to obtain audio segments; Extract audio features from the audio segment; Retrieve global CMVN data; The audio features of the audio segment of the audio to be processed are subjected to CMVN processing.

12. The method according to claim 11, wherein, The audio features of the audio segment to be processed are subjected to CMVN processing, including: The audio features of the audio segment to be processed are subtracted from the mean in the CMVN data and then divided by the standard deviation in the CMVN data.

13. The method of claim 12, further comprising: Downsample the feature data after CMVN processing; The downsampled feature data is then batch normalized. The first feature data is obtained by performing a linear transformation on the batch normalized feature data.

14. The method according to claim 13, wherein, Downsampling of the CMVN-processed feature data includes: A CNN network is used to perform temporal dimensionality reduction on the feature data processed by CMVN.

15. An acoustic model training device, comprising: An absolute position encoding module is used to add a layer of normalized absolute position encoding to the first feature data corresponding to the sample audio through an acoustic model to obtain second feature data; wherein, the first network of the acoustic model is used to process the audio segment of the audio to obtain the first feature data, the second network of the acoustic model is used to normalize the preset first absolute position encoding layer to obtain the second absolute position encoding, and the second feature data is obtained by adding the first feature data and the second absolute position encoding; A rotation position encoding module is used to process the second feature data through the acoustic model and then add a rotation position encoding after layer normalization to obtain the third feature data; wherein, the third network of the acoustic model is a conformer network, which is used to perform a linear transformation on the second feature data to obtain query information, key information and value information. After adding the rotation position encoding after layer normalization to the query information and the key information respectively, attention parameters are obtained, and the third feature data is obtained based on the attention parameters and the value information. The training module is used to train the acoustic model based on the third feature data.

16. The apparatus according to claim 15, wherein, The acoustic model includes a convolution-enhanced transformer network, and the rotation position encoding module is further used for: By adding layer-normalized rotational position encoding to the query information and the key information respectively using the multi-head self-attention MHSA of the Conformer network, the coded query information and the coded key information are obtained. The attention parameters are obtained by performing matrix multiplication on the encoded query information and the encoded key information.

17. The apparatus according to claim 15 or 16, wherein, The training module includes: The output submodule is used to process the third feature data through the acoustic model to obtain output data; The loss submodule is used to input the output data into the loss function to calculate the loss value of the acoustic model; The update submodule is used to update the parameters of the acoustic model based on the loss value of the acoustic model.

18. The apparatus according to claim 17, wherein, The update submodule is further configured to update at least one of the layer normalization parameters, batch normalization parameters, and linear transformations in the acoustic model based on the loss value of the acoustic model.

19. The apparatus according to claim 15 or 16, further comprising: The frame segmentation module is used to segment the sample audio into frames to obtain audio segments; The feature extraction module is used to extract audio features from the audio segment; The CMVN data module is used to calculate the global cepstral mean and variance-normalized CMVN data of the training set, which includes audio features corresponding to multiple sample audios. The CMVN processing module is used to perform CMVN processing on the audio features of the audio segments in the training set.

20. The apparatus according to claim 19, wherein, The CMVN processing module is further configured to subtract the mean of the CMVN data from the audio features of the audio segments in the training set, and then divide by the standard deviation of the CMVN data.

21. The apparatus of claim 19, further comprising: The downsampling module is used to downsample the feature data after CMVN processing; The batch normalization module is used to perform batch normalization processing on the downsampled feature data; The linear transformation module performs a linear transformation on the batch-normalized feature data to obtain the first feature data.

22. The apparatus according to claim 21, wherein, The downsampling module is also used to perform temporal dimensionality reduction on the feature data processed by the CMVN using a CNN network.

23. An audio processing apparatus, comprising: An absolute position encoding module is used to add a layer of normalized absolute position encoding to the first feature data corresponding to the audio to be processed through an acoustic model to obtain second feature data; wherein, the first network of the acoustic model is used to process the audio segment of the audio to obtain the first feature data, the second network of the acoustic model is used to normalize the preset first absolute position encoding layer to obtain the second absolute position encoding, and the second feature data is obtained by adding the first feature data and the second absolute position encoding; A rotation position encoding module is used to process the second feature data through the acoustic model and then add a layer of normalized rotation position encoding to obtain the third feature data. The third network of the acoustic model is a conformer network, which performs a linear transformation on the second feature data to obtain query information, key information, and value information. After adding a layer of normalized rotation position encoding to the query information and the key information, attention parameters are obtained. The third feature data is then obtained based on the attention parameters and the value information. The processing module is used to process the third feature data through the acoustic model to obtain the audio processing result.

24. The apparatus according to claim 23, wherein, The acoustic model includes a convolution-enhanced transformer network, and the rotation position encoding module is further used for: The multi-head self-attention (MHSA) of the conformer network is used to add rotation position encoding after layer normalization to the query information and key information respectively to obtain the encoded query information and the encoded key information; The attention parameters are obtained by matrix multiplication of the encoded query information and the encoded key information.

25. The apparatus according to claim 23 or 24, further comprising: The frame segmentation module is used to segment the audio to be processed into frames to obtain audio segments; The feature extraction module is used to extract audio features from the audio segment; The CMVN data module is used to obtain global CMVN data; The CMVN processing module is used to perform CMVN processing on the audio features of the audio segment of the audio to be processed.

26. The apparatus according to claim 25, wherein, The CMVN processing module is further configured to subtract the mean value from the CMVN data from the audio features of the audio segment to be processed, and then divide by the standard deviation of the CMVN data.

27. The apparatus of claim 25, further comprising: The downsampling module is used to downsample the feature data after CMVN processing; The batch normalization module is used to perform batch normalization processing on the downsampled feature data; The linear transformation module is used to perform a linear transformation on the batch normalized feature data to obtain the first feature data.

28. The apparatus according to claim 27, wherein, The downsampling module is also used to perform temporal dimensionality reduction on the feature data processed by the CMVN using a CNN network.

29. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-14.

30. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-14.

31. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-14.

Images