Neural network training method and apparatus, audio processing method and apparatus

Abstract

The present disclosure relates to a neural network training method and device, and an audio processing method and device. The method comprises: inputting training audio to an encoding subnetwork for encoding to obtain a first encoding result, and inputting a text label corresponding to the training audio to a prediction subnetwork for prediction to obtain a first prediction result; combining the first encoding result and the first prediction result to obtain a first combined result; according to the first combined result, respectively clipping the first encoding result and the first prediction result to obtain corresponding second encoding result and second prediction result; inputting the second encoding result and the second prediction result to a joint subnetwork for joint processing to obtain a second combined result, and adjusting network parameters of the encoding subnetwork, the prediction subnetwork and the joint subnetwork according to the second combined result.

Description

Technical Field

[0001] This disclosure relates to the field of audio processing technology, specifically to a neural network training method and apparatus, and an audio processing method and apparatus. Background Technology

[0002] In recent years, audio processing technologies such as audio recognition have gradually developed, with increasingly higher accuracy, playing an important role in many fields. Currently, the audio processing field includes technologies such as Connectionist Temporal Classification (CTC), Attention-based models, and Recurrent Neural Network Transducers (RNN-T), among which RNN-T technology has shown the best performance in practice. However, the computational complexity and memory consumption during neural network training in RNN-T technology hinders rapid computation, significantly reducing its practicality. Summary of the Invention

[0003] To overcome the problems existing in the related technologies, the present disclosure provides a neural network training method and apparatus, and an audio processing method and apparatus, to solve the defects in the related technologies.

[0004] According to a first aspect of the present disclosure, a neural network training method is provided, comprising:

[0005] The training audio is input into the encoding subnetwork for encoding to obtain the first encoding result, and the text label corresponding to the training audio is input into the prediction subnetwork for prediction to obtain the first prediction result;

[0006] The first encoding result and the first prediction result are combined to obtain the first joint result;

[0007] Based on the first joint result, the first encoding result and the first prediction result are respectively trimmed to obtain the corresponding second encoding result and second prediction result;

[0008] The second encoding result and the second prediction result are input into the joint sub-network for joint processing to obtain the second joint result. The network parameters of the encoding sub-network, the prediction sub-network, and the joint sub-network are then adjusted based on the second joint result.

[0009] In one embodiment, the step of combining the first encoding result and the first prediction result to obtain a first joint result includes:

[0010] The first encoding result and the first prediction result are input into the joint module for joint processing to obtain the first joint result.

[0011] In one embodiment, each element within the first joint result has at least coordinates in the audio frame number dimension and coordinates in the text label sequence dimension.

[0012] In one embodiment, the step of pruning the first encoding result and the first prediction result according to the first joint result to obtain the corresponding second encoding result and second prediction result includes:

[0013] Obtain the gradient information of each element in the first joint result;

[0014] Based on the gradient information of each element in the first joint result, the clipping boundary is determined in the plane composed of the audio frame number dimension and the text label sequence dimension;

[0015] Based on the pruning boundary, the first encoding result and the first prediction result are pruned respectively to obtain the corresponding second encoding result and the second prediction result.

[0016] In one embodiment, obtaining the gradient information of each element within the first joint result includes:

[0017] Obtain the gradient information of the probability of outputting text labels for each element in the first joint result, and the gradient information of the probability of outputting empty characters.

[0018] In one embodiment, determining the clipping boundary in the plane composed of the audio frame number dimension and the text tag sequence dimension based on the gradient information of each element in the first joint result includes:

[0019] In the t-th frame of the audio frame number dimension, the sum of the gradient information of each element in the sequence interval of a preset size starting from each index in the text label sequence dimension is determined as the total gradient of the corresponding index.

[0020] The sequence interval of a preset size, starting from the index with the largest total gradient, is determined as the clipping boundary of the t-th frame.

[0021] In one embodiment, the index of the largest total gradient also satisfies at least one of the following:

[0022] The difference between the index of the largest total gradient and the index of the largest dimension of the text label sequence is greater than or equal to a preset size.

[0023] The index of the largest total gradient is less than or equal to the starting index of the clipping boundary of the (t+1)th frame;

[0024] The difference between the index of the largest gradient and the starting index of the cropping boundary of the (t+1)th frame is less than the preset size.

[0025] In one embodiment, the step of cropping the first encoding result and the first prediction result according to the cropping boundary to obtain the second encoding result and the second prediction result respectively includes:

[0026] The first encoding result and the first prediction result are respectively expanded in at least one dimension so that each element of the first encoding result has coordinates in the audio frame number dimension and the text label sequence dimension;

[0027] Within the plane formed by the audio frame number dimension and the text label sequence dimension, the first encoding result and the first prediction result are cropped according to the cropping boundary to obtain the second encoding result and the second prediction result.

[0028] In one embodiment, adjusting the network parameters of the coding subnetwork, the prediction subnetwork, and the joint subnetwork based on the second joint result includes:

[0029] A first network loss value is determined based on the second joint result, and the network parameters of the coding sub-network, the prediction sub-network, and the joint sub-network are adjusted based on the first network loss value.

[0030] In one embodiment, it also includes:

[0031] The second network loss value is determined based on the first joint result;

[0032] The step of adjusting the network parameters of the coding sub-network, the prediction sub-network, and the joint sub-network based on the first network loss value includes:

[0033] The network parameters of the coding subnetwork, the prediction subnetwork, and the joint subnetwork are adjusted based on the first network loss value and the second network loss value.

[0034] According to a second aspect of the present disclosure, an audio processing method is provided, comprising:

[0035] The audio to be processed is input into a neural network to obtain the target text, wherein the neural network is pre-trained using the neural network training method described in the first aspect.

[0036] According to a third aspect of the present disclosure, a neural network training apparatus is provided, comprising:

[0037] The input module is used to input the training audio into the encoding sub-network for encoding to obtain the first encoding result, and input the text label corresponding to the training audio into the prediction sub-network for prediction to obtain the first prediction result;

[0038] The first joint module is used to combine the first encoding result and the first prediction result to obtain a first joint result;

[0039] The pruning module is used to prune the first encoding result and the first prediction result according to the first joint result to obtain the corresponding second encoding result and second prediction result.

[0040] The second joint module is used to input the second encoding result and the second prediction result into the joint sub-network for joint processing to obtain the second joint result, and to adjust the network parameters of the encoding sub-network, the prediction sub-network and the joint sub-network according to the second joint result.

[0041] In one embodiment, the first joint module is specifically used for:

[0042] The first encoding result and the first prediction result are input into the joint module for joint processing to obtain the first joint result.

[0043] In one embodiment, each element within the first joint result has at least coordinates in the audio frame number dimension and coordinates in the text label sequence dimension.

[0044] In one embodiment, the cropping module is specifically used for:

[0045] Obtain the gradient information of each element in the first joint result;

[0046] Based on the gradient information of each element in the first joint result, the clipping boundary is determined in the plane composed of the audio frame number dimension and the text label sequence dimension;

[0047] Based on the pruning boundary, the first encoding result and the first prediction result are pruned respectively to obtain the corresponding second encoding result and the second prediction result.

[0048] In one embodiment, when the pruning module is used to obtain the gradient information of each element in the first joint result, it is specifically used for:

[0049] Obtain the gradient information of the probability of outputting text labels for each element in the first joint result, and the gradient information of the probability of outputting empty characters.

[0050] In one embodiment, when the cropping module determines the cropping boundary in the plane composed of the audio frame number dimension and the text tag sequence dimension based on the gradient information of each element in the first joint result, it is specifically used for:

[0051] In the t-th frame of the audio frame number dimension, the sum of the gradient information of each element in the sequence interval of a preset size starting from each index in the text label sequence dimension is determined as the total gradient of the corresponding index.

[0052] The sequence interval of a preset size, starting from the index with the largest total gradient, is determined as the clipping boundary of the t-th frame.

[0053] In one embodiment, the index of the largest total gradient also satisfies at least one of the following:

[0054] The difference between the index of the largest total gradient and the index of the largest dimension of the text label sequence is greater than or equal to a preset size.

[0055] The index of the largest total gradient is less than or equal to the starting index of the clipping boundary of the (t+1)th frame;

[0056] The difference between the index of the largest gradient and the starting index of the cropping boundary of the (t+1)th frame is less than the preset size.

[0057] In one embodiment, the cropping module is used to crop the first encoding result and the first prediction result according to the cropping boundary, respectively, to obtain the second encoding result and the second prediction result, specifically for:

[0058] The first encoding result and the first prediction result are respectively expanded in at least one dimension so that each element of the first encoding result has coordinates in the audio frame number dimension and the text label sequence dimension;

[0059] Within the plane formed by the audio frame number dimension and the text label sequence dimension, the first encoding result and the first prediction result are cropped according to the cropping boundary to obtain the second encoding result and the second prediction result.

[0060] In one embodiment, when the second joint module adjusts the network parameters of the coding subnetwork, the prediction subnetwork, and the joint subnetwork based on the second joint result, it is specifically used for:

[0061] A first network loss value is determined based on the second joint result, and the network parameters of the coding sub-network, the prediction sub-network, and the joint sub-network are adjusted based on the first network loss value.

[0062] In one embodiment, a loss module is also included, for:

[0063] The second network loss value is determined based on the first joint result;

[0064] When the second joint module adjusts the network parameters of the coding sub-network, the prediction sub-network, and the joint sub-network based on the first network loss value, it is specifically used for:

[0065] The network parameters of the coding subnetwork, the prediction subnetwork, and the joint subnetwork are adjusted based on the first network loss value and the second network loss value.

[0066] According to a fourth aspect of the present disclosure, an audio processing apparatus is provided, including an input module for:

[0067] The audio to be processed is input into a neural network to obtain the target text, wherein the neural network is pre-trained using the neural network training method described in the first aspect.

[0068] According to a fifth aspect of the present disclosure, an electronic device is provided, the electronic device including a memory and a processor, the memory being configured to store computer instructions executable on the processor, and the processor being configured to execute the computer instructions based on the method described in the first aspect or the second aspect.

[0069] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first or second aspect.

[0070] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0071] The neural network training method provided in this disclosure involves inputting training audio into an encoding sub-network for encoding to obtain a first encoding result, and inputting the text labels corresponding to the training audio into a prediction sub-network for prediction to obtain a first prediction result. The first encoding result and the first prediction result can then be combined to obtain a first joint result. Based on the first joint result, the first encoding result and the first prediction result are respectively pruned to obtain a second encoding result and a second prediction result. Finally, the second encoding result and the second prediction result can be input into a joint sub-network for joint processing to obtain a second joint result. The network parameters of the encoding sub-network, the prediction sub-network, and the joint sub-network are adjusted based on the second joint result. Because the first encoding result and the first prediction result are pruned before being input into the joint sub-network, the computational load of the joint sub-network and the computational power required to adjust the network parameters using the second joint result can be reduced. This reduces the memory usage during the training process of RNN-T and other models, enabling fast computation and improving the practicality of audio processing technology. Attached Figure Description

[0072] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0073] Figure 1 This is a flowchart illustrating a neural network training method according to an exemplary embodiment of this disclosure;

[0074] Figure 2 This is a schematic diagram illustrating the calculation method of the RNN-T loss function in related technologies;

[0075] Figure 3 This is a schematic diagram illustrating the calculation method of the RNN-T loss function according to an exemplary embodiment of this disclosure;

[0076] Figure 4 This is a flowchart illustrating a neural network training method according to another exemplary embodiment of this disclosure;

[0077] Figure 5 This is a schematic diagram of the structure of a neural network training device shown in an exemplary embodiment of the present disclosure;

[0078] Figure 6 This is a structural block diagram of an electronic device illustrated in an exemplary embodiment of the present disclosure. Detailed Implementation

[0079] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0080] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0081] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0082] The field of speech recognition has undergone decades of development, from initial sequence similarity matching to modeling based on Gaussian mixture models and hidden Markov models, and later to hybrid systems based on neural networks. For decades, building a speech recognition system has been a highly complex task, requiring a cumbersome data alignment process before model building. In recent years, end-to-end models have entered a phase of rapid development, which not only greatly simplifies the speech recognition modeling process by eliminating complex alignment procedures, but also achieves superior recognition results.

[0083] In general, current end-to-end model implementations fall into three main categories: Connectionist Temporal Classification (CTC), attention-based models, and Recurrent Neural Network Transducers (RNN-T). Among these, CTC and RNN-T are inherently streaming models, capable of frame-synchronous decoding. Attention-based models, however, require additional modifications to achieve streaming decoding, which are not only cumbersome but also result in a loss of recognition accuracy. Both CTC and RNN-T models rely on the assumption that frames are probabilistically independent, making it difficult to achieve excellent recognition rates without an external language model. Therefore, considering all factors, RNN-T models have greater applicability in production practice.

[0084] Because the definition of the RNN-T loss function involves calculations on a four-dimensional matrix, namely the (N, T, U, V) matrix, where N represents the number of samples in a batch, T represents the audio frame count, U represents the text label sequence count, and V represents the modeling unit count, the computation of such a large four-dimensional matrix makes calculating the RNN-T loss function not only computationally inefficient but also requires a significant amount of memory, limiting the number of batches that can be parallelized within a given memory limit. In short, existing RNN-T implementations cannot achieve fast computation, resulting in very slow training of RNN-T models and significantly reducing their practicality.

[0085] Based on this, in a first aspect, at least one embodiment of this disclosure provides a neural network training method, please refer to the appendix. Figure 1 The diagram illustrates the process of the method, including steps S101 to S104.

[0086] This method can be applied to the training process of neural networks for audio processing, such as the training process of RNN-T models. An RNN-T model typically consists of three parts: an encoder network, a prediction network, and a joint network. The output of the encoder network is a vector of dimension (T, E), the output of the prediction network is a vector of dimension (U, D), and the output of the joint network is a vector of dimension (T, U, V), where T is the number of audio frames, E is the output dimension of the encoder network, U is the number of text label sequences corresponding to the audio, D is the output dimension of the prediction network, and V is the number of modeling units, i.e., the number of characters. If the batch dimension N is added, the output of the RNN-T model is a matrix of dimension (N, T, U, V). In related technologies, the calculation of the RNN-T loss function involves using a forward-backward algorithm to traverse all paths matching the text label sequences within this matrix, and then summing the scores of these paths, as shown in the attached diagram. Figure 2 As shown in the figure, only two dimensions, T and U, are displayed. This makes the computation of RNN-T not only very memory-intensive but also extremely computationally dense.

[0087] In step S101, the training audio is input into the encoding subnetwork for encoding to obtain the first encoding result, and the text label corresponding to the training audio is input into the prediction subnetwork for prediction to obtain the first prediction result.

[0088] The first encoding result is the vector with dimension (T, E), and the first prediction result is the vector with dimension (U, D). When the training audio and corresponding text labels in the input encoding and prediction subnetworks are input in batches, the first encoding result is a vector with dimension (N, T, E), and the first prediction result is a vector with dimension (N, U, D).

[0089] In step S102, the first encoding result and the first prediction result are combined to obtain a first combined result.

[0090] For example, the first encoding result and the first prediction result can be input into the joint module for joint processing to obtain the first joint result. The joint module can be a trivial joint network, which is a simplified version of a commonly used joint network. Commonly used joint networks contain nonlinear transformations (such as tanh or ReLU), while trivial joint networks only contain a linear addition operation. For example, the first encoding result and the second prediction result can be weighted and summed using pre-configured weights to obtain the first joint result.

[0091] In one possible embodiment, each element in the first joint result has at least coordinates in the audio frame number dimension and coordinates in the text label sequence dimension; each element may have a probability of outputting a text label and a probability of outputting an empty character.

[0092] Specifically, the first encoding result can be defined as The first prediction result is defined as Then the first joint result It can be calculated using the following formula:

[0093]

[0094]

[0095] The first joint result is a matrix L of (T, U, V) (assuming the batch size N is 1). Each element l(t, u, v) in the matrix represents the probability value of the element with coordinate t in the audio frame dimension and coordinate u in the text tag sequence dimension (i.e., the element with coordinate (t, u)) outputting the character v (character v can be a text tag or an empty character). Therefore, it can be defined as follows:

[0096]

[0097] Where y(t,u) represents the logarithmic probability of the element at coordinate (t,u) being shifted upwards (i.e., the probability of outputting the text label). This represents the logarithmic probability of the element at coordinate (t, u) shifting to the right (i.e., the probability of outputting an empty character).

[0098] With this definition, y(t,u) and y(t,u) can be constructed using simple matrix multiplication and query algorithms. This avoids allocating a huge four-dimensional matrix, thus achieving the goals of reducing memory usage and increasing speed.

[0099] It is understandable that in this step, the first encoding result and the first prediction result are combined to obtain the first joint result. Compared with the joint processing of the joint sub-network, the required computing power is smaller, the memory usage is less, and the processing efficiency is higher.

[0100] It should be understood that the calculation method of the first joint result is described in step S102. However, it should be understood that other possible methods for calculating the joint result are also within the scope of protection of this disclosure. That is, the first joint result can also be obtained by combining the first encoding result and the first prediction result based on other calculation methods.

[0101] In step S103, based on the first joint result, the first encoding result and the first prediction result are pruned respectively to obtain the corresponding second encoding result and second prediction result.

[0102] Based on experience, audio and text tags should be monotonous; that is, it is almost impossible to output a large number of text tags in a single frame of audio. Therefore, a path closer to reality should be as shown in the attached figure. Figure 3 As shown, only a certain number of text labels will be output on a specific frame. Therefore, the dimensions of the text labels can be clipped, so that the calculations can be performed on a reduced matrix.

[0103] In one possible embodiment, this step can be performed as follows:

[0104] First, the gradient information of each element in the first joint result is obtained. For example, the gradient information of the probability of outputting text labels for each element in the first joint result is obtained. and gradient information of the probability of outputting an empty character. .

[0105] Next, based on the gradient information of each element in the first joint result, the clipping boundary is determined in the plane formed by the audio frame number dimension and the text label sequence dimension. For example, firstly, at the t-th frame of the audio frame number dimension, the sum of the gradient information of each element within a sequence interval of a preset size (s_range, denoted as S in the formula) starting from each index in the text label sequence dimension can be determined as the total gradient for the corresponding index, where t is an integer not less than 1, i.e., the t-th frame is each frame in the audio frame number dimension; then, the sequence interval of the preset size starting from the index with the largest total gradient is determined as the clipping boundary of the t-th frame. After the clipping boundary of each frame is determined, the clipping boundary in the plane formed by the audio frame number dimension and the text label sequence dimension can be obtained.

[0106] Specifically, the index p of the maximum gradient in frame t can be determined using the following formula. t :

[0107]

[0108] Where U is the maximum sequence number in the dimension of the text label sequence.

[0109] In addition, the index with the largest total gradient can satisfy at least one of the following:

[0110] The difference between the index of the largest gradient and the index of the largest dimension of the text label sequence is greater than or equal to a preset size, i.e., 0 ≤ p. t ≤U-S+1.

[0111] The index of the largest total gradient is less than or equal to the starting index of the clipping boundary of the (t+1)th frame, i.e., p t ≤p t+1 .

[0112] The difference between the index of the largest gradient and the starting index of the cropping boundary of the (t+1)th frame is less than the preset size, i.e., p t+1 - p t ≤S.

[0113] Finally, based on the pruning boundaries, the first encoding result and the first prediction result are pruned respectively to obtain the corresponding second encoding result and second prediction result. For example, the first encoding result and the first prediction result can be expanded in at least one dimension so that each element of the first encoding result has coordinates in the audio frame number dimension and the text label sequence dimension. That is, the first encoding result is expanded from (N, T, V) dimension to (N, T, U, V) dimension, and the first prediction result is expanded from (N, U, V) dimension to (N, T, U, V) dimension. Then, within the plane formed by the audio frame number dimension and the text label sequence dimension, based on the pruning boundaries, the first encoding result and the first prediction result are pruned respectively to obtain the corresponding second encoding result and second prediction result. Both the second encoding result and the second prediction result are matrices of (N, T, s_range, V) dimensions. Since s_range is much smaller than U, and these two matrices are much smaller than a four-dimensional matrix of (N, T, U, V) dimensions, the memory usage (i.e., computational density) during joint network computation can be greatly reduced.

[0114] It should be understood that the trimming method in step S103 can be implemented alone, or it can be implemented in combination with some or all of one or more embodiments of this application. For example, the trimming method in step S103 can trim the first combined result obtained through step S102, or it can trim other combined results.

[0115] In step S104, the second encoding result and the second prediction result are input into the joint sub-network for joint processing to obtain the second joint result. The network parameters of the encoding sub-network, the prediction sub-network, and the joint sub-network are adjusted according to the second joint result. The network parameters can be parameters such as the weights and biases of nodes in the network layer.

[0116] Since s_range is much smaller than U, and these two matrices are much smaller than a four-dimensional matrix of dimension (N, T, U, V), the memory usage (i.e., computational density) during joint network computation can be greatly reduced. The second joint result can be a matrix of dimension (N, T, s_range, V).

[0117] In one possible embodiment, a first network loss value can be determined based on the second joint result, and the network parameters of the encoding subnetwork, the prediction subnetwork, and the joint subnetwork can be adjusted based on the first network loss value. When calculating the first network loss value, a forward-backward algorithm can be used to traverse all paths matching the text label sequence under the matrix of the second joint result, and then the scores of these paths can be summed. Adjustment of the network parameters can be stopped when the first network loss value reaches a preset error range, or when the number of iterations reaches a preset threshold.

[0118] Furthermore, a second network loss value can be determined based on the first joint result, and the network parameters of the encoding sub-network, the prediction sub-network, and the joint sub-network can be adjusted based on the first network loss value and the second network loss value. When calculating the first network loss value, a forward-backward algorithm can be used to traverse all paths matching the text label sequence under the matrix of the second joint result, and then the scores of these paths can be summed. For example, the first network loss value and the second network loss value can be weighted and summed to obtain a comprehensive network loss value, and then the network parameters of the neural network can be adjusted using the comprehensive network loss, thereby further increasing the stability of network training.

[0119] The neural network training method provided in this disclosure involves inputting training audio into an encoding sub-network for encoding to obtain a first encoding result, and inputting the text labels corresponding to the training audio into a prediction sub-network for prediction to obtain a first prediction result. The first encoding result and the first prediction result can then be combined to obtain a first joint result. Based on the first joint result, the first encoding result and the first prediction result are respectively pruned to obtain a second encoding result and a second prediction result. Finally, the second encoding result and the second prediction result can be input into a joint sub-network for joint processing to obtain a second joint result. The network parameters of the encoding sub-network, the prediction sub-network, and the joint sub-network are adjusted based on the second joint result. Because the first encoding result and the first prediction result are pruned before being input into the joint sub-network, the computational load of the joint sub-network and the computational power required to adjust the network parameters using the second joint result can be reduced, thus reducing memory usage during training, achieving fast computation, and improving the practicality of audio processing technology.

[0120] The RNN-T training method disclosed herein effectively bypasses iterative calculations on a four-dimensional matrix. Without sacrificing modeling capabilities, it utilizes a pruning algorithm to achieve efficient calculation of the RNN-T loss function. Compared to existing implementations, it achieves a computational speed improvement of over 10 times while reducing memory usage by several times. This solves the problem of excessive memory consumption and low computational efficiency in RNN-T loss function calculation, significantly improving the training speed of RNN-T models and promoting their application in production practice.

[0121] In some embodiments of this disclosure, the first encoding result and the first prediction result can be trimmed in the following manner to obtain the corresponding second encoding result and second prediction result:

[0122] First, the gradient information of each element within the joint result of the first encoding result and the first prediction result is obtained. For example, the gradient information of the probability of outputting the text label for each element within the joint result is obtained. and gradient information of the probability of outputting an empty character. .

[0123] Next, based on the gradient information of each element in the joint result, the clipping boundary is determined in the plane formed by the audio frame number dimension and the text label sequence dimension. For example, first, at the t-th frame of the audio frame number dimension, the sum of the gradient information of each element within a sequence interval of a preset size (s_range, denoted as S in the formula) starting from each index in the text label sequence dimension can be determined as the total gradient for the corresponding index, where t is an integer not less than 1, i.e., the t-th frame is each frame in the audio frame number dimension; then, the sequence interval of the preset size starting from the index with the largest total gradient is determined as the clipping boundary of the t-th frame. After the clipping boundary of each frame is determined, the clipping boundary in the plane formed by the audio frame number dimension and the text label sequence dimension can be obtained.

[0124] Specifically, the index p of the largest gradient in frame t can be determined using the following formula. t :

[0125]

[0126] Where U is the maximum sequence number in the dimension of the text label sequence.

[0127] In addition, the index with the largest total gradient can satisfy at least one of the following:

[0128] The difference between the index of the largest gradient and the index of the largest dimension of the text label sequence is greater than or equal to a preset size, i.e., 0 ≤ p. t≤U-S+1.

[0129] The index of the largest total gradient is less than or equal to the starting index of the clipping boundary of the (t+1)th frame, i.e., p t ≤p t+1 .

[0130] The difference between the index of the largest gradient and the starting index of the cropping boundary of the (t+1)th frame is less than the preset size, i.e., p t+1 - p t ≤S.

[0131] Finally, based on the pruning boundaries, the first encoding result and the first prediction result are pruned respectively to obtain the corresponding second encoding result and second prediction result. For example, the first encoding result and the first prediction result can be expanded in at least one dimension so that each element of the first encoding result has coordinates in the audio frame number dimension and coordinates in the text label sequence dimension. That is, the first encoding result is expanded from (N, T, V) dimension to (N, T, U, V) dimension, and the first prediction result is expanded from (N, U, V) dimension to (N, T, U, V) dimension. Then, within the plane formed by the audio frame number dimension and the text label sequence dimension, based on the pruning boundaries, the first encoding result and the first prediction result are pruned respectively to obtain the corresponding second encoding result and second prediction result. Both the second encoding result and the second prediction result are matrices of (N, T, s_range, V) dimension. Since s_range is much smaller than U, and these two matrices are much smaller than a four-dimensional matrix of (N, T, U, V) dimension, the memory usage (i.e., computational density) during joint network computation can be greatly reduced.

[0132] Please refer to the appendix. Figure 4 The attached diagram illustrates, by way of example, a flowchart of the neural network training method provided in this disclosure. Figure 4As can be seen, firstly, the encoding sub-network am encodes the training audio to obtain a first encoding result with dimensions (N, T, V), and the prediction sub-network lm predicts the text labels corresponding to the training audio to obtain a first prediction result with dimensions (N, U+1, V). Then, the first encoding result and the first prediction result are combined to calculate the loss function (rnnt_loss_simple) and gradient information, and the gradient information is used to calculate pruning bounds with dimensions (N, T, s_range, V). Then, the first encoding result and the first prediction result are pruned using the pruning bounds respectively to obtain a second encoding result (pruned am) with dimensions (N, T, s_range, V) and a second prediction result (pruned am) with dimensions (N, T, s_range, V). The second encoding result and the second prediction result are then input into the joint sub-network for joint processing to obtain the second joint result of (N, T, s_range, V). The loss function (rnnt_loss_pruned) is then calculated based on the second joint result. Finally, the loss functions (rnnt_loss_simple) and (rnnt_loss_pruned) are weighted and summed. The network parameters of the encoding sub-network, prediction sub-network and joint sub-network are adjusted based on the results until convergence.

[0133] In this embodiment, by pruning the first encoding result and the first prediction result before combining them, the computational load and memory usage in the combination process can be reduced, thereby reducing the computational load and memory usage in the training process of the neural network model.

[0134] According to a second aspect of the present disclosure, an audio processing method is provided, comprising:

[0135] The audio to be processed is input into a neural network to obtain the target text, wherein the neural network is pre-trained using the neural network training method described in the first aspect.

[0136] According to a third aspect of the embodiments of this disclosure, a neural network training apparatus is provided. Please refer to the appendix. Figure 5 ,include:

[0137] The input module 501 is used to input the training audio into the encoding sub-network for encoding to obtain the first encoding result, and input the text label corresponding to the training audio into the prediction sub-network for prediction to obtain the first prediction result;

[0138] The first joint module 502 is used to combine the first encoding result and the first prediction result to obtain a first joint result;

[0139] The trimming module 503 is used to trim the first encoding result and the first prediction result according to the first joint result to obtain the corresponding second encoding result and second prediction result.

[0140] The second joint module 504 is used to input the second encoding result and the second prediction result into the joint sub-network for joint processing to obtain the second joint result, and to adjust the network parameters of the encoding sub-network, the prediction sub-network and the joint sub-network according to the second joint result.

[0141] In some embodiments of this disclosure, the first joint module is specifically used for:

[0142] The first encoding result and the first prediction result are input into the joint module for joint processing to obtain the first joint result.

[0143] In some embodiments of this disclosure, each element in the first joint result has at least coordinates in the audio frame number dimension and coordinates in the text label sequence dimension.

[0144] In some embodiments of this disclosure, the cropping module is specifically used for:

[0145] Obtain the gradient information of each element in the first joint result;

[0146] Based on the gradient information of each element in the first joint result, the clipping boundary is determined in the plane composed of the audio frame number dimension and the text label sequence dimension;

[0147] Based on the pruning boundary, the first encoding result and the first prediction result are pruned respectively to obtain the corresponding second encoding result and the second prediction result.

[0148] In some embodiments of this disclosure, when the pruning module is used to obtain the gradient information of each element in the first joint result, it is specifically used for:

[0149] Obtain the gradient information of the probability of outputting text labels for each element in the first joint result, and the gradient information of the probability of outputting empty characters.

[0150] In some embodiments of this disclosure, when the cropping module determines the cropping boundary in the plane composed of the audio frame number dimension and the text tag sequence dimension based on the gradient information of each element in the first joint result, it is specifically used for:

[0151] In the t-th frame of the audio frame number dimension, the sum of the gradient information of each element in the sequence interval of a preset size starting from each index in the text label sequence dimension is determined as the total gradient of the corresponding index.

[0152] The sequence interval of a preset size, starting from the index with the largest total gradient, is determined as the clipping boundary of the t-th frame.

[0153] In some embodiments of this disclosure, the index of the largest total gradient also satisfies at least one of the following:

[0154] The difference between the index of the largest total gradient and the index of the largest dimension of the text label sequence is less than or equal to a preset size.

[0155] The index of the largest total gradient is less than or equal to the starting index of the clipping boundary of the (t+1)th frame;

[0156] The difference between the index of the largest gradient and the starting index of the cropping boundary of the (t+1)th frame is less than the preset size.

[0157] In some embodiments of this disclosure, the cropping module is used to crop the first encoding result and the first prediction result according to the cropping boundary, respectively, to obtain the second encoding result and the second prediction result, specifically for:

[0158] The first encoding result and the first prediction result are respectively expanded in at least one dimension so that each element of the first encoding result has coordinates in the audio frame number dimension and the text label sequence dimension;

[0159] Within the plane formed by the audio frame number dimension and the text label sequence dimension, the first encoding result and the first prediction result are cropped according to the cropping boundary to obtain the second encoding result and the second prediction result.

[0160] In some embodiments of this disclosure, when the second joint module adjusts the network parameters of the coding subnetwork, the prediction subnetwork, and the joint subnetwork based on the second joint result, it is specifically used for:

[0161] A first network loss value is determined based on the second joint result, and the network parameters of the coding sub-network, the prediction sub-network, and the joint sub-network are adjusted based on the first network loss value.

[0162] In some embodiments of this disclosure, a loss module is also included, for:

[0163] The second network loss value is determined based on the first joint result;

[0164] When the second joint module adjusts the network parameters of the coding sub-network, the prediction sub-network, and the joint sub-network based on the first network loss value, it is specifically used for:

[0165] The network parameters of the coding subnetwork, the prediction subnetwork, and the joint subnetwork are adjusted based on the first network loss value and the second network loss value.

[0166] According to a fourth aspect of the present disclosure, an audio processing apparatus is provided, including an input module for:

[0167] The audio to be processed is input into a neural network to obtain the target text, wherein the neural network is pre-trained using the neural network training method described in the first aspect.

[0168] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments of the method in the first aspect, and will not be elaborated upon here.

[0169] According to a third aspect of the embodiments of this disclosure, please refer to the appendix. Figure 6 The diagram illustrates, for example, a block diagram of an electronic device. For instance, device 600 could be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0170] Reference Figure 6 The device 600 may include one or more of the following components: a processing component 602, a memory 604, a power supply component 606, a multimedia component 608, an audio component 610, an input / output (T / O) interface 612, a sensor component 614, and a communication component 616.

[0171] Processing component 602 typically controls the overall operation of device 600, such as operations associated with display, telephone calls, data communication, camera program operation, and recording operation. Processing component 602 may include one or more processors 620 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 602 may include one or more modules to facilitate interaction between processing component 602 and other components. For example, processing component 602 may include a multimedia module to facilitate interaction between multimedia component 608 and processing component 602.

[0172] Memory 604 is configured to store various types of data to support the operation of device 600. Examples of this data include instructions for any application or method operating on device 600, contact data, phonebook data, messages, pictures, videos, etc. Memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0173] The power supply component 606 provides power to the various components of the device 600. The power supply component 606 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 600.

[0174] Multimedia component 608 includes a screen that provides an output interface between the device 600 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 608 includes a front-facing camera and / or a rear-facing camera. When the device 600 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0175] Audio component 610 is configured to output and / or input audio signals. For example, audio component 610 includes a microphone (MTC) configured to receive external audio signals when device 600 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 604 or transmitted via communication component 616. In some embodiments, audio component 610 also includes a speaker for outputting audio signals.

[0176] T / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, start buttons, and lock buttons.

[0177] Sensor assembly 614 includes one or more sensors for providing state assessments of various aspects of device 600. For example, sensor assembly 614 may detect the on / off state of device 600, the relative positioning of components such as the display and keypad of device 600, changes in position of device 600 or a component of device 600, the presence or absence of user contact with device 600, the orientation or acceleration / deceleration of device 600, and temperature changes of device 600. Sensor assembly 614 may also include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 614 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0178] Communication component 616 is configured to facilitate wired or wireless communication between device 600 and other devices. Device 600 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G or 5G, or combinations thereof. In one exemplary embodiment, communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 616 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFTD) technology, Infrared Data Association (TrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0179] In an exemplary embodiment, the device 600 may be implemented by one or more application-specific integrated circuits (ASTC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field-programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors, or other electronic components to perform the power supply method of the aforementioned electronic device.

[0180] Fourthly, in exemplary embodiments, this disclosure also provides a non-transitory computer-readable storage medium including instructions, such as a memory 604 including instructions. These instructions can be executed by a processor 620 of the device 600 to complete the power supply method for the electronic device. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0181] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0182] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A neural network training method, characterized by, The method comprises the following steps: inputting a training audio into an encoding subnetwork for encoding to obtain a first encoding result, and inputting a text label corresponding to the training audio into a prediction subnetwork for prediction to obtain a first prediction result; jointly processing the first encoding result and the first prediction result to obtain a first joint result; according to the first joint result, respectively clipping the first encoding result and the first prediction result to obtain corresponding second encoding result and second prediction result; inputting the second encoding result and the second prediction result into a joint subnetwork for joint processing to obtain a second joint result, and adjusting network parameters of the encoding subnetwork, the prediction subnetwork and the joint subnetwork according to the second joint result; each element in the first joint result has at least a coordinate in an audio frame number dimension and a coordinate in a text label sequence dimension; the clipping of the first encoding result and the first prediction result according to the first joint result to obtain the corresponding second encoding result and the second prediction result comprises: obtaining gradient information of each element in the first joint result; determining a clipping boundary in a plane composed of the audio frame number dimension and the text label sequence dimension according to the gradient information of each element in the first joint result; clipping the first encoding result and the first prediction result according to the clipping boundary to obtain the corresponding second encoding result and the second prediction result.

2. The neural network training method of claim 1, wherein, the joint processing of the first encoding result and the first prediction result to obtain the first joint result comprises: inputting the first encoding result and the first prediction result into a joint module for joint processing to obtain the first joint result.

3. The neural network training method of claim 1, wherein, the obtaining of the gradient information of each element in the first joint result comprises: obtaining gradient information of a probability of outputting a text label and gradient information of a probability of outputting a null character of each element in the first joint result.

4. The neural network training method of claim 1, wherein, the determining of the clipping boundary in the plane composed of the audio frame number dimension and the text label sequence dimension according to the gradient information of each element in the first joint result comprises: determining a sum of gradient information of each element in a sequence interval of a preset size with each sequence number as a starting point in the text label sequence dimension under the tth frame in the audio frame number dimension, as a gradient total amount of the corresponding sequence number, wherein t is an integer not less than 1; determining the sequence interval of the preset size with the sequence number as the starting point and having the maximum gradient total amount as the clipping boundary of the tth frame.

5. The neural network training method of claim 4, wherein, the sequence number with the maximum gradient total amount also satisfies at least one of the following conditions: a difference between the sequence number with the maximum gradient total amount and a maximum sequence number in the text label sequence dimension is greater than or equal to the preset size; the sequence number with the maximum gradient total amount is less than or equal to a starting sequence number of a clipping boundary of a (t+1)th frame; a difference between the sequence number with the maximum gradient total amount and the starting sequence number of the clipping boundary of the (t+1)th frame is less than the preset size.

6. The neural network training method of claim 1, wherein, the clipping of the first encoding result and the first prediction result according to the clipping boundary to obtain the corresponding second encoding result and the second prediction result comprises: The first encoding result and the first prediction result are respectively expanded in at least one dimension, so that each element of the first encoding result has coordinates in an audio frame number dimension and a text label sequence dimension; According to the clipping boundary, the first encoding result and the first prediction result are respectively clipped to obtain a second encoding result and a second prediction result.

7. The neural network training method of any one of claims 1 to 6, characterized in that, The adjusting of the network parameters of the encoding sub-network, the prediction sub-network and the joint sub-network according to the second joint result comprises: A first network loss value is determined according to the second joint result, and the network parameters of the encoding sub-network, the prediction sub-network and the joint sub-network are adjusted according to the first network loss value.

8. The neural network training method of claim 7, wherein, Further comprising: A second network loss value is determined according to the first joint result. The adjusting of the network parameters of the encoding sub-network, the prediction sub-network and the joint sub-network according to the first network loss value comprises: The network parameters of the encoding sub-network, the prediction sub-network and the joint sub-network are adjusted according to the first network loss value and the second network loss value.

9. An audio processing method, characterized by, Comprise: Input the audio to be processed into a neural network to obtain the target text, wherein the neural network is trained by using the neural network training method of any one of claims 1 to 8.

10. A neural network training apparatus characterized by comprising: Comprise: An input module is configured to input training audio into an encoding sub-network to obtain a first encoding result, and input a text label corresponding to the training audio into a prediction sub-network to obtain a first prediction result; A first joint module is configured to jointly process the first encoding result and the first prediction result to obtain a first joint result; A clipping module is configured to clip the first encoding result and the first prediction result according to the first joint result to obtain a corresponding second encoding result and a second prediction result; A second joint module is configured to input the second encoding result and the second prediction result into a joint sub-network to obtain a second joint result, and adjust network parameters of the encoding sub-network, the prediction sub-network and the joint sub-network according to the second joint result; Each element in the first joint result has at least coordinates in an audio frame number dimension and coordinates in a text label sequence dimension; The clipping module is specifically configured to: Obtain gradient information of each element in the first joint result; Determine a clipping boundary in a plane formed by the audio frame number dimension and the text label sequence dimension according to the gradient information of each element in the first joint result; According to the clipping boundary, the first encoding result and the first prediction result are respectively clipped to obtain a corresponding second encoding result and a second prediction result.

11. The neural network training apparatus according to claim 10, characterized in that, The first joint module is specifically configured to: Input the first encoding result and the first prediction result into a joint module to obtain a first joint result.

12. The neural network training apparatus according to claim 10, characterized by, The clipping module is configured to obtain gradient information of each element in the first joint result, and specifically configured to: obtain gradient information of a probability of outputting a text label and gradient information of a probability of outputting a null character for each element in the first joint result.

13. The neural network training apparatus of claim 10, wherein, The clipping module is configured to determine a clipping boundary in a plane formed by the audio frame number dimension and the text label sequence dimension according to the gradient information of each element in the first joint result, and specifically configured to: determine a sum of gradient information of each element in a sequence interval of a preset size with a starting point being each sequence number in the text label sequence dimension under the t-th frame of the audio frame number dimension, as a gradient total amount corresponding to the sequence number; determine a sequence interval of the preset size with a starting point being a sequence number with the largest gradient total amount as the clipping boundary of the t-th frame.

14. The neural network training apparatus according to claim 13, characterized by, The sequence number with the largest gradient total amount also satisfies at least one of the following conditions: a difference between the sequence number with the largest gradient total amount and a maximum sequence number in the text label sequence dimension is greater than or equal to the preset size; the sequence number with the largest gradient total amount is less than or equal to a starting point sequence number of a clipping boundary of a (t+1)-th frame; a difference between the sequence number with the largest gradient total amount and the starting point sequence number of the clipping boundary of the (t+1)-th frame is less than the preset size. 15.The neural network training apparatus of claim 10, wherein, The clipping module is configured to clip the first encoding result and the first prediction result according to the clipping boundary, respectively, to obtain a second encoding result and a second prediction result, and specifically configured to: expand the first encoding result and the first prediction result in at least one dimension, respectively, so that each element of the first encoding result has coordinates in the audio frame number dimension and the text label sequence dimension; clip the first encoding result and the first prediction result according to the clipping boundary in the plane formed by the audio frame number dimension and the text label sequence dimension, respectively, to obtain a second encoding result and a second prediction result. 16.The neural network training apparatus of any one of claims 10-15, wherein, The second joint module is configured to adjust network parameters of the encoding subnetwork, the prediction subnetwork and the joint subnetwork according to the second joint result, and specifically configured to: determine a first network loss value according to the second joint result, and adjust the network parameters of the encoding subnetwork, the prediction subnetwork and the joint subnetwork according to the first network loss value.

17. The neural network training apparatus of claim 16, wherein, Further comprising a loss module configured to: determine a second network loss value according to the first joint result. The second joint module is configured to adjust network parameters of the encoding subnetwork, the prediction subnetwork and the joint subnetwork according to the first network loss value, and specifically configured to: adjust the network parameters of the encoding subnetwork, the prediction subnetwork and the joint subnetwork according to the first network loss value and the second network loss value.

18. An audio processing device, comprising: The input module is configured to: input an audio to be processed into a neural network to obtain a target text, wherein the neural network is trained by using the neural network training method in any one of claims 1 to 8.

19. An electronic device, comprising: The electronic device comprises a memory for storing computer instructions executable on a processor, and a processor for executing the computer instructions to implement the method according to any one of claims 1-9.

20. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method according to any one of claims 1-9.

Images