Speech recognition model training method and device, computer equipment and storage medium

Abstract

The invention belongs to the technical field of finance, medical treatment and artificial intelligence, and discloses a voice recognition model training method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining specific task voice training data, defining a specific task input instruction according to a specific scene of voice recognition, adjusting an input format of a prior text sequence of the pre-trained speech recognition model; inputting the prior text sequence with the adjusted input format and the audio coding vector into a decoder of a pre-training speech recognition model for decoding, and predicting a character sequence output by the pre-training speech recognition model according to a decoding result; defining a monotonicity limitation loss function according to the predicted character sequence, and constructing a target loss function based on the defined monotonicity limitation loss function; and updating the voice recognition model during training based on the target loss function to obtain a trained voice recognition model. According to the invention, the speech recognition accuracy of the speech recognition model in a specific scene is improved.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a training method, device, computer equipment and storage medium for a speech recognition model. Background Art

[0002] Speech recognition technology is a technology that can convert human speech signals into corresponding text or instructions. It has a wide range of applications in artificial intelligence, natural language processing, machine translation and other fields. Traditional speech recognition often relies mainly on manual input, which is not only inefficient but also generates high labor costs. With the help of training on massive Internet speech data, speech recognition models have the ability to recognize multiple languages ​​naturally and fluently, which greatly improves the efficiency of speech recognition work and significantly reduces labor costs.

[0003] Pre-trained speech recognition models are usually trained on large-scale general speech data and can handle a wide range of speech recognition scenarios. However, in the face of specific speech recognition scenarios such as finance and medical care, the speech recognition effect of the model is not good. Fine-tuning the speech recognition model during training can make it better adapt to the needs of these specific scenarios, thereby improving the recognition accuracy of the speech recognition model in specific scenarios. Traditional speech recognition model fine-tuning methods mostly use the Transformer structure as the basic architecture. First, collect and label speech data for specific scenarios, and then use this data to train the pre-trained speech model. However, the Transformer structure lacks the monotonic alignment characteristics of the input speech signal and the output text that are unique to speech recognition tasks, and lacks the command input for specific tasks, resulting in the trained speech recognition model having a low accuracy rate in speech recognition in specific scenarios. Summary of the invention

[0004] The present invention provides a method, apparatus, computer equipment and storage medium for training a speech recognition model to solve the technical problem that the existing speech recognition model has poor speech recognition effect in specific scenarios.

[0005] In a first aspect, a method for training a speech recognition model is provided, comprising:

[0006] Acquire a pre-trained speech recognition model and multi-task speech training data corresponding to the pre-trained speech recognition model, and perform data preprocessing on the acquired multi-task speech training data;

[0007] Performing audio feature extraction on the multi-task speech training data after data preprocessing to obtain an audio feature sequence of the multi-task speech training data;

[0008] Inputting the extracted audio feature sequence of the multi-task speech training data into the encoder of the pre-trained speech recognition model for encoding to obtain an audio encoding vector of the multi-task speech training data;

[0009] Acquire task-specific speech training data, define task-specific input instructions according to a specific scenario of speech recognition, and adjust the input format of a priori text sequence of a pre-trained speech recognition model according to the defined task-specific input instructions;

[0010] Inputting the prior text sequence after adjusting the input format and the audio encoding vector of the multi-task speech training data into the decoder of the pre-trained speech recognition model for decoding, and predicting the character sequence output by the pre-trained speech recognition model according to the decoding result;

[0011] Constructing a monotonicity restriction loss function of the speech recognition model during training according to the predicted character sequence output by the pre-trained speech recognition model, and constructing a target loss function of the speech recognition model during training based on the constructed monotonicity restriction loss function;

[0012] The speech recognition model during training is updated based on the constructed target loss function to obtain a trained speech recognition model.

[0013] In a second aspect, a training device for a speech recognition model is provided, wherein the device is used to implement the training method for the speech recognition model as described in the first aspect, including:

[0014] An acquisition module, used to acquire a pre-trained speech recognition model and multi-task speech training data corresponding to the pre-trained speech recognition model, and perform data preprocessing on the acquired multi-task speech training data;

[0015] A feature extraction module is used to extract audio features from the multi-task speech training data after data preprocessing to obtain an audio feature sequence of the multi-task speech training data;

[0016] An encoding module, used for inputting the extracted audio feature sequence of the multi-task speech training data into the encoder of the pre-trained speech recognition model for encoding, so as to obtain an audio encoding vector of the multi-task speech training data;

[0017] An adjustment module, used to obtain task-specific speech training data, define task-specific input instructions according to a specific scenario of speech recognition, and adjust the input format of a priori text sequence of a pre-trained speech recognition model according to the defined task-specific input instructions;

[0018] A decoding module, used to input the prior text sequence after adjusting the input format and the audio encoding vector of the multi-task speech training data into the decoder of the pre-trained speech recognition model for decoding, and predict the character sequence output by the pre-trained speech recognition model according to the decoding result;

[0019] A construction module, used to construct a monotonicity restriction loss function of the speech recognition model during training according to the character sequence output by the predicted pre-trained speech recognition model, and to construct a target loss function of the speech recognition model during training based on the constructed monotonicity restriction loss function;

[0020] The updating module is used to update the speech recognition model during training based on the constructed target loss function to obtain a trained speech recognition model.

[0021] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned speech recognition model training method when executing the computer program.

[0022] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the training method of the above-mentioned speech recognition model are implemented.

[0023] In the scheme implemented by the training method, device, computer equipment and storage medium based on the above-mentioned speech recognition model, a pre-trained speech recognition model and multi-task speech training data corresponding to the pre-trained speech recognition model can be obtained through the client, and the obtained multi-task speech training data can be pre-processed; audio features are extracted from the multi-task speech training data after data pre-processing to obtain an audio feature sequence of the multi-task speech training data; the extracted audio feature sequence of the multi-task speech training data is input into the encoder of the pre-trained speech recognition model for encoding to obtain an audio encoding vector of the multi-task speech training data; specific task speech training data is obtained, specific task input instructions are defined according to the specific scenario of speech recognition, and the input format of the prior text sequence of the pre-trained speech recognition model is adjusted according to the defined specific task input instructions; the prior text sequence after the input format adjustment and the audio encoding vector of the multi-task speech training data are input into the encoder of the pre-trained speech recognition model for encoding to obtain an audio encoding vector of the multi-task speech training data. The speech recognition model is input into a decoder of a pre-trained speech recognition model for decoding, and a character sequence output by the pre-trained speech recognition model is predicted based on the decoding result; a monotonicity restriction loss function of the speech recognition model during training is constructed based on the predicted character sequence output by the pre-trained speech recognition model, and a target loss function of the speech recognition model during training is constructed based on the constructed monotonicity restriction loss function; the speech recognition model during training is updated based on the constructed target loss function to obtain a trained speech recognition model. In the present invention, specific task input instructions are defined for specific scenarios of speech recognition to guide the speech recognition model to pay more attention to the semantics of specific scenarios during training, and a monotonicity restriction loss function of the speech recognition model during training is constructed, so that the speech recognition model can pay attention to the monotonic alignment characteristics of the input speech signal and the output text sequence of the specific scenario, thereby improving the accuracy of the speech recognition model in speech recognition in specific scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0025] Figure 1 is a schematic diagram of an application environment of a method for training a speech recognition model in one embodiment of the present invention;

[0026] Figure 2 is a flow chart of a method for training a speech recognition model in one embodiment of the present invention;

[0027] Figure 3 A schematic diagram of the model structure of a speech recognition model provided by an embodiment of the present invention;

[0028] Figure 4 yes Figure 2 A schematic flow chart of a specific implementation of step S10;

[0029] Figure 5 yes Figure 2 A schematic flow chart of a specific implementation of step S20;

[0030] Figure 6 yes Figure 2 A schematic flow chart of a specific implementation of step S30;

[0031] Figure 7 yes Figure 2 A schematic flow chart of a specific implementation of step S50;

[0032] Figure 8 is a structural schematic diagram of a training device for a speech recognition model in one embodiment of the present invention;

[0033] Fig. 9 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0034] Fig.10 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] The training method of the speech recognition model provided by the embodiment of the present invention can be applied to Figure 1 In the application environment, Figure 1It is a schematic diagram of an application environment of a method for training a speech recognition model in one embodiment of the present invention; wherein, the client communicates with the server via a network. The server can obtain a pre-trained speech recognition model and multi-task speech training data corresponding to the pre-trained speech recognition model through the client, and perform data preprocessing on the obtained multi-task speech training data; perform audio feature extraction on the multi-task speech training data after data preprocessing to obtain an audio feature sequence of the multi-task speech training data; input the extracted audio feature sequence of the multi-task speech training data into the encoder of the pre-trained speech recognition model for encoding to obtain an audio coding vector of the multi-task speech training data; obtain specific task speech training data, define specific task input instructions according to specific scenarios of speech recognition, and adjust the input format of the prior text sequence of the pre-trained speech recognition model according to the defined specific task input instructions; input the prior text sequence after adjusting the input format and the audio coding vector of the multi-task speech training data into the decoder of the pre-trained speech recognition model. Decoding is performed, and the character sequence output by the pre-trained speech recognition model is predicted according to the decoding result; a monotonicity restriction loss function of the speech recognition model during training is constructed according to the predicted character sequence output by the pre-trained speech recognition model, and a target loss function of the speech recognition model during training is constructed based on the constructed monotonicity restriction loss function; the speech recognition model during training is updated based on the constructed target loss function to obtain a trained speech recognition model. In the present invention, specific task input instructions are defined for specific scenarios of speech recognition to guide the speech recognition model to pay more attention to the semantics of specific scenarios during training, and a monotonicity restriction loss function of the speech recognition model during training is constructed, so that the speech recognition model can pay attention to the monotonic alignment characteristics of the input speech signal and the output text sequence of the specific scenario, thereby improving the accuracy of the speech recognition model in speech recognition in specific scenarios. The present invention is described in detail below through specific embodiments.

[0037] See also Figure 2-3 As shown, Figure 2 A flow chart of a method for training a speech recognition model provided in an embodiment of the present invention, Figure 3 A schematic diagram of a model structure of a speech recognition model provided in an embodiment of the present invention, wherein a training method of the speech recognition model comprises the following steps:

[0038] S10: Acquire a pre-trained speech recognition model and multi-task speech training data corresponding to the pre-trained speech recognition model, and perform data preprocessing on the acquired multi-task speech training data.

[0039] When obtaining a pre-trained speech recognition model, the present invention can obtain the pre-trained speech recognition model from an open source platform. Specifically, the model can be selected according to the type of language that needs to be processed in the actual application scenario. For example, if multiple speech recognition tasks are to be processed, a pre-trained speech recognition model that supports multiple languages ​​is selected. The multi-task speech training data corresponding to the pre-trained speech recognition model can be downloaded from the official channel designated by the institution or team that obtained the pre-trained language recognition model, or it can be obtained by integrating public data sets.

[0040] After obtaining the multi-task speech training data, it is necessary to perform data preprocessing on the obtained multi-task speech training data to ensure the quality and standardization of the multi-task speech training data. Specifically, Figure 4 As stated, Figure 4 yes Figure 2 A specific implementation flow diagram of step S10 in the embodiment of the present invention specifically includes the following steps S11-S14:

[0041] S11: The speech signal in the multi-task speech training data is framed, and the speech signal after the frame processing is windowed. Specifically, the present invention first determines the two key parameters of the frame length and frame interval of the speech signal. The frame length is generally selected to be around 20-40 milliseconds, which determines the duration of each frame containing the speech signal, and the frame interval represents the time interval between two adjacent frames. By sequentially intercepting the corresponding time segments from the starting position of the speech signal according to the set frame length and frame interval, the frame processing is completed to obtain a series of discrete speech frames. For example, for a speech signal with a total duration of 5 seconds, framing with a frame length of 25 milliseconds and a frame interval of 10 milliseconds will obtain multiple speech with a length of 25 milliseconds and a sequential interval of 10 milliseconds. After the speech signal is framed, each frame of the speech signal can be multiplied by a specific window function so that each frame of the signal can smoothly transition at the truncation point, thereby making the subsequent spectrum analysis more accurate.

[0042] S12: Performing discrete Fourier transform processing on the speech signal after windowing processing to obtain the frequency spectrum characteristics of the multi-task speech training data. Specifically, the present invention calls the corresponding discrete Fourier transform function to calculate each frame of speech signal after windowing processing to obtain the frequency spectrum characteristics of each frame of speech signal, that is, to obtain the amplitude spectrum and phase spectrum of each frame of speech signal.

[0043] S13: Using a Mel frequency filter to convert the spectral features of the multi-task speech training data to obtain the Mel spectrum of the multi-task speech training data. Specifically, the present invention uses the constructed Mel frequency filter to filter the spectral features of the multi-task speech training data obtained in step S12 to obtain the energy distribution of the multi-task speech training data on the Mel frequency scale, that is, to obtain the Mel spectrum of the multi-task speech training data.

[0044] S14: Performing logarithmic operation processing on the Mel spectrum of the multi-task speech training data to obtain a logarithmic Mel spectrum diagram of the multi-task speech training data. Specifically, the present invention can perform logarithmic operation processing on the Mel spectrum of the multi-task speech training data. Since the energy difference of speech signals in different Mel frequency intervals is large, the difference between the energy of speech signals can be relatively reduced after taking the logarithm, so as to facilitate the processing of the subsequent speech recognition model, and the Mel spectrum after taking the logarithm is more consistent with the human ear's perception of sound intensity.

[0045] S20: extracting audio features from the multi-task speech training data after data preprocessing to obtain an audio feature sequence of the multi-task speech training data. The present invention can convert the original complex speech signal into a low-dimensional audio feature sequence by extracting audio features from the multi-task speech training data, and extract information reflecting the key characteristics of the speech. Specifically, Figure 5 As stated, Figure 5 yes Figure 2 A specific implementation flow diagram of step S20 in the embodiment of the present invention specifically includes the following steps S21-S22:

[0046] S21: Input the multi-task speech training data after data preprocessing into the convolution layer, and extract the audio feature vector of the multi-task speech training data. Specifically, in the present invention, the preprocessed multi-task speech training data is sequentially sent to the convolution layer in a set batch format. Feature information related to different positions and different local areas is extracted. For each speech sample in the multi-task speech training data, after being processed by multiple convolution layers, corresponding feature maps will be obtained on different channels, and these feature maps will represent the audio feature vectors of the task speech training data after fusion.

[0047] S22: Input the extracted audio feature information of the multi-task speech training data into a first position encoder for position encoding to obtain an audio feature sequence of the multi-task speech training data.

[0048] After step S21, the audio feature vectors of the multi-task speech training data have been obtained. These feature vectors contain some acoustic feature information of the audio itself, but lack position-related information. A sinusoidal position encoder can be used to generate coding values ​​corresponding to different positions. For each element in the audio feature vector, the corresponding position coding information is found in the sinusoidal position encoder according to its sequential position in the audio feature vector, and the position coding information is integrated into the audio feature vector to obtain an audio feature sequence containing position information.

[0049] S30: Input the extracted audio feature sequence of the multi-task speech training data into the encoder of the pre-trained speech recognition model for encoding, so as to obtain an audio encoding vector of the multi-task speech training data.

[0050] After the audio feature sequence of the multi-task speech training data is obtained in step S20, the extracted audio feature sequence of the multi-task speech training data is input into the encoder of the pre-trained speech recognition model for encoding to obtain the audio encoding vector of the multi-task speech training data. Specifically, Figure 6 As stated, Figure 6 yes Figure 2 A specific implementation flow diagram of step S30 in the embodiment of the present invention specifically includes the following steps S31-S32:

[0051] S31: Input the extracted audio feature sequence of the multi-task speech training data into the self-attention mechanism layer of the encoder, and assign self-attention weights to the audio feature sequence of the multi-task speech training data. Specifically, in the present invention, according to the specific content of the input speech feature sequence, the self-attention mechanism can adaptively adjust the feature representation of each position and integrate information from different positions. For example, the association of the internal elements of the speech of different speakers, speech of different speaking speeds, etc. is different. The self-attention mechanism can dynamically change the weight distribution of the feature vector according to these differences, so that the output feature sequence is more in line with the actual speech situation, which is convenient for subsequent further processing.

[0052] S32: Input the audio sequence after assigning the self-attention weight to the feedforward linear layer of the encoder, perform nonlinear change processing on the audio sequence, and obtain the audio encoding vector of the multi-task speech training data. Specifically, in the present invention, the audio sequence processed by the self-attention mechanism often has a certain dimension, and the feedforward linear layer can perform nonlinear transformation on the dimension of the input feature by adjusting the number of neurons, etc. High-dimensional and complex input features can be mapped to a relatively low-dimensional and more compact feature space, which plays a role in feature integration. During the model training process, the weight parameters of the feedforward linear layer will be continuously adjusted and optimized according to the training data and the set loss function, so that the feature vector output by it can be better applied to specific speech recognition scenarios.

[0053] S40: Acquire task-specific speech training data, define task-specific input instructions according to a specific scenario of speech recognition, and adjust the input format of the prior text sequence of the pre-trained speech recognition model according to the defined task-specific input instructions.

[0054] In order to improve the speech recognition ability in specific scenarios, the present invention needs to first collect specific task speech training data in the fields of finance, medical care, etc., define specific task input instructions according to the specific scenarios of speech recognition, and use the specific task input instructions as prior information for fine-tuning the speech recognition model to assist the model in optimizing the recognition performance in specific scenarios.

[0055] In the pre-trained speech recognition model, the inputs of the encoder and decoder are audio feature sequences and text sequences, respectively. The text sequence is usually based on the “ <sot><language id><task id><timestamp><text>". After defining a specific task input instruction according to a specific scenario of speech recognition, the present invention can adjust the input format of the priori text sequence of the pre-trained speech recognition model according to the defined specific task input instruction, and adjust the input format of the priori text sequence of the pre-trained speech recognition model to " <sop><Instruction> <sot><language id><task id><timestamp><text>” format.

[0056] When processing the acquired specific task speech training data, the present invention first defines the specific task output instruction, then starts transcribing the acquired specific task speech training data, detects the language of the input specific task speech training data, and ends the transcription if it is detected that the input specific task language training data is in a silent state; if it is detected that the input specific task language training data is in a non-silent state, transcribes or translates the input specific task language training data according to the input specific task instruction; after transcribing or translating the input specific task language training data, it is also necessary to determine whether timestamp information is input; if the timestamp information is input, time-aligned transcription is performed according to the input timestamp information, and a priori text sequence is decoded; if the timestamp information is not input, the priori text sequence is directly decoded.

[0057] S50: Inputting the prior text sequence after adjusting the input format and the audio encoding vector of the multi-task speech training data into the decoder of the pre-trained speech recognition model for decoding, and predicting the character sequence output by the pre-trained speech recognition model according to the decoding result.

[0058] After adjusting the input format of the priori text sequence of the pre-trained speech recognition model in step S50, the priori text sequence after the input format adjustment and the audio encoding vector of the multi-task speech training data can be input into the decoder of the pre-trained speech recognition model for decoding, and the character sequence output by the pre-trained speech recognition model can be predicted based on the decoding result. Specifically, Figure 7 As stated, Figure 7 yes Figure 2 A specific implementation flow diagram of step S50 in the embodiment of the present invention specifically includes the following steps S51-S55:

[0059] S51: Input the prior text sequence after adjusting the input format into the second position encoder, perform position encoding on the text feature sequence in the specific task speech training data, and obtain the text encoding vector of the specific task speech training data. For the prior text sequence after adjusting the input format, find the corresponding position encoding information in the second position encoder according to its sequential position in the sequence, and obtain the text encoding vector of the specific task speech training data carrying the position encoding information.

[0060] S52: Input the text encoding vector of the specific task speech training data and the audio encoding vector of the multi-task speech training data into the self-attention mechanism layer of the decoder, and assign self-attention weights to the text encoding vector of the specific task speech training data and the audio encoding vector of the multi-task speech training data. Specifically, in the present invention, the attention mechanism layer of the decoder comprehensively considers the semantic information of each part such as the beginning, middle and end of the text sequence, so that the final generated prior text sequence is more consistent with the input of the pre-trained speech recognition model decoder.

[0061] S53: Input the text encoding vector of the specific task speech training data and the audio encoding vector of the multi-task speech training data after the self-attention weights are assigned to the cross-attention mechanism layer of the decoder, and fuse the text encoding vector of the specific task speech training data and the audio encoding vector of the multi-task speech training data to obtain the target encoding vector. Specifically, the present invention establishes an association between the text encoding vector sequence and the audio encoding vector sequence through the cross-attention mechanism layer, realizes the deep fusion of the encoding vector, and enables the model to better utilize the feature information of text and audio for speech recognition.

[0062] S54: Input the target coding vector into the feedforward linear layer of the decoder, and perform nonlinear change processing on the target coding vector. Specifically, the present invention inputs the target coding vector into the feedforward linear layer of the decoder, and through the fully connected relationship between neurons, the current coding information in each dimension can be fully considered.

[0063] S55: Based on the target coding vector after nonlinear change processing, predict the character sequence output by the pre-trained speech recognition model to obtain the prediction result of the character sequence output by the pre-trained speech recognition model. The present invention predicts the character sequence output by the pre-trained speech recognition model based on the learned model parameters and the feature information contained in the target coding vector, and can calculate the probability values ​​of different characters appearing at corresponding positions according to the target coding vector, and select the character with the largest probability value as the character at the corresponding position in the prediction result according to the calculated probability distribution of the character appearance.

[0064] S60: Define a monotonicity restriction loss function of the speech recognition model during training according to the predicted character sequence output by the pre-trained speech recognition model, and construct a target loss function of the speech recognition model during training based on the defined monotonicity restriction loss function.

[0065] The structure of the decoder of the pre-trained speech recognition model in the present invention is composed of 32 layers of sub-modules, each of which is composed of 20 self-attention units, 20 cross-attention units and a feed-forward linear layer. The present invention adds a monotonicity restriction loss function to the cross-attention matrix. Since the 20 cross-attention heads pay different attention to the alignment characteristics of the input and output, and it is impossible to intuitively select a specific head to add monotonicity restriction, the present invention defines a monotonicity restriction loss function weight λ that changes with the gradient for each head self-attention unit. i , with an initial value of 0.1. As the training process converges, the model will gradually learn the optimal weights for each head.

[0066] The mapping vector δ of the expected alignment position of the input audio sequence corresponding to the character sequence output by the decoder of the present invention is i As shown in formula (1):

[0067]

[0068] Among them, i is the i-th character output by the decoder, j is the j-th element of the input audio sequence, T1 is the length of the input audio feature sequence, α is the attention matrix of the cross attention unit, and I is the position of the input audio sequence.

[0069] Among them, in order to construct the monotonicity restriction loss function of the speech recognition model during training, the position mapping vector Δδ i The calculation formula is shown in formula (2):

[0070] Δδ i =δ i -δ i-1 ,1≤i≤T2-1,Δδ i ≥1 (2)

[0071] Where T2 is the length of the output text sequence, Δδ i is the difference between the character sequence currently output by the pre-trained speech recognition model and the character sequence output at the previous moment; the difference must be greater than or equal to 1 to ensure that each output character corresponds to at least one frame of input audio. Based on the above restrictions, the monotonicity restriction loss function of the speech recognition model during training is defined according to the predicted character sequence output by the pre-trained speech recognition model, where the monotonicity restriction loss function of the i-th cross-attention layer is shown in formula (3):

[0072] L monoi =Relu(1-Δδ) (3)

[0073] Among them, L monoi represents the monotonicity restriction loss function of the i-th cross attention layer, Relu(.) represents the activation function, △δ is the difference between the character sequence currently output by the pre-trained speech recognition model and the character sequence output at the previous moment, and the calculation formula of the monotonicity restriction loss function of the speech recognition model during training is shown in formula (4)

[0074] L mono =λ1L momo1 +λ2L momo2 +…+λ 20 L momo20 (4)

[0075] Among them, L mono represents the monotonicity restriction loss function of the speech recognition model during training, λ i Represents the weight of the monotonicity constraint loss function of the speech recognition model during training.

[0076] In addition, the output of the last feed-forward linear layer of the decoder of the pre-trained speech recognition model uses a cross-entropy loss function, and the calculation formula of the cross-entropy loss function is shown in formula (5):

[0077] L CE (p,q)=-∑p(x)log(q(x)) (5)

[0078] Among them, L CE (p,q) represents the cross entropy loss function of the speech recognition model during training, p(x) represents the actual character sequence, and q(x) represents the predicted output character sequence.

[0079] In addition, when calculating the loss of the matching degree between the character sequence q(x) generated by the training optimization model and the actual character sequence p(x), since the character sequence q(x) generated by the model does not contain the instruction part, it is necessary to mask the instruction part of the actual character sequence p(x) to avoid inconsistency in the calculation of the loss function.

[0080] After defining the monotonicity restriction loss function of the speech recognition model during training, the present invention can construct the target loss function of the speech recognition model during training based on the defined monotonicity restriction loss function, including: weighting the monotonicity restriction loss function and the cross loss function of the speech recognition model to obtain the target loss function of the speech recognition model during training, and the calculation formula of the target loss function of the speech recognition model during training is shown in formula (6):

[0081] L total =k1L Mono +k2L CE (6)

[0082] Among them, L total represents the target loss function of the speech recognition model, L Mono represents the monotonicity restriction loss function of the speech recognition model, L CE represents the cross entropy loss function of the speech recognition model, k1 and k2 represent the monotonicity restriction loss function L of the speech recognition model Mono And the cross entropy loss function L of the speech recognition model CE ’s weight coefficient, and k1+k2=1.

[0083] S70: Based on the constructed target loss function, the speech recognition model during training is updated to obtain a trained speech recognition model. In the present invention, the probability distribution of the character sequence predicted by the speech recognition model and the corresponding character sequence are substituted into the constructed target loss function, the loss value during the speech recognition model training is calculated, and the various parameter information in the speech recognition model training is updated based on the calculated loss value.

[0084] It can be seen that in the above scheme, for speech recognition in specific scenarios, by defining specific task input instructions for specific scenarios of speech recognition, the speech recognition model can be guided to pay more attention to the semantics of specific scenarios during training, and a monotonicity restriction loss function of the speech recognition model during training is constructed, so that the speech recognition model can pay attention to the monotonic alignment characteristics of the input speech signal and the output text sequence of the specific scenario, thereby improving the accuracy of the speech recognition model in speech recognition in specific scenarios.

[0085] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0086] In one embodiment, a training device for a speech recognition model is provided, and the training device for the speech recognition model corresponds one-to-one to the training method for the speech recognition model in the above embodiment. Figure 8 As shown, Figure 8 8 is a schematic diagram of a structure of a training device for a speech recognition model in an embodiment of the present invention, the training device for the speech recognition model includes an acquisition module 81, a feature extraction module 82, an encoding module 83, an adjustment module 84, a decoding module 85, a construction module 86 and an update module 87. The functional modules are described in detail as follows:

[0087] An acquisition module 81 is used to acquire a pre-trained speech recognition model and multi-task speech training data corresponding to the pre-trained speech recognition model, and perform data preprocessing on the acquired multi-task speech training data;

[0088] A feature extraction module 82 is used to extract audio features from the multi-task speech training data after data preprocessing to obtain an audio feature sequence of the multi-task speech training data;

[0089] An encoding module 83, used for inputting the extracted audio feature sequence of the multi-task speech training data into an encoder of the pre-trained speech recognition model for encoding, so as to obtain an audio encoding vector of the multi-task speech training data;

[0090] An adjustment module 84 is used to obtain task-specific speech training data, define task-specific input instructions according to characteristic scenarios of speech recognition, and adjust the input format of the prior text sequence of the pre-trained speech recognition model according to the defined task-specific input instructions;

[0091] A decoding module 85, for inputting the priori text sequence after adjusting the input format and the audio encoding vector of the multi-task speech training data into the decoder of the pre-trained speech recognition model for decoding, and predicting the character sequence output by the pre-trained speech recognition model according to the decoding result;

[0092] A construction module 86 is used to define a monotonicity restriction loss function of the speech recognition model during training according to the character sequence output by the predicted pre-trained speech recognition model, and to construct a target loss function of the speech recognition model during training based on the defined monotonicity restriction loss function;

[0093] The updating module 87 is used to update the speech recognition model during training based on the constructed target loss function to obtain a trained speech recognition model.

[0094] In one embodiment, the acquisition module 81 is specifically used to:

[0095] Performing frame processing on the speech signal in the multi-task speech training data, and performing window processing on the frame-processed speech signal;

[0096] Performing discrete Fourier transform processing on the speech signal after windowing to obtain the frequency spectrum characteristics of the multi-task speech training data;

[0097] Using a Mel frequency filter to convert the frequency spectrum features of the multi-task speech training data to obtain a Mel frequency spectrum of the multi-task speech training data;

[0098] A logarithmic operation is performed on the Mel spectrum of the multi-task speech training data to obtain a logarithmic Mel spectrum graph of the multi-task speech training data.

[0099] In one embodiment, the feature extraction module 82 is specifically used to:

[0100] Input the multi-task speech training data after data preprocessing into the convolution layer to extract the audio feature vector of the multi-task speech training data;

[0101] The extracted audio feature information of the multi-task speech training data is input into a first position encoder for position encoding to obtain an audio feature sequence of the multi-task speech training data.

[0102] In one embodiment, the encoding module 83 is specifically configured to:

[0103] Inputting the extracted audio feature sequence of the multi-task speech training data into the self-attention mechanism layer of the encoder, and assigning self-attention weights to the audio feature sequence of the multi-task speech training data;

[0104] The audio sequence after the self-attention weights are assigned is input into the feedforward linear layer of the encoder, and the audio sequence is processed by nonlinear changes to obtain the audio encoding vector of the multi-task speech training data.

[0105] In one embodiment, the decoding module 85 is specifically configured to:

[0106] Inputting the priori text sequence after adjusting the input format into the second position encoder, performing position encoding on the text feature sequence in the specific task speech training data, and obtaining a text encoding vector of the specific task speech training data;

[0107] Input the text encoding vector of the specific task speech training data and the audio encoding vector of the multi-task speech training data into the self-attention mechanism layer of the decoder, and assign self-attention weights to the text encoding vector of the specific task speech training data and the audio encoding vector of the multi-task speech training data;

[0108] Input the text encoding vector of the specific task speech training data and the audio encoding vector of the multi-task speech training data after assigning the self-attention weights into the cross attention mechanism layer of the decoder, fuse the text encoding vector of the specific task speech training data and the audio encoding vector of the multi-task speech training data to obtain the target encoding vector;

[0109] Inputting the target encoding vector into a feedforward linear layer of a decoder, and performing nonlinear change processing on the target encoding vector;

[0110] Based on the target encoding vector after nonlinear change processing, the character sequence output by the pre-trained speech recognition model is predicted to obtain a prediction result of the character sequence output by the pre-trained speech recognition model.

[0111] In one embodiment, the construction module 86 is specifically configured to:

[0112] The calculation formula of the monotonicity restriction loss function of the speech recognition model during the training is:

[0113] L monoi =Relu(1-Δδ)

[0114] Among them, L monoi represents the monotonicity restriction loss function of the i-th cross attention layer, Relu(.) represents the activation function, △δ is the difference between the character sequence currently output by the pre-trained speech recognition model and the character sequence output at the previous moment;

[0115] L mono =λ1L momo1 +λ2L momo1 +…+λ 20 L momo1

[0116] Among them, L mono represents the monotonicity restriction loss function of the speech recognition model during training, λ i Represents the weight of the monotonicity constraint loss function of the speech recognition model during training.

[0117] In one embodiment, the construction module 86 is specifically configured to:

[0118] The monotonicity restriction loss function and the cross loss function of the speech recognition model are weighted to obtain the target loss function of the speech recognition model during training;

[0119] The calculation formula of the target loss function of the speech recognition model during the training is:

[0120] L total =k1L Mono +k2L CE

[0121] Among them, L total represents the target loss function of the speech recognition model, L Mono represents the monotonicity restriction loss function of the speech recognition model, L CE represents the cross entropy loss function of the speech recognition model, k1 and k2 represent the monotonicity restriction loss function L of the speech recognition model Mono And the cross entropy loss function L of the speech recognition model CE ’s weight coefficient, and k1+k2=1.

[0122] The present invention provides a training device for a speech recognition model. For speech recognition in specific scenarios, by defining specific task input instructions for the specific scenarios of speech recognition, the speech recognition model is guided to pay more attention to the semantics of the specific scenarios during training, and a monotonicity restriction loss function of the speech recognition model during training is constructed, so that the speech recognition model can pay attention to the monotonic alignment characteristics of the input speech signal and the output text sequence of the specific scenario, thereby improving the accuracy of the speech recognition model in speech recognition in the specific scenario.

[0123] For the specific definition of the training device of the speech recognition model, please refer to the definition of the training method of the speech recognition model in the above text, which will not be repeated here. The various modules in the above-mentioned training device of the speech recognition model can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0124] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown, Fig. 9 1 is a schematic diagram of the structure of a computer device in one embodiment of the present invention. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the functions or steps on the server side of a training method for a speech recognition model are implemented.

[0125] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Fig.10 As shown, Fig.10 1 is another structural diagram of a computer device in one embodiment of the present invention. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, the functions or steps on the client side of a method for training a speech recognition model are implemented.

[0126] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0127] Acquire a pre-trained speech recognition model and multi-task speech training data corresponding to the pre-trained speech recognition model, and perform data preprocessing on the acquired multi-task speech training data;

[0128] Performing audio feature extraction on the multi-task speech training data after data preprocessing to obtain an audio feature sequence of the multi-task speech training data;

[0129] Inputting the extracted audio feature sequence of the multi-task speech training data into the encoder of the pre-trained speech recognition model for encoding to obtain an audio encoding vector of the multi-task speech training data;

[0130] Acquire task-specific speech training data, define task-specific input instructions according to a specific scenario of speech recognition, and adjust the input format of a priori text sequence of a pre-trained speech recognition model according to the defined task-specific input instructions;

[0131] Inputting the prior text sequence after adjusting the input format and the audio encoding vector of the multi-task speech training data into the decoder of the pre-trained speech recognition model for decoding, and predicting the character sequence output by the pre-trained speech recognition model according to the decoding result;

[0132] Defining a monotonicity restriction loss function of the speech recognition model during training according to the predicted character sequence output by the pre-trained speech recognition model, and constructing a target loss function of the speech recognition model during training based on the defined monotonicity restriction loss function;

[0133] The speech recognition model during training is updated based on the constructed target loss function to obtain a trained speech recognition model.

[0134] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0135] Acquire a pre-trained speech recognition model and multi-task speech training data corresponding to the pre-trained speech recognition model, and perform data preprocessing on the acquired multi-task speech training data;

[0136] Performing audio feature extraction on the multi-task speech training data after data preprocessing to obtain an audio feature sequence of the multi-task speech training data;

[0137] Inputting the extracted audio feature sequence of the multi-task speech training data into the encoder of the pre-trained speech recognition model for encoding to obtain an audio encoding vector of the multi-task speech training data;

[0138] Acquire task-specific speech training data, define task-specific input instructions according to a specific scenario of speech recognition, and adjust the input format of a priori text sequence of a pre-trained speech recognition model according to the defined task-specific input instructions;

[0139] Inputting the prior text sequence after adjusting the input format and the audio encoding vector of the multi-task speech training data into the decoder of the pre-trained speech recognition model for decoding, and predicting the character sequence output by the pre-trained speech recognition model according to the decoding result;

[0140] Defining a monotonicity restriction loss function of the speech recognition model during training according to the predicted character sequence output by the pre-trained speech recognition model, and constructing a target loss function of the speech recognition model during training based on the defined monotonicity restriction loss function;

[0141] The speech recognition model during training is updated based on the constructed target loss function to obtain a trained speech recognition model.

[0142] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0144] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0145] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.< / sot> < / sop> < / sot>

Claims

1. A method for training a speech recognition model, characterized in that: include: Acquire a pre-trained speech recognition model and multi-task speech training data corresponding to the pre-trained speech recognition model, and perform data preprocessing on the acquired multi-task speech training data; Performing audio feature extraction on the multi-task speech training data after data preprocessing to obtain an audio feature sequence of the multi-task speech training data; Inputting the extracted audio feature sequence of the multi-task speech training data into the encoder of the pre-trained speech recognition model for encoding to obtain an audio encoding vector of the multi-task speech training data; Acquire task-specific speech training data, define task-specific input instructions according to a specific scenario of speech recognition, and adjust the input format of a priori text sequence of a pre-trained speech recognition model according to the defined task-specific input instructions; Inputting the prior text sequence after adjusting the input format and the audio encoding vector of the multi-task speech training data into the decoder of the pre-trained speech recognition model for decoding, and predicting the character sequence output by the pre-trained speech recognition model according to the decoding result; Constructing a monotonicity restriction loss function of the speech recognition model during training according to the predicted character sequence output by the pre-trained speech recognition model, and constructing a target loss function of the speech recognition model during training based on the constructed monotonicity restriction loss function; The speech recognition model during training is updated based on the constructed target loss function to obtain a trained speech recognition model.

2. The method for training a speech recognition model according to claim 1, characterized in that: The data preprocessing of the acquired multi-task speech training data includes: Performing frame processing on the speech signal in the multi-task speech training data, and performing window processing on the frame-processed speech signal; Performing discrete Fourier transform processing on the speech signal after the windowing processing to obtain the frequency spectrum characteristics of the multi-task speech training data; Using a Mel frequency filter to convert the frequency spectrum features of the multi-task speech training data to obtain a Mel frequency spectrum of the multi-task speech training data; A logarithmic operation is performed on the Mel spectrum of the multi-task speech training data to obtain a logarithmic Mel spectrum graph of the multi-task speech training data.

3. The method for training a speech recognition model according to claim 1, characterized in that: The step of extracting audio features from the multi-task speech training data after data preprocessing to obtain an audio feature sequence of the multi-task speech training data includes: Input the multi-task speech training data after data preprocessing into the convolution layer to extract the audio feature vector of the multi-task speech training data; The extracted audio feature information of the multi-task speech training data is input into a first position encoder for position encoding to obtain an audio feature sequence of the multi-task speech training data.

4. The method for training a speech recognition model according to claim 1, characterized in that: The step of inputting the extracted audio feature sequence of the multi-task speech training data into the encoder of the pre-trained speech recognition model for encoding processing to obtain the audio encoding vector of the multi-task speech training data includes: Inputting the extracted audio feature sequence of the multi-task speech training data into the self-attention mechanism layer of the encoder, and assigning self-attention weights to the audio feature sequence of the multi-task speech training data; The audio sequence after the self-attention weights are assigned is input into the feedforward linear layer of the encoder, and the audio sequence is processed by nonlinear changes to obtain the audio encoding vector of the multi-task speech training data.

5. The method for training a speech recognition model according to claim 1, characterized in that: The method comprises inputting the priori text sequence after adjusting the input format and the audio coding vector of the multi-task speech training data into the decoder of the pre-trained speech recognition model for decoding, and predicting the character sequence output by the pre-trained speech recognition model according to the decoding result, including: Inputting the priori text sequence after adjusting the input format into the second position encoder, performing position encoding on the text feature sequence in the specific task speech training data, and obtaining a text encoding vector of the specific task speech training data; Input the text encoding vector of the specific task speech training data and the audio encoding vector of the multi-task speech training data into the self-attention mechanism layer of the decoder, and assign self-attention weights to the text encoding vector of the specific task speech training data and the audio encoding vector of the multi-task speech training data; Input the text encoding vector of the specific task speech training data and the audio encoding vector of the multi-task speech training data after assigning the self-attention weights into the cross attention mechanism layer of the decoder, fuse the text encoding vector of the specific task speech training data and the audio encoding vector of the multi-task speech training data to obtain the target encoding vector; Inputting the target encoding vector into a feedforward linear layer of a decoder, and performing nonlinear change processing on the target encoding vector; Based on the target encoding vector after nonlinear change processing, the character sequence output by the pre-trained speech recognition model is predicted to obtain a prediction result of the character sequence output by the pre-trained speech recognition model.

6. The method for training a speech recognition model according to claim 1, characterized in that: The monotonicity restriction loss function of the speech recognition model during training is constructed according to the character sequence output by the predicted pre-trained speech recognition model, including: The calculation formula of the monotonicity restriction loss function of the speech recognition model during the training is: L monoi =Relu(1-Δδ) Among them, L monoi represents the monotonicity restriction loss function of the i-th cross attention layer, Relu(.) represents the activation function, △δ is the difference between the character sequence currently output by the pre-trained speech recognition model and the character sequence output at the previous moment; L mono =λ1L momo1 +λ2L momo2 +…+l 20 L momo20 Among them, L mono represents the monotonicity restriction loss function of the speech recognition model during training, λ i Represents the weight of the monotonicity constraint loss function of the speech recognition model during training.

7. The method for training a speech recognition model according to claim 6, characterized in that: The target loss function of the speech recognition model is constructed based on the constructed monotonicity restriction loss function, including: The monotonicity restriction loss function and the cross loss function of the speech recognition model are weighted to obtain the target loss function of the speech recognition model during training; The calculation formula of the target loss function of the speech recognition model during the training is: IT total =k1L Mono +k2L CE Among them, L total represents the target loss function of the speech recognition model, L Mono represents the monotonicity restriction loss function of the speech recognition model, L CE represents the cross entropy loss function of the speech recognition model, k1 and k2 represent the monotonicity restriction loss function L of the speech recognition model Mono And the cross entropy loss function L of the speech recognition model CE ’s weight coefficient, and k1+k2=1.

8. A speech recognition model training device, the device being used to implement the speech recognition model training method according to any one of claims 1 to 7, characterized in that: include: An acquisition module, used to acquire a pre-trained speech recognition model and multi-task speech training data corresponding to the pre-trained speech recognition model, and perform data preprocessing on the acquired multi-task speech training data; A feature extraction module is used to extract audio features from the multi-task speech training data after data preprocessing to obtain an audio feature sequence of the multi-task speech training data; An encoding module, used for inputting the extracted audio feature sequence of the multi-task speech training data into the encoder of the pre-trained speech recognition model for encoding, so as to obtain an audio encoding vector of the multi-task speech training data; An adjustment module, used to obtain task-specific speech training data, define task-specific input instructions according to a specific scenario of speech recognition, and adjust the input format of a priori text sequence of a pre-trained speech recognition model according to the defined task-specific input instructions; A decoding module, used to input the prior text sequence after adjusting the input format and the audio encoding vector of the multi-task speech training data into the decoder of the pre-trained speech recognition model for decoding, and predict the character sequence output by the pre-trained speech recognition model according to the decoding result; A construction module, used to construct a monotonicity restriction loss function of the speech recognition model during training according to the character sequence output by the predicted pre-trained speech recognition model, and to construct a target loss function of the speech recognition model during training based on the constructed monotonicity restriction loss function; The updating module is used to update the speech recognition model during training based on the constructed target loss function to obtain a trained speech recognition model.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for training a speech recognition model as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for training a speech recognition model as described in any one of claims 1 to 7 are implemented.

Images